feat: add run_experiment script (#77)

* refactor: Update to property model

Aim is to not include PropertyDomain objects where not required reducing size of models and schemas.

Another aim is to fix recursion in ExperimentReference due to the PropertyValue model having field that can be ObservedPropertyValue, which in turn will include an Experiment Reference

Changes:
- Add PropertyDescriptors: variants of Property types that do not have a domain or metadata
- Change models that only require Descriptors to use them (Entity,PropertyValue)
- Allow models that take PropertyDescriptors to still be initialised with Properties (converting them) for flexibility and compatibility
- Update tests
- Add additional conversions where required

* chore: add types

* feat: Specific PropertyValue subclasses for Constitutive, Observed and Virtual properties

You cannot use the base PropertyValue for Observed or Virtual property values

* refactor: Update schema models to use new PropertyValue types.

In particular update input and return types to strictly declare which type they use (in particular to take/use ConstitutivePropertyValue if this is expected not PropertyValue which is compatible but generic)

* test: Update schema tests and fixtures to new PropertyValue subclasses

* refactor: Update other orchestrator code to new PropertyValue types

* test: Update other orchestrator tests and fixtures to new PropertyType

* refactor: Update sfttrainer and related tests to new PropertyType models

* refactor: Update raytune operator to new PropertyType models

* refactor: Update examples to new PropertyType subclasses

* refactor: Change experiment target properties to be descriptors

* refactor: Change experiment targetProperty field type to descriptors

* fix: Missing __str__ method

* chore: remove stray prints

* chore: remove stray prints

* fix: models of incorrect type raise validationerror immediately

The before validator is never called

* refactor: remove concretePropertyIdentifiers field

Never used.

* fix: revert change in before validator

This class method can also be called directly in which case it can convert a Property instance to a descriptor

* test: measurement_for_entity

* chore: update comment

* chore: revert part of 5ea1e98

Keep concretePropertyIdentifiers field in AbstractProperty as otherwise we break stored models.

* fix: property_to_descriptor for AbstractPropertyDescriptor

* Apply suggestion from @AlessandroPomponio

Co-authored-by: Alessandro Pomponio <[email protected]>
Signed-off-by: Michael Johnston <[email protected]>

* Apply suggestion from @AlessandroPomponio

Co-authored-by: Alessandro Pomponio <[email protected]>
Signed-off-by: Michael Johnston <[email protected]>

* feat: add validate_point_against_properties function

A base function for comparing a set of "identifier:value" pairs against a set of constitutive properties.

* refactor: use validate_point_against_properties

code in isPointInSpace is basis of new function

* refactor: move SpacePoint to schema

Also change experiment field to be list of ExperimentReferences.

* feat: add validate_entity

* chore: add classmethod decorator

Co-authored-by: Alessandro Pomponio <[email protected]>
Signed-off-by: Michael Johnston <[email protected]>

* feat: add check against optional properties

* feat: add run_experiment script

* chore: typing

* fix: not accessing items

* chore: tidy

* chore: missing headers

* fix: non-deterministic results from nevergrad optimization test function

by default translation is 1.0 which randomly translates the space of the ArtificialFunction. Multiple calls to the experiment then were actually are using different randomly translated functions.

* feat: remote execution and typer

* docs(website): add run_experiment docs

* refactor: rename wrappers

* feat: handle various errors from remote call

* feat: improve typer usage

* chore: formatting

* fix: remove unnecessary code

* fix: missing return

* chore: use timedelta

* docs(examples): example point.yaml

* test: add test of run_experiment

* docs: improvements

* chore: missing copyright header

* fix: allow int or string debug level

---------

Signed-off-by: Michael Johnston <[email protected]>
Co-authored-by: Alessandro Pomponio <[email protected]>

Author: michael-johnston

examples/optimization_test_functions/custom_experiments/ado_actuators/optimization_test_functions/optimization_test_functions.py

@@ -30,6 +30,7 @@ def artificial_function(
         block_dimension=int(
             len(entity.constitutive_property_values) / parameters["num_blocks"]
         ),
+        translation_factor=0.0,
     )
 
     # Call the nevergrad function

examples/optimization_test_functions/point.yaml

@@ -0,0 +1,10 @@
+# Copyright (c) IBM Corporation
+# SPDX-License-Identifier: MIT
+
+entity:
+ x0: 1
+ x1: 2
+ x2: -1
+experiments:
+- actuatorIdentifier: custom_experiments
+  experimentIdentifier: nevergrad_opt_3d_test_func

orchestrator/cli/models/space.py

@@ -2,11 +2,8 @@
 # SPDX-License-Identifier: MIT
 
 import math
-import typing
 from io import StringIO
-from typing import Annotated
 
-import pydantic
 import rich.table
 
 import orchestrator.metastore.project
@@ -430,16 +427,3 @@ def to_dataframe(
                 columns_to_hide=columns_to_hide,
             )
         )
-
-
-class SpacePoint(pydantic.BaseModel):
-    entity: Annotated[
-        dict[str, typing.Any] | None,
-        pydantic.Field(
-            description="A dictionary of property names and values to be searched."
-        ),
-    ] = None
-    experiments: Annotated[
-        list[str] | None,
-        pydantic.Field(description="A list of experiments to be searched."),
-    ] = None

orchestrator/cli/resources/discovery_space/get.py

@@ -8,7 +8,6 @@
 from rich.status import Status
 
 from orchestrator.cli.models.parameters import AdoGetCommandParameters
-from orchestrator.cli.models.space import SpacePoint
 from orchestrator.cli.models.types import (
     AdoGetSupportedOutputFormats,
 )
@@ -32,6 +31,7 @@
 )
 from orchestrator.core.resources import CoreResourceKinds
 from orchestrator.schema.entityspace import EntitySpaceRepresentation
+from orchestrator.schema.point import SpacePoint
 
 if typing.TYPE_CHECKING:
     import pandas as pd
@@ -118,10 +118,14 @@ def _find_spaces_matching_point(
         raise
 
     experiment_selectors = []
-    for experiment in target_point.experiments or []:
+    for reference in target_point.experiments or []:
         experiment_selectors.extend(
             prepare_query_filters_for_db(
-                {"config.experiments": {"experiments": {"identifier": experiment}}}
+                {
+                    "config.experiments": {
+                        "experiments": {"identifier": reference.experimentIdentifier}
+                    }
+                }
             )
         )
 

orchestrator/schema/entityspace.py

@@ -6,7 +6,10 @@
 from orchestrator.schema.domain import VariableTypeEnum
 from orchestrator.schema.entity import Entity
 from orchestrator.schema.property import ConstitutiveProperty
-from orchestrator.schema.property_value import constitutive_property_values_from_point
+from orchestrator.schema.property_value import (
+    constitutive_property_values_from_point,
+    validate_point_against_properties,
+)
 from orchestrator.schema.result import MeasurementResult
 
 
@@ -224,52 +227,20 @@ def isPointInSpace(
             Dictionary representing the point with constitutive property identifiers as keys and
             constitutive property values as values.
         :param allow_partial_matches:
-            If True, ill return True if at least one property matches; otherwise the point must
-             match all the constitutive properties of the entities.
+            If True, all key:values in point must have a matching constitutive property in the
+            entity space and be in its domain.
+            If False (default), in addition to above, all constitutive properties in the entity space
+            must have a matching key in point
 
         :return: True if the point is in the space, False otherwise.
         """
-        constitutive_property_identifiers_for_point = set(point.keys())
-        constitutive_property_identifiers_for_entity_space = {
-            cp.identifier for cp in self._constitutiveProperties
-        }
 
-        matching_constitutive_property_identifiers = (
-            constitutive_property_identifiers_for_point.intersection(
-                constitutive_property_identifiers_for_entity_space
-            )
+        return validate_point_against_properties(
+            point=point,
+            constitutive_properties=self._constitutiveProperties,
+            allow_partial_matches=allow_partial_matches,
         )
 
-        # If we don't allow partial matches, all properties must match
-        if not allow_partial_matches and len(
-            matching_constitutive_property_identifiers
-        ) != len(constitutive_property_identifiers_for_entity_space):
-            return False
-
-        # If we allow partial matches, all properties for the point must
-        # match
-        if len(matching_constitutive_property_identifiers) != len(
-            constitutive_property_identifiers_for_point
-        ):
-            return False
-
-        # Once we have checked that the identifiers match, we must
-        # check that the values specified by the point are in the domain
-        # of the constitutive properties.
-        for constitutive_property in self.constitutiveProperties:
-            if (
-                constitutive_property.identifier
-                not in matching_constitutive_property_identifiers
-            ):
-                continue
-
-            if not constitutive_property.propertyDomain.valueInDomain(
-                point[constitutive_property.identifier]
-            ):
-                return False
-
-        return True
-
     def isEntityInSpace(self, entity: Entity):
         """Returns True if entity is in the space otherwise false
 

orchestrator/schema/experiment.py

@@ -2,6 +2,7 @@
 # SPDX-License-Identifier: MIT
 
 import importlib.metadata
+import logging
 import typing
 from typing import Annotated
 
@@ -18,7 +19,11 @@
     MeasuredPropertyTypeEnum,
     Property,
 )
-from orchestrator.schema.property_value import ConstitutivePropertyValue, PropertyValue
+from orchestrator.schema.property_value import (
+    ConstitutivePropertyValue,
+    PropertyValue,
+    validate_point_against_properties,
+)
 from orchestrator.schema.reference import (
     ExperimentReference,
     check_parameterization_validity,
@@ -615,6 +620,74 @@ def propertyValuesFromEntity(self, entity: "Entity", target=False) -> dict:
 
         return identifierValueMap
 
+    def validate_entity(self, entity: "Entity", strict_optional=False) -> bool:
+        """Returns True if Experiment can be applied to entity, false otherwise
+
+        This method only checks constitutive properties.
+        - All properties of the Entity that match the experiments required or optional properties must have
+        values in the domain of that property
+        - All required properties of the experiment must have a matching constitutive property
+        - If strict_optional is True all properties of the Entity that are not required properties of the Experiment
+        must match optional properties of the experiment.
+        """
+
+        point = {
+            v.property.identifier: v.value for v in entity.constitutive_property_values
+        }
+        if validate_point_against_properties(
+            point,
+            constitutive_properties=self.requiredConstitutiveProperties,
+        ):
+            return True
+
+        # It's not an exact match - check if partial match
+        if not validate_point_against_properties(
+            point,
+            constitutive_properties=self.requiredConstitutiveProperties,
+            allow_partial_matches=True,
+        ):
+            # no partial match - missing required properties or has incorrect values for them
+            logging.getLogger("experiment").warning(
+                f"The entity is missing or has invalid values for required properties of "
+                f" {self.identifier}"
+            )
+            return False
+
+        # It has the required properties with valid values but there are additional properties
+        # See if these properties are optional propertiesof the experiment
+        potential_optional_properties: set[str] = point.keys() - {
+            cp.identifier for cp in self.requiredProperties
+        }
+        optional_properties = potential_optional_properties & {
+            cp.identifier for cp in self.optionalProperties
+        }
+        # If strict_optional is on all the additional properties must be optional properties
+        if (
+            len(optional_properties) != len(potential_optional_properties)
+            and strict_optional
+        ):
+            logging.getLogger("experiment").warning(
+                f"Strict property checking is on and the following entity "
+                f"properties are not required or optional properties of {self.identifier}:"
+                f"{potential_optional_properties-optional_properties} "
+            )
+            return False
+
+        is_valid = validate_point_against_properties(
+            point={key: point[key] for key in optional_properties},
+            constitutive_properties=list(self.optionalProperties),
+            allow_partial_matches=True,
+        )
+        if not is_valid:
+            logging.getLogger("experiment").warning(
+                f"The entity has properties that match optional properties"
+                f"of {self.identifier} - "
+                f"{potential_optional_properties - optional_properties} - "
+                f"but its values for those properties are not in the domain of the optional properties"
+            )
+
+        return is_valid
+
 
 class ParameterizedExperiment(Experiment):
     """Represents an Experiment where default optional parameters have been overridden

orchestrator/schema/point.py

@@ -0,0 +1,37 @@
+# Copyright (c) IBM Corporation
+# SPDX-License-Identifier: MIT
+
+import typing
+from typing import Annotated
+
+import pydantic
+
+from orchestrator.schema.entity import Entity
+from orchestrator.schema.property import ConstitutivePropertyDescriptor
+from orchestrator.schema.property_value import ConstitutivePropertyValue
+from orchestrator.schema.reference import ExperimentReference
+
+
+class SpacePoint(pydantic.BaseModel):
+    """A simplified representation of an Entity and an associated set of experiments"""
+
+    entity: Annotated[
+        dict[str, typing.Any] | None,
+        pydantic.Field(description="A dictionary of property name:value pairs"),
+    ] = None
+    experiments: list[ExperimentReference] | None = (
+        pydantic.Field(default=None, description="A list of experiments"),
+    )
+
+    def to_entity(self) -> Entity:
+
+        return Entity(
+            constitutive_property_values=tuple(
+                [
+                    ConstitutivePropertyValue(
+                        value=v, property=ConstitutivePropertyDescriptor(identifier=k)
+                    )
+                    for k, v in self.entity.items()
+                ]
+            )
+        )

orchestrator/schema/property_value.py

@@ -2,6 +2,7 @@
 # SPDX-License-Identifier: MIT
 
 import enum
+import typing
 
 import pydantic
 
@@ -161,3 +162,65 @@ def constitutive_property_values_from_point(
         ConstitutivePropertyValue(value=point[c.identifier], property=c)
         for c in properties
     ]
+
+
+def validate_point_against_properties(
+    point: dict[str, typing.Any],
+    constitutive_properties: list[ConstitutiveProperty],
+    allow_partial_matches: bool = False,
+):
+    """point is valid if all its keys have a constitutive_property with
+    a matching identifier and all its values are in the domain of this
+    property. If allow_partial_matches is False an additional condition is that
+    every key in point has a matching property AND vice versa
+
+    Params:
+        point: A dictionary whose keys are property identifiers and values are values for those properties
+        constitutive_properties: A list of ConstitutiveProperty instances to validate the point against
+        allow_partial_matches: If True a point is valid if all its points have matching properties, even if
+            there are more constitutive properties
+
+    Returns:
+        - True if point is compatible with space otherwise false
+    """
+
+    constitutive_property_identifiers_for_point = set(point.keys())
+    constitutive_property_identifiers_for_entity_space = {
+        cp.identifier for cp in constitutive_properties
+    }
+
+    matching_constitutive_property_identifiers = (
+        constitutive_property_identifiers_for_point.intersection(
+            constitutive_property_identifiers_for_entity_space
+        )
+    )
+
+    # If we don't allow partial matches, all properties must match
+    if not allow_partial_matches and len(
+        matching_constitutive_property_identifiers
+    ) != len(constitutive_property_identifiers_for_entity_space):
+        return False
+
+    # If we allow partial matches, all properties for the point must
+    # match
+    if len(matching_constitutive_property_identifiers) != len(
+        constitutive_property_identifiers_for_point
+    ):
+        return False
+
+    # Once we have checked that the identifiers match, we must
+    # check that the values specified by the point are in the domain
+    # of the constitutive properties.
+    for constitutive_property in constitutive_properties:
+        if (
+            constitutive_property.identifier
+            not in matching_constitutive_property_identifiers
+        ):
+            continue
+
+        if not constitutive_property.propertyDomain.valueInDomain(
+            point[constitutive_property.identifier]
+        ):
+            return False
+
+    return True