Error passing endpoint_setup to compute endpoint from metadata

[hholb]

I am trying to pass an endpoint_setup arg to the MEP on edith, here is the config I am working with:

# /// script
# requires-python = "==3.12.*"
# dependencies = [
#     "ase",
#     "torch==2.4.0",
#     "torch-geometric",
#     "cuequivariance",
#     "cuequivariance-torch",
#     "cuequivariance-ops-torch-cu12",
#     "torch-sim-atomistic[mace] @ git+https://github.com/TorchSim/torch-sim.git",
#     "mace-torch[queq-cuda-12]",
#     "parsl",
#     "pymatgen",
#     "pandas",
#     "numpy",
#     "pyarrow",
#     "globus-compute-endpoint",
# ]
#
# [tool.hog.edith]
# endpoint = "a01b9350-e57d-4c8e-ad95-b4cb3c4cd1bb"
# worker_init = """
# set -eux;
# module load openmpi/5.0.3;
# """
# endpoint_setup = """
# conda create -y -n test-env
# conda activate test-env
# pip install uv
# """

Here is the error I am seeing:

 ❯ uv run hog run mace_batch_relax.py test_edith
Relaxing on edith...
| mace_batch_relax | task pending | status pending | 3.7s | ☁️🦫
Error: ('POST', 'https://compute.api.globus.org/v3/endpoints/a01b9350-e57d-4c8e-ad95-b4cb3c4cd1bb/submit', 'Bearer', 422, 'SEMANTICALLY_INVALID', 'Request payload failed validation: Failed to start or unexpected error:\n  (SystemExit) 78')

I don't see the same error if I pass the endpoint_setup as a kwarg to the remote call:

@hog.harness()
def test_edith():
    with open("./test_structures.xyz", "r") as f:
        xyz_content = f.read()

    endpoint_setup = """
    conda create -y -n test-env
    conda activate test-env
    pip install uv
    """
    print("Relaxing on edith...")
    contents = mace_batch_relax.remote(
        xyz_content,
        endpoint="edith",
        endpoint_setup=endpoint_setup,
    )
    print("Edith result:\n", contents)

The job at least starts (still having issues on edith, but that's a different issue):

 ❯ uv run hog run mace_batch_relax.py test_edith
Relaxing on edith...
| mace_batch_relax | ea4fb4fb-5693-4ffb-816b-4dd11da22ad4 | waiting-for-ep | 4m 11s | ☀️🦫️

[OwenPriceSkelly]

very weird! I wonder if it has to do with the way multiline strings are parsed by the toml library vs the pure python multiline string?

One quick thing you can do to test is to use submit() instead of remote(), and inspect the future.user_endpoint_config['endpoint_setup'] to see what final config was actually passed to the executor in either case

[OwenPriceSkelly]

(also as a side note -- you don't need globus-compute-endpoint in the dependencies, might speed things up a little to cut that out)