Hyperion 1.6.4 testing

[rtuck99]

Items to test for next i03 deployment:

• Implement robot unload for udc default state #1607 - FAIL - robot unload causes external callbacks to exit due to no visit
  • Addressed by Fixes for UDC Default State robot unload and panda PCAP path provider setting #1669
• Debug, file logging handlers for hyperion-blueapi #1377
• hyperion-supervisor cannot run load_centre_collect unable to serialize parameter_model_version #1564
• Hyperion: Think of a way to do a global timeout/heartbeat for plans as well callbacks #1362
• i03 to use latest GDA in production #1589 will only test I03-1052
• Read beamline parameters through config server #1511 - will need config server deployed to i03 beamline
• nfs4_setfacl script changes (SCHD-10817)
• Wait for the robot to report the puck/pin value instead of just waiting for beamline enable flag dodal#1951

Also test latest attempt at #1598

[jacob720]

#1511 - will need config server deployed to i03 beamline

This has already been deployed to i03 - I'll come down when you test all this to help test the config server stuff
https://argocd.diamond.ac.uk/applications/i03-beamline/i03-daq-config-server
https://i03-daq-config.diamond.ac.uk/docs

[rtuck99]

deployment script:

lots of

setfacl: /dls_sw/i03/software/bluesky/mx-bluesky_1.6.4/dodal/tests/devices/beamlines/p60: Operation not supported

still pending updated permissions script

• uv complains

         If the cache and target directories are on different filesystems, hardlinking may not be supported.
         If this is intentional, set `export UV_LINK_MODE=copy` or use `--link-mode=copy` to suppress this warning.

possibly we don't care about this - since we don't expect to be doing much in the way of operations with uv on live systems.

[rtuck99]

Hotfix for missing spring client beans for I03-1052

[rtuck99]

It seems that the install script has symlinked python to my personal install directory:

lrwxrwxrwx. 1 yqf46943 dls_dasc       85 Mar  3 11:50 python -> /home/yqf46943/.local/share/uv/python/cpython-3.11.14-linux-x86_64-gnu/bin/python3.11

Fixed by:

• Use the python defined by module load python #1659

[rtuck99]

Hyperion does not start with the correct port in UDC mode

Fixed by:

• mx-bluesky starts in UDC mode with wrong port number #1662

[rtuck99]

Hyperion refused to respond to further baton handovers when agamemnon returned an error:

[31m[E 260303 15:48:52.877 run_engine:1755] Run abortedESC[0m
Traceback (most recent call last):
  File "/dls_sw/i03/software/bluesky/mx-bluesky_1.6.4/mx-bluesky/.venv/lib/python3.11/site-packages/bluesky/run_engine.py", line 1610, in _run
    msg = self._plan_stack[-1].send(resp)
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/dls_sw/i03/software/bluesky/mx-bluesky_1.6.4/mx-bluesky/src/mx_bluesky/hyperion/baton_handler.py", line 121, in collect_then_release
    yield from bpp.contingency_wrapper(
  File "/dls_sw/i03/software/bluesky/mx-bluesky_1.6.4/mx-bluesky/.venv/lib/python3.11/site-packages/bluesky/preprocessors.py", line 622, in contingency_wrapper
    ret = yield from plan
          ^^^^^^^^^^^^^^^
  File "/dls_sw/i03/software/bluesky/mx-bluesky_1.6.4/mx-bluesky/src/mx_bluesky/hyperion/baton_handler.py", line 96, in collect
    current_visit = yield from _fetch_and_process_agamemnon_instruction(
                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/dls_sw/i03/software/bluesky/mx-bluesky_1.6.4/mx-bluesky/src/mx_bluesky/hyperion/baton_handler.py", line 160, in _fetch_and_process_agamemnon_instruction
    parameter_list: Sequence[BaseModel] = create_parameters_from_agamemnon()
                                          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/dls_sw/i03/software/bluesky/mx-bluesky_1.6.4/mx-bluesky/src/mx_bluesky/hyperion/external_interaction/agamemnon.py", line 80, in create_parameters_from_agamemnon
    agamemnon_instruction = _get_next_instruction(beamline_name)
                            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/dls_sw/i03/software/bluesky/mx-bluesky_1.6.4/mx-bluesky/src/mx_bluesky/hyperion/external_interaction/agamemnon.py", line 192, in _get_next_instruction
    return _get_parameters_from_url(get_agamemnon_url() + f"getnextcollect/{beamline}")
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/dls_sw/i03/software/bluesky/mx-bluesky_1.6.4/mx-bluesky/src/mx_bluesky/hyperion/external_interaction/agamemnon.py", line 168, in _get_parameters_from_url
    response.raise_for_status()
  File "/dls_sw/i03/software/bluesky/mx-bluesky_1.6.4/mx-bluesky/.venv/lib/python3.11/site-packages/requests/models.py", line 1026, in raise_for_status
    raise HTTPError(http_error_msg, response=self)
requests.exceptions.HTTPError: 500 Server Error: Internal Server Error for url: http://agamemnon.diamond.ac.uk/getnextcollect/i03
ESC[32m[I 260303 15:48:53.044 run_engine:189] Change state on <bluesky.run_engine.RunEngine object at 0x7fb681f4f8d0> from 'running' -> 'idle'ESC[0m

This seemed to then cause GDA to hang when handing over the baton to Hyperion due to

2026-03-03 16:04:49,449 ERROR gda.px.unattendeddatacollection.HyperionUDCRunner - Failed to start UDC
java.util.concurrent.TimeoutException: The requested value gda.px.unattendeddatacollection.HyperionUDCRunner$$Lambda/0x00000008011e7c58@10e21566 was not observed from the PV BL03I-CS-BATON-01:CURRENT_USER within the specified timeout of 5s
	at gda.epics.LazyPVFactory$LazyPV.waitForValue(LazyPVFactory.java:472)
	at gda.px.unattendeddatacollection.HyperionUDCRunner.startEventLoop(HyperionUDCRunner.java:132)
	at gda.px.unattendeddatacollection.UDCBatonHandler.handleBatonChanged(UDCBatonHandler.java:199)
	at gda.px.unattendeddatacollection.UDCBatonHandler.handleEventByType(UDCBatonHandler.java:179)
	at gda.observable.ObservableComponent.sendEventToObserver(ObservableComponent.java:144)

[rtuck99]

We get a lot of log spam from opentelemetry:

  "message": "Invalid type NoneType for attribute 'settle_time' value. Expected one of ['bool', 'str', 'bytes', 'int', 'float'] or a sequence of those types",

Raised

• Hyperion-blueapi generates log spam from opentelemetry #1664

[rtuck99]

Using hyperion in blueapi mode reports

File "/dls_sw/i03/software/bluesky/mx-bluesky_1.6.4/dodal/src/dodal/common/beamlines/beamline_utils.py", line 175, in get_path_provider
    return PATH_PROVIDER
           ^^^^^^^^^^^^^
NameError: name 'PATH_PROVIDER' is not defined

Fixed by

• Initialise i03 path provider, commissioning mode signal when beamline is instantiated dodal#1958
• Fixes for UDC Default State robot unload and panda PCAP path provider setting #1669

[rtuck99]

set_commissioning_signal() not invoked when blueapi mode is used leading to exceptions when no crystal found if commissioning mode is on

Fixed by

• Initialise i03 path provider, commissioning mode signal when beamline is instantiated dodal#1958

[rtuck99]

in blueapi mode, pin is not unloaded after handing back the baton because the supervisor does not read the return value from the load_centre_collect plan to get the current sample

• hyperion-blueapi does not do UDC cleanup after handing back the baton if we don't have a visit #1672

[rtuck99]

If the supervisor dies, passing baton to hyperion hangs GDA for ages. Also hyperion_restart for supervisor does not detect if supervisor failed to start

• Passing the baton when Hyperion not running causes a locked up client #1539