Job Not Found Error

[davramov]

I am encountering an issue where submit_job() raises a Job Not Found error, even though job IDs are generated, successfully scheduled, and run on Perlmutter. This complicates an automated workflow I am developing where we need to wait for the results of one task before starting the next one.

From my script job_controller.py:

try:
  logger.info("Submitting reconstruction job script to Perlmutter.")
  job = self.client.perlmutter.submit_job(job_script)
except Exception as e:
  logger.error(f"Failed to submit or complete reconstruction job: {e}")

Error log from the exception:

13:11:27.498 | INFO    | orchestration.flows.bl832.job_controller - Submitting reconstruction job script to Perlmutter.
13:12:03.894 | ERROR   | orchestration.flows.bl832.job_controller - Failed to submit or complete reconstruction job: Job not found: 33821565

It seems like this could arise from one of the SfApiErrors raised by the submit_job() function defined in sfapi_client/_sync/compute.py

[cjh1]

@davramov Sorry that you are running into this issue. I believe this is related to a change make to the underlying SF API, to do with caching of job status. While the issues related to this are resolved I will cut a release that passes the appropriate parameter when fetching the job status to ask SLURM directly, this should resolve the problem for you.

[cjh1]

@davramov 0.3.2 now contains the fix.

[davramov]

@cjh1 Thanks Chris, I appreciate it. Things seem to be working on my end now.