Skip to content

worker: one-shot (non --loop) run exits 1 on success, so supervisors see every processed job as a failure #168

Description

@RealDiligent

Problem

main() in validator/src/eval_backend/worker.py runs a single pass like this:

if not args.loop:
    raise SystemExit(process_once(session_factory, settings))

process_once returns an integer that is a "did work" signal for the --loop sleep decision, not a process exit status:

  • returns 1 when it processed a job (or handled a bad job),
  • returns 0 when the queue was empty.

The --loop branch uses it correctly (if processed == 0: sleep(...)). But the one-shot branch passes it straight to SystemExit, so a run that successfully processes a job exits with code 1 — which any supervisor, cron wrapper, systemd oneshot, or CI step reads as a failure. The semantics are inverted: doing work looks like an error, and an empty queue (nothing done) looks like success.

Impact

Any deployment that invokes the worker one-shot (e.g. a timer/cron that runs python -m eval_backend.worker per tick instead of --loop) will treat every successful evaluation as a failed process — spurious alerts, retries, or a unit stuck in failed state.

Reproduction

$ python -m eval_backend.worker      # one queued job present
$ echo $?
1                                    # success, but exit code says failure

Suggested fix

A completed one-shot pass should exit 0 whether or not the queue had work; genuine failures already raise and exit nonzero. Replace the SystemExit(process_once(...)) with a call followed by raise SystemExit(0).

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions