Skip to content

Parallel composite actions race on shared runner state #4789

Open

Description

@ronocod

Describe the bug

I’m seeing failures when composite actions overlap through background: true or parallel:. The clearest failure seems to be two composite children processing GITHUB_STATE at the same time: the runner reports that a concurrent update corrupted a non-thread-safe collection.

In the last two days we had 52 failed CI jobs with Collection was modified; enumeration operation may not execute., plus one job with the explicit state corruption below. Some failures happen before the first child action starts; others happen inside a nested composite or after its commands have succeeded. Most pass when we rerun them.

To reproduce

I don’t have a standalone repro yet. The failing Docker build job looks roughly like this:

- name: Setup PostgreSQL Replica with Redis
  id: dbs
  background: true
  uses: ./.github/actions/steps/setup-dbs
  with:
    pg-version: "18.6-alpine"

- name: Setup frontend workspace
  id: workspace
  background: true
  uses: ./.github/actions/setup-workspace

The database composite runs aws-actions/amazon-ecr-login. The workspace composite contains another composite that runs actions/cache. Both child actions finish at almost the same instant and fail while the runner processes their state files. Subsequent steps and post actions also fail against the corrupted collection.

Expected behavior

Concurrent composite actions should be able to save their state without corrupting shared runner data or failing unrelated steps.

Runner version and platform

  • Observed on runner 2.337.0, on Depot Linux runners using Ubuntu 22.04 and 24.04.
  • I also checked the 2.338.0 source: the state handler and ordinary dictionary are still present. I haven’t reproduced the failure on that version.

Job log output

These lines are from one job, with unrelated output omitted:

2026-10-16T09:37:25.0566526Z ##[error]Unable to process file command 'state' successfully.
2026-10-16T09:37:25.0568602Z ##[error]Unable to process file command 'state' successfully.
2026-10-16T09:37:25.0568687Z ##[error]Operations that change non-concurrent collections must have exclusive access. A concurrent update was performed on this collection and corrupted its state. The collection's state is no longer correct.
2026-10-16T09:37:25.0581819Z ##[error]Operations that change non-concurrent collections must have exclusive access. A concurrent update was performed on this collection and corrupted its state. The collection's state is no longer correct.
2026-10-16T09:37:25.0667610Z ##[end-action id=workspace.__self.__actions_cache;outcome=failure;conclusion=failure;duration_ms=670]
2026-10-16T09:37:25.0675370Z ##[end-action id=dbs.login-ecr;outcome=failure;conclusion=failure;duration_ms=337]

Suspected source

SaveStateFileCommand.ProcessCommand handles embedded actions by checking and writing context.Root.EmbeddedIntraActionState without synchronization. That shared collection is a Dictionary<Guid, Dictionary<string, string>>, initialized once for the job. Distinct child IDs still write to the same outer dictionary. This matches the explicit concurrent-update failure above.

Activity

  1. erezcor commented on Nov 13, 2026

    @erezcor

    We hit this twice this week on 2.338.2 (the current release), on self-hosted Linux runners: 2026-11-10 and 2026-11-12. The same step passed on other attempts.

    Our failures come at a different point from the GITHUB_STATE case in the report. The step fails while the runner prepares a nested local composite, before that composite runs any step.

    The job starts 9–18 background: true steps at once. Each one uses the same local composite, which nests a second local composite. The nested composite contains only run: steps and uses no remote actions.

    - name: Backend unit
      background: true
      uses: ./.github/actions/vitest-suite
    
    # ./.github/actions/vitest-suite/action.yml
    runs:
      using: composite
      steps:
        - run: ...
        - run: ...
        - name: Run the suite
          uses: ./.github/actions/sharded-run

    The outer composite's two run: steps succeed. The nested uses: step then fails about 490 ms after it starts:

    2026-11-12T21:39:47.1887664Z ##[start-action display=Run the suite;id=__self_4.__self]
    2026-11-12T21:39:47.6596059Z Prepare all required actions
    2026-11-12T21:39:47.6597086Z Getting action download info
    2026-11-12T21:39:47.6728549Z ##[error]Collection was modified; enumeration operation may not execute.
    2026-11-12T21:39:47.6809318Z ##[end-action id=__self_4.__self;outcome=failure;conclusion=failure;duration_ms=492]
    

    On 2026-11-10 a different background step failed the same way (duration_ms=485), with 18 background steps started together. Background step output is interleaved in the log, so the Prepare all required actions lines may belong to a sibling step. In both cases the failing step never reached a child action, so no state file of its own had been written yet.

  2. erezcor commented on Nov 16, 2026

    @erezcor

    Follow-up on our failures above: this is the race #4799 fixes.

    In 2.338.2, CreateEmbeddedChild gives each composite step the job root's GitHubContext, and CompositeActionHandler then calls ShallowCopy() on it. While background steps run, the job's step loop keeps writing github.action to that same root context on another thread, so the copy's enumeration throws Collection was modified. That fits our job, which starts 9–18 background composites at once.

    An L0 test reproduces it: from a background step, call CreateEmbeddedChild and ShallowCopy() the child's github context in a loop while another task calls jobContext.SetGitHubContext("action", ...). It throws InvalidOperationException on stock v2.338.2 and passes with #4799's change applied unconditionally.

    #4799 isn't in v2.338.2, and it's behind the actions_fix_embedded_github_context flag. Could it ship in the next release with the flag turned on, or the fix made unconditional? Until then, any background step that uses a composite action can fail at random.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't working

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions