[OpenMP][libomp][AArch64] Sporadic incorrect results with task #239958

Description

I ran into what looks like a memory ordering issue with OpenMP task dependencies on my Apple M5 (Homebrew LLVM 23.1.4).

The reproducer below is fairly simple: One task writes an array, and another task reads it. The dependency is expressed using depend(out: ...) and depend(in: ...), so I would expect the consumer to always see what the producer has written.

But occasionally it doesn't.

The problem seems to depend on whether the producer finishes shortly before or around the time the consumer task is created. The weird loop

// Create the consumer around the time the producer finishes.
for (volatile int k = 0; k < 42 * 42; ++k) {
}

is my attempt that with a certain probability > 0 in at least one run the consumer task gets scheduled the moment the producer task is just done.

Minimal reproducer

#include <cstdio>
#include <omp.h>

constexpr int P = 32;
constexpr int N = 64;

int a[P][N];
char token[P];

int
main()
{
    int errors = 0;

    for (int r = 1; r <= 500000; ++r) {
#pragma omp parallel num_threads(4)
#pragma omp single
        {
            for (int p = 0; p < P; ++p) {
#pragma omp task depend(out : token[p]) firstprivate(p, r)
                {
                    for (int i = 0; i < N; ++i) {
                        a[p][i] = r;
                    }
                }

                // Create the consumer around the time the producer finishes.
                for (volatile int k = 0; k < 42 * 42; ++k) {
                }

#pragma omp task depend(in : token[p]) firstprivate(p, r)
                {
                    for (int i = 0; i < N; ++i) {
                        if (a[p][i] != r) {
#pragma omp atomic
                            ++errors;
                            break;
                        }
                    }
                }
            }
        }
    }

    std::printf("Errors: %d\n", errors);
}

Expected output:

Errors: 0

Actual output: Occasionally, Errors is nonzero.

What I tried

I experimented a bit with adding a memory fence at the end of the producer task:

#include <atomic>

// At the end of the producer task:
std::atomic_thread_fence(std::memory_order_release);

With this change, I could no longer reproduce the error.

Interestingly, the same happened with both memory_order_acquire and memory_order_seq_cst.

I also tried replacing the fence with either of these:

asm volatile("" ::: "memory");
asm volatile("nop");

Neither helped. The errors still occurred.

The timing matters quite a bit. Doubling the delay from 42 * 42 to 42 * 42 * 2 also made the errors disappear in my test, even without a fence.

So apparently 42 * 42 is the magic number on my machine 😆

Of course, this doesn't prove that a missing memory barrier is the actual problem. But the fact that a hardware memory fence makes the errors disappear, while a compiler barrier or a nop does not, seems worth investigating.

Where I suspect the problem might be

I had a look at kmp_taskdeps.cpp.

One thing that caught my attention is the check of dep->dn.task before acquiring the dependency-node lock.

As far as I understand it, when the producer has already finished, the runtime can take a shorter path when registering the consumer's dependency.

I wonder whether that path always establishes the necessary memory ordering between the producer and consumer, particularly on ARM64.

This is just a guess. I haven't verified that this is where things go wrong.

Environment

• Hardware: Apple M5 (ARM64)
• OS: macOS (Darwin 25.6.0)
• Compiler: Homebrew Clang 23.1.4
• OpenMP runtime: Homebrew LLVM libomp 23.1.4
• Compiler flags: -fopenmp -std=c++17 -O2
• OpenMP threads: 4

$ clang++ --version
Homebrew clang version 23.1.4
Target: arm64-apple-darwin25.6.0
Thread model: posix
InstalledDir: /opt/homebrew/Cellar/llvm/23.1.4/bin
Configuration file: /opt/homebrew/Cellar/llvm/23.1.4/etc/clang/arm64-apple-darwin25.cfg

The reproducer is deliberately small and only needs OpenMP.

I originally encountered the problem while experimenting with an OpenMP task-based implementation of tiled LU and Cholesky factorizations. After seeing occasional incorrect results there, I tried to reduce the problem to something that doesn't involve any numerical algorithms.

Hopefully this reproducer is useful for tracking it down.