Vreman closure reads uninitialized halo cells and crashes Debug builds with SIGFPE

[madDipanjan]

Summary

The Vreman branch of closure() reads uninitialized ekm halo cells before boundary conditions initialize them. On ARCHER2 with the Cray Debug flags, this produces a reproducible SIGFPE before the first timestep completes.

The same whole-array operations are present on current origin/master and the performance branch.

Reproduction

Case: simulations/benchmark-standard-900

• neutral urban IBM LES
• Vreman SGS (lvreman = .true.)
• 256×256×128 grid
• 64 MPI ranks
• ARCHER2 standard nodes (AMD EPYC 7742)
• PrgEnv-cray/8.4.0, cce/16.0.1, cray-mpich/8.1.27
• Debug flags: -s real64 -N 1023 -G0 -O0 -ei -R bcdps -m 0 -K trap=divz,inv,ovf -em -J.

Build with:

./tools/build_executable.sh archer debug

Run the standard benchmark with 64 ranks. The crash occurs before the first normal timestep output. It reproduced in two separate Slurm jobs on different nodes.

Backtrace from the generated core file:

#0 closure () at src/modsubgrid.f90:357
   ekh(:,:,:) = ekm(:,:,:)*prandtli
#1 subgrid () at src/modsubgrid.f90:139
#2 udales () at src/program.f90:147
   call subgrid

Rebuilding Debug without Cray's -ei uninitialized-value poisoning still crashes at the same statement under -K trap=divz,inv,ovf. This is therefore a real invalid floating-point operation on garbage data, not an -ei-only false positive.

Root cause

In the lvreman branch, ekm is assigned only over the physical/interior range:

do k = kb, ke
  do j = jb, je
    do i = ib, ie
      ...
      ekm(i,j,k) = ...
    end do
  end do
end do

The subsequent statements operate on the complete allocated arrays, including halo cells that have not yet been initialized:

ekh(:,:,:) = ekm(:,:,:)*prandtli
ekm(:,:,:) = ekm(:,:,:) + numol
ekh(:,:,:) = ekh(:,:,:) + numol*prandtlmoli

closurebc, which sets/copies the closure boundary and halo values, is called only later at the end of closure(). Release builds can silently survive because the invalid halo values are overwritten by closurebc, but reading them is undefined and Debug trapping exposes it.

Proposed fix

Fold the diffusivity and molecular additions into an interior i,j,k loop (or into the existing Vreman loop) and do not touch halo cells before closurebc:

do k = kb, ke
  do j = jb, je
    do i = ib, ie
      ekh(i,j,k) = ekm(i,j,k)*prandtli + numol*prandtlmoli
      ekm(i,j,k) = ekm(i,j,k) + numol
    end do
  end do
end do

This also removes three whole-array passes over unused halo cells.

Moving the operations after closurebc is not equivalent without carefully preserving the boundary-condition semantics, so restricting the operations to the already-defined interior is the safer change.

Acceptance criteria

• ARCHER2 Debug build completes the Vreman standard benchmark without SIGFPE.
• Release and Debug builds both compile.
• Release behavior remains physics-equivalent (restart comparison at compiler-roundoff tolerance, not necessarily bitwise).
• Other closure branches are unchanged.
• A regression test or focused Debug smoke case exercises lvreman through the first timestep.

[madDipanjan]

Implemented locally on branch performance by restricting the Vreman molecular-viscosity/diffusivity update to the initialized interior domain (ib:ie, jb:je, kb:ke):

do k = kb,ke
  do j = jb,je
    do i = ib,ie
      ekh(i,j,k) = ekm(i,j,k)*prandtli + numol*prandtlmoli
      ekm(i,j,k) = ekm(i,j,k) + numol
    end do
  end do
end do

This preserves the existing interior calculation while leaving halo population to the immediately following closurebc calls. It also replaces three full-array passes with one fused interior loop.

Validation on ARCHER2:

• Debug build: passed
• Release build: passed
• Focused Vreman Debug smoke test: Slurm job 14468455, one standard node / 64 MPI ranks
• Job state: COMPLETED, exit code 0:0
• Advanced through 11 steps (t = 0.05 to 0.55 s); the unfixed Debug executable failed during the first Vreman closure call
• slurm.err: empty
• No SIGFPE, floating-point exception, NaN, or solver error in the output
• Normal completion marker: TOTAL CPU time = 18.897947019000185

The repository-mandated Python-library stream could not be run because the canonical tools/python/.venv/bin/python environment is absent on this checkout; no alternate environment was created as part of this focused Fortran fix.