BMDB nightly: four models give different results run to run, and BIOMD0000000613's is not numerical

[jcschaff]

The BMDB nightly gate has failed every night since 2026-08-31 (last green: run
33298492096). Each failure is real in the sense that a result differed from the baseline —
but the same models keep changing their minds, so accepting a baseline clears the signal
for a night and not longer.

What the last seven nightlies actually did

model	behaviour
BIOMD0000001065	deviated (PASSED) on six of seven nights; matched its FAIL baseline once, on 09-06
BIOMD0000000547	MATH_GENERATION_FAILURE on 09-04/05/06/07; SOLVER_FAILURE on 08-31, 09-01, 09-03
BIOMD0000000613	MATH_GENERATION_FAILURE on 09-03/05/06/07; DIVIDE_BY_ZERO on 09-04
BIOMD0000000457	flipped PASSED → SOLVER_FAILURE on 09-05 only, and came back

Note the 08-30 pass and the 08-31 failure are on the same commit (f0bb594443), which
is what rules out a code change as the cause.

Two of them are numerically borderline, and that is expected

The recorded failures say so:

• 1065 — CV_REPTD_RHSFUNC_ERR ... pow(u,v) and u=-0.000000<0 and v=2.619640 not an integer, with EN_0_0 = -0.000000. A quantity sitting on zero, and pow of it is only
  defined if the last bit lands positive.
• 547 — At t = 8.09814e-18, mxstep steps taken before reaching tout. The solver
  stalling at a denormal-scale step.

Nothing about VCell is going to make those two deterministic; they are models balanced on a
knife edge, and the right treatment is probably to stop gating on them rather than to fix
them.

BIOMD0000000613 is different, and is the reason for this issue

It is not arithmetic. It alternates between two failures two phases apart, both about
the same initial condition:

	raised where	message
some nights	writing the solver input	divide by zero '(M_initConc / OC_initConc)'
other nights	math generation	Initial condition for variable 'M' references variable 'L'. Initial conditions cannot reference variables.

In one run the initial condition has been hoisted to the global parameter M_initConc
(SBMLImporter.hoistInitialConditionToGlobalParameter), which then divides by zero. In the
other it still holds a reference to species L, which Equation.checkInitialCondition
rejects outright.

That is the same model, the same input, and the same code producing two structurally
different MathDescriptions.

Correction (see the investigation comment below). I first read this as
ordering-dependent math generation. Running the model ten times in ten separate JVMs
gives the same answer every time, and identity hash codes differ on every JVM start —
so per-run hash ordering is not the mechanism. The variation is between environments, not
between runs, and the leading candidate is now the nightly's nightly-rebuilt Docker
image.

BIOMD0000000547 moving from SOLVER_FAILURE to MATH_GENERATION_FAILURE fits the same
shape: on some runs it reaches the solver, on others math generation refuses it first, with
the same checkInitialCondition message.

Suggested next steps

1. Find the ordering dependence. Running 613 and 547 through math generation repeatedly
   in one JVM, and across JVMs, should reproduce it without needing the nightly.
2. Decide what to do about the knife-edge cases. The baseline can say PASS, FAIL or
   SKIP, and none of those means "either outcome is fine". Options: mark 1065 and 547
   SKIP with the reason recorded, the way the workflow's 11 slow models already are; or
   give the baseline a way to accept a set of outcomes.

Today's three deviations are accepted in #2078 so the signal is clear again, with the
reasoning recorded in the commit.

[jcschaff]

things to think about

• should we use ordered collections to avoid the nondeterminism (even when run single threaded)
• should we look for multithreaded influences (are the nightly tests run in parallel on multiple cores)a
• should we just fix the underlying problem(s) so that the ordering doesn't matter anyway.

[jcschaff]

Chased all three. The short version: (2) is ruled out, (1) is weaker than I claimed, and
the evidence now points somewhere I had not considered. Details, including the negative
results, because those are what moved the answer.

(2) Multithreaded influences — ruled out

• NightlyBMDB_CLI.yml runs a 20-way matrix, but within a set the models are executed
  one at a time in a for file in … loop, each as its own docker run execute-omex.
  Separate container, separate JVM, one model each.
• vcell-cli contains no ExecutorService, CompletableFuture, parallelStream, or
  new Thread — grep finds nothing.

So each result is produced single-threaded in a fresh JVM, and models cannot perturb each
other.

(1) Ordered collections — a real smell, but not what is happening here

Worth separating two things, because they behave differently:

• HashMap<String, …> iteration order is deterministic across JVM runs — String.hashCode
  is specified by the language.
• HashMap<SomeDomainObject, …> where the key does not override hashCode uses the
  identity hash, which does vary per JVM run.

cbit.vcell.math.Variable overrides neither hashCode nor equals, and the mapping layer
does hold maps keyed on it — DiffEquMathMapping:319-320, HashMap<VolVariable, …> and
HashMap<Variable, …>. But those two are only ever put/get, never iterated, so their
order cannot escape. (The package overall is 36 hash-ordered vs 43 ordered collections, so
there is tidying to do, but it is not obviously load-bearing here.)

The experiment that settles it. I pulled BIOMD0000000613.omex from the pinned corpus
and ran it through execute-omex ten times, in ten separate JVMs, on this machine:

run 1..10: FAILED DIVIDE_BY_ZERO      (all ten identical)

Identity hash codes differ on every JVM start, so if hash-order were driving the fork I
would expect this to vary. It does not. Within one machine it is perfectly
deterministic — and it lands on the old baseline value, not the new one.

That is the result that undermines what I wrote in the issue above, and I have corrected the
description accordingly.

Where the fork actually is

Both outcomes come from one decision — whether SBMLImporter hoists M's initial condition:

INFO SBMLImporter:1098 - hoisted initial condition of 'M' to global parameter 'M_initConc'

adjustExpression only hoists when the symbol fails to resolve in the current namescope
(infix().contains(UNRESOLVED_PREFIX)). If it hoists, M's initial condition becomes the
parameter M_initConc, math generation is happy, and the run dies later on
divide by zero '(M_initConc / OC_initConc)'. If it does not hoist, the initial condition
still references species L, and Equation.checkInitialCondition rejects it. Same model,
one branch, two failure modes.

Hoisting landed 2026-08-17 (6f4c62345c), a fortnight before the failures started, so
its arrival is not the trigger either.

(3) So what is varying?

Not the input: the corpus is pinned to sys-bio/temp-biomodels@6a09daf. Not the code: the
08-30 pass and the 08-31 failure are on the same commit f0bb594443. Not threads. Not,
apparently, per-JVM hash order.

What is left is the environment, and there is a specific candidate: the nightly
rebuilds its Docker image every night, and that image installs packages with apt. We
watched exactly that kind of drift break a release build today — bullseye-security
expiring mid-release (#2075) — so "same Dockerfile, different resulting image" is not
hypothetical here.

Two cheap next steps, in order:

1. Diff the image between a passing and a failing night — package versions, JDK build,
   libm. If something moved on 2026-08-30/31, that is the answer.
2. If the image is identical, compare CI runner hardware. Two of the four flapping cases
   sit on a floating-point knife edge (pow(u,v) and u=-0.000000<0;
   At t = 8.09814e-18, mxstep steps taken), which is exactly what differs between
   microarchitectures.

On your (3) — fixing the underlying problem so ordering does not matter — I think that is
right regardless of the outcome above, but it is a different fix for each case. For 613 the
real question is what VCell should do with an SBML initial assignment that references
another species: hoisting makes it representable and then it divides by zero, so neither
branch is a good answer, and picking one deterministically would just make the failure
stable rather than correct.

[jcschaff]

Chased the JSCL timeout. The mechanism is exactly as you described, and it is a real
latent hazard — but it is not what decides these two models. Both halves are worth
recording.

The mechanism is there, and it is worse than a timeout

jscl.math.Expression.timeoutMS is a ThreadLocal<Long> holding an absolute wall-clock
deadline (System.currentTimeMillis() + budget), checked inside Factorization and the
Groebner Standard basis code, which throw ExpressionTimeoutException.

What makes it a nondeterminism source rather than just a timeout is the default and the
fallback, in cbit/vcell/parser/Expression.java:

public Expression simplifyJSCL() throws ExpressionException {
    return simplifyJSCL(100, false);        // 100 ms of wall clock
}
...
} catch (jscl.math.Expression.ExpressionTimeoutException e) {
    if (bFailOnTimeout) { throw ... ; }
    else {
        logger.debug("timeout simplify using JSCL, reverting to flatten(): " + e.getMessage());
        return flattenSafe();               // a DIFFERENT expression
    }
}

So on a fast or idle machine you get the simplified expression; on a slow or loaded one you
get the flattened one, and downstream sees something structurally different. The fallback is
logger.debug, so at normal levels it happens silently. bFailOnTimeout is false for
every caller that uses the no-arg form, which is most of them — SedMLImporter alone calls
it in six places.

That is a genuine "same code, same input, different answer, depending on how busy the
machine was" and it should be fixed regardless of this issue.

But it does not explain 547 or 613

I made the budget settable locally and drove it down, which simulates an arbitrarily slow
machine:

model	budget 100 ms	20 ms	5 ms	1 ms
BIOMD0000000613	DIVIDE_BY_ZERO	DIVIDE_BY_ZERO	DIVIDE_BY_ZERO	DIVIDE_BY_ZERO
BIOMD0000000547	MATH_GENERATION_FAILURE	—	—	MATH_GENERATION_FAILURE

No movement. And for 613 there is no JSCL activity in the run's own debug log at all. The
deadline is only checked inside factorization and Groebner, so an expression that never
reaches those never notices the budget — which is presumably the case here.

Worth noting from the same run: 547 gives MATH_GENERATION_FAILURE on my machine, i.e.
the new baseline value, while 613 gives DIVIDE_BY_ZERO 10/10, the old one. So this
machine agrees with CI on one and disagrees on the other, which is consistent with an
environmental difference rather than anything per-run.

Where that leaves it

• Fix the JSCL budget anyway. A wall-clock deadline that silently changes the answer is
  a bug independent of these two models, and it is a strong candidate for the other
  flappers — BIOMD0000001065 and BIOMD0000000457 both flip in ways that a different
  simplification would explain. Making it deterministic (an operation or step count rather
  than elapsed milliseconds) would remove the whole class. Failing that, bFailOnTimeout
  should probably default to true so it is loud rather than silent.
• 613's fork is still unexplained, and the next probe is the one from my previous
  comment: diff the nightly's Docker image between a passing and failing night, since the
  image is rebuilt every night while the corpus and the commit are pinned.