PoC to detect where a flaky test diverge

[socketopp]

Good evening, ava community!

I have written a tool divergent (FIRST DRAFT!!) that run a specific ava test case until it both passes and fails, then I use njstrace (Instrumentation and tracing tool) to analyse the trace of both runs and provide the user with useful information where the test first diverge from a passing and a failing run.

I am not a dedicated ava user, however, I was intrigued to use instrumentation tool along a test framework. However, it was not possible to get traces of njtrace with Mocha or Jest, so I figured I'll try ava. To my astonishment, it worked!

Here is an output of the program.

What do you think? I was wondering if I should make this a npm module or just keep the code in a stand alone repository, or maybe it would be nice to incorporate it in ava? Who knows, let me know what you think.

All input is much obliged!

$ node src/index.js foo
Running testcase iteration 1
Running testcase iteration 2
Running testcase iteration 3
Running testcase iteration 4
Running testcase iteration 5
Analyzing results...
Successful run.
{
  pass: {
    functionName: 'getRandomZeroOrOne',
    file: 'C:\\divergent\\src\\randomNumberGenerator.js'
,
    divergent: 10
  },
  fail: {
    functionName: 'getRandomZeroOrOne',
    file: 'C:\\divergent\\src\\randomNumberGenerator.js'
,
    divergent: 12
  }
}

Kind regards,
socketopp

[novemberborn]

Neat!

How do you run the test cases to see which are passing and failing? Are you using t.try()?

Having had a quick look at njstrace, does it work with ESM? I suspect it's currently hooking into the CJS loading?

[socketopp]

How do you run the test cases to see which are passing and failing? Are you using t.try()?

I have a test case callled 'foo' in flaky.test.js

test('foo', async ava => {
  const randomBoolValue = getRandomZeroOrOne();
  ava.is(true, randomBoolValue);
});

When running ava, I specify the test case with --match='foo'

Having had a quick look at njstrace, does it work with ESM? I suspect it's currently hooking into the CJS loading?

I would believe it only works with ESM based on what they're writing in the docs:

_node.js Module.compile() method, this method is called whenever a module is being "required"

I have only tried it with ESM, not CJS as of yet.

[novemberborn]

I have a test case callled 'foo' in flaky.test.js

test('foo', async ava => {
  const randomBoolValue = getRandomZeroOrOne();
  ava.is(true, randomBoolValue);
});

When running ava, I specify the test case with --match='foo'

Have you looked at t.try()? It lets you perform assertions within a test, and you can decide to discard failures.

I would believe it only works with ESM based on what they're writing in the docs:

_node.js Module.compile() method, this method is called whenever a module is being "required"

I have only tried it with ESM, not CJS as of yet.

It's CJS that requires modules. ESM has a different (experimental) hook mechanism.

[socketopp]

Have you looked at t.try()? It lets you perform assertions within a test, and you can decide to discard failures.

I haven't tried using t.try(). In what sense would I benefit from using t.try in the test?

It's CJS that requires modules. ESM has a different (experimental) hook mechanism.

I'm sorry, I mixed those two up. Of course, it's CommonJS I was referring to, which works with nodejs.

[novemberborn]

Have you looked at t.try()? It lets you perform assertions within a test, and you can decide to discard failures.

I haven't tried using t.try(). In what sense would I benefit from using t.try in the test?

It would let you do the divergence within a test run — so you don't have to re-run AVA matching a specific test.