libjob: rabbit jobspec tests

[jameshcorbett]

WIP PR to address #6710. Several of the tests in t/t0022-jj-reader.t fail. Some of them because I didn't bother to spend the time to get them right, but others because now jj-reader can incorrectly count the nodes. A jobspec like this:

version: 1
resources:
  - type: slot
    count: 2
    label: rabbit
    with:
    - type: ssd
      count: 1
    - type: node
      exclusive: true
      count: 1
      with:
      - type: slot
        label: task
        count: 1
        with:
        - type: core
          count: 1

should give a nodecount of 2, but it currently gives a nodecount of 1 because the reader doesn't take into account the count of the higher-level slot entry. That could be easily fixed. However, it will probably get a wrong slot count but I don't know how the slot count is supposed to be used so I'm not as sure how to fix it.

@grondo @cmoussa1 I have to step away from development on this issue for a while to get back to SCR work, but I wanted to put it up so you could see what I was thinking.

I am wondering if flux-coral2 should drop the use of jobspec_update and just update the job-manager's version of the jobspec in-place like it used to, so that only the scheduler sees the updated version (I think that's how it works). That would let us punt on the issue until we have a better way to handle it. But if I'm wrong about only the scheduler seeing the updated jobspec, and in particular if the mf_priority plugin sees the in-place-updated jobspec, we'd still hit the same problem and that wouldn't save us anything.

[grondo]

But if I'm wrong about only the scheduler seeing the updated jobspec, and in particular if the mf_priority plugin sees the in-place-updated jobspec, we'd still hit the same problem and that wouldn't save us anything.

Yeah, I don't think it would help. jobspec-update events modify the job manager's copy of the jobspec in place, then future callbacks that unpack from jobspec key of plugins args reference the updated copy.

We'll need to think of some simple update to correctly count resources in jobspec V1+ or whatever this is. I actually didn't realize that jobspec could have more than one slot. We may need to sit down and devise some rules for a v2 jobspec that scheduling rabbits requires, encode that in an RFC, then update libjj using those rules. Maybe we can discuss this at the next meeting.

[jameshcorbett]

After discussion in the meeting yesterday, I think this is ready for review @grondo .

[grondo]

LGTM! Thanks!

[codecov]

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 83.80%. Comparing base (52f9060) to head (d159c71).
Report is 5 commits behind head on master.

Additional details and impacted files

@@            Coverage Diff             @@
##           master    #6727      +/-   ##
==========================================
- Coverage   83.82%   83.80%   -0.02%     
==========================================
  Files         534      534              
  Lines       89281    89279       -2     
==========================================
- Hits        74838    74823      -15     
- Misses      14443    14456      +13     

Files with missing lines	Coverage Δ
src/common/libjob/jj.c	93.24% <100.00%> (-0.18%)	⬇️

... and 11 files with indirect coverage changes

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.