PLT-3535 MarloweBulkSync Server

[jhbertra]

• Added new bulk sync protocol.
• Added a server implementation to marlowe-sync
• Rewrote marlowe-finder to use bulk sync.

Pre-submit checklist:

• Branch
  • Tests are provided (if possible)
  • Test report is updated (if relevant)
  • Commit sequence broadly makes sense
  • Key commits have useful messages
  • Relevant tickets are mentioned in commit messages
  • Formatting, PNG optimization, etc. are updated
• PR
  • Self-reviewed the diff
  • Useful pull request description
    • Review required
    • Substantial changes to code, test, or documentation
    • Change made to Marlowe validator (@bwbush and @palas must be included as reviewers)
    • Review not required
    • Minor changes to non-critical code, documentation, nix derivations, configuration files, or scripts
    • Formatting, spelling, grammar, or reorganization
  • Reviewer requested

[bwbush]

Won't the operables, deployment, and compose derivations need updating, too?

In fact, should we add this reminder to the PR template?

[jhbertra]

That might be a good way to deal with this. You're right we often forget this. I can't really think of a good way to automate this check.

[bwbush]

It's great to finally have this API.

However, marlowe-finder does not report about 30% of the contracts on preprod:

$ marlowe-finder --end-at-tip \
| jq -r 'select(.NewContract) | .NewContract.fields.contractId \
| (.txId + "#" + (.txIx | tostring))' \
| wc -l
5123

$ psql preprod
preprod=> select count(*) from marlowe.createtxout;
 count 
-------
  7055
(1 row)

[bwbush]

The situation is more serious on preview:

$ marlowe-finder --end-at-tip \
| jq -r 'select(.NewContract) | .NewContract.fields.contractId | (.txId + "#" + (.txIx | tostring))' \
| wc -l
15868

$ psql preview
preview=> select count(*) from marlowe.createtxout;
 count 
-------
 52969
(1 row)

[jhbertra]

Thanks for performing that test, I'll investigate!

Issue addressed

[bwbush]

@jhbertra, I reran the tests on the new commit and results are correct both on preprod and preview.