fix(note-transport): recover cursors after database recreation

[kkovaacs]

Summary

Implements cursor reset described in 0xMiden/note-transport-service#97.

We now detect if the client's trying to use an invalid (too high) cursor value and reset the cursor to 0 in the response.

Changelog

[[entry]]
scope       = "note-transport"
impact      = "breaking"
description = "Reset fetch_notes cursor stranded above the seq high-water."

---

_{Stack created with GitHub Stacks CLI • Give Feedback 💬}

[igamigo]

I mentioned this on one of the Slack group conversations but never commented on the issue (apologies for this, definitely should've left a comment):

I don't recall what the idea was behind 0xMiden/note-transport-service#97 (I don't think I was involved in that conversation), but I would vote against merging that. From reading the description alone I'd have to ask:

• What happens when the DB gets reset but a client does not realize? for example, a client synced until seq 100. Then they don't sync for a while but the DB grows above seq 100. When they sync, they'll never know the DB got reset because the check does not trigger, and so they will miss 100 seq's worth of entries.
• Why is the fact that the DB can be wiped from one minute to the next one fine? I could be wrong but I don't think this was a possibility before so is this a new constraint? Applications need to work with these invariants, so:
  • If a DB can get wiped, we need a better protocol that solves the scenario I outlined above
  • If we decide DBs will never get wiped (there will be backups, multiple instances, etc.), then this solution is incorrect

Maybe these concerns are no longer applicable for some reason (I have not followed recent changes here, like the ones in the base branch of this PR), but just wanted to comment on it here just in case.

[kkovaacs]

I mentioned this on one of the Slack group conversations but never commented on the issue (apologies for this, definitely should've left a comment):

I don't recall what the idea was behind 0xMiden/note-transport-service#97 (I don't think I was involved in that conversation), but I would vote against merging that. From reading the description alone I'd have to ask:

• What happens when the DB gets reset but a client does not realize? for example, a client synced until seq 100. Then they don't sync for a while but the DB grows above seq 100. When they sync, they'll never know the DB got reset because the check does not trigger, and so they will miss 100 seq's worth of entries.

• Why is the fact that the DB can be wiped from one minute to the next one fine? I could be wrong but I don't think this was a possibility before so is this a new constraint? Applications need to work with these invariants, so:

  • If a DB can get wiped, we need a better protocol that solves the scenario I outlined above
  • If we decide DBs will never get wiped (there will be backups, multiple instances, etc.), then this solution is incorrect

Maybe these concerns are no longer applicable for some reason (I have not followed recent changes here, like the ones in the base branch of this PR), but just wanted to comment on it here just in case.

I think the primary issue that after a catastrophic event (which shouldn't be "normal") client behavior is just bad. Should we lose the database clients will be eventually "stuck" receiving new notes because the cursor they're persistently storing is way ahead in the future (and invalid).

Frankly, I think we should probably make cursor a bit more structured: in addition to the sequence number we should maybe have a "generation count" in the cursor so that the note transport service can detect if a cursor submitted by a client is invalid?

[igamigo]

I think the primary issue that after a catastrophic event (which shouldn't be "normal") client behavior is just bad. Should we lose the database clients will be eventually "stuck" receiving new notes because the cursor they're persistently storing is way ahead in the future (and invalid).

I agree with this, but on worst case (e.g., the catastrophic event) a client can at least always sync from 0.

Frankly, I think we should probably make cursor a bit more structured: in addition to the sequence number we should maybe have a "generation count" in the cursor so that the note transport service can detect if a cursor submitted by a client is invalid?

I realize now that I also didn't copy the second message in my reply 🫠, but I suggested this as well:

If we want an overall more complete solution we could have something like a "gen_id" or similar tied to the DB or cursor (we can pack it in the same response), that communicates to clients whether they are syncing against hte same version of the DB as they were before. If it's not, they would sync from the starting point.

[kkovaacs]

Closing this for now, will open an new PR implementing a generation counter instead.