lightning-liquidity persistence: Add serialization logic for services and event queue
#4059
Conversation
|
👋 Thanks for assigning @TheBlueMatt as a reviewer! |
tnull
force-pushed
the
2025-01-liquidity-persistence
branch
2 times, most recently
from
124211d to
26f3ce3
Compare
tnull
force-pushed
the
2025-01-liquidity-persistence
branch
4 times, most recently
from
a98dff6 to
d630c4e
Compare
Codecov Report❌ Patch coverage is Additional details and impacted files@@ Coverage Diff @@
## main #4059 +/- ##
==========================================
- Coverage 88.39% 88.23% -0.16%
==========================================
Files 177 179 +2
Lines 131314 131833 +519
Branches 131314 131833 +519
==========================================
+ Hits 116069 116321 +252
- Misses 12596 12852 +256
- Partials 2649 2660 +11
Flags with carried forward coverage won't be shown. Click here to find out more. ☔ View full report in Codecov by Sentry. 🚀 New features to boost your workflow:
|
tnull
force-pushed
the
2025-01-liquidity-persistence
branch
from
d630c4e to
70118e7
Compare
tnull
force-pushed
the
2025-01-liquidity-persistence
branch
from
70118e7 to
dd43edc
Compare
|
this all LGTM. I have a small concern: maybe I’m being a little paranoid, but read_lsps2_service_peer_states and read_lsps5_service_peer_states pull every entry from the KVStore into memory with no limit. That could lead to unbounded state, exhausting memory and crash. Maybe we can add a limit on how many entries we load into memory to protect against this dos? not sure how realistic this is though. maybe an attacker could have access to or share the same storage with the victim, and they could dump effectively infinite data onto disk. in this scenario, probably the victim would be vulnerable to other attacks too, but still.. |
Reading state from disk (currently) happens on startup only, so crashing wouldn't be the worst thing, we would simply fail to start up properly. Some even argue that we need to panic if we hit any IO errors at this point to escalate to an operator. We could add some safeguard/upper bound, but I'm honestly not sure what it would protect against.
Heh, well, if we assume the attacker has write access to our |
|
🔔 1st Reminder Hey @TheBlueMatt! This PR has been waiting for your review. |
tnull
force-pushed
the
2025-01-liquidity-persistence
branch
from
dd43edc to
f73146b
Compare
| @@ -45,6 +46,10 @@ pub struct LSPS2GetInfoRequest { | |||
| pub token: Option<String>, | |||
| } | |||
|
|
|||
| impl_writeable_tlv_based!(LSPS2GetInfoRequest, { | |||
There was a problem hiding this comment.
Do we really want to have two ways to serialize all these types? Wouldn't it make more sense to just use the serde serialization we already have and wrap that so that it can't all be misused?
There was a problem hiding this comment.
Yes, I think I'd be in favor of using TLV serialization for our own persistence.
Note that the compat guarantees of LSPS0/the JSON/serde format might not exactly match what we require in LDK, and our Rust representation might also diverge from the pure JSON impl. On top of that JSON is of course much less efficient.
There was a problem hiding this comment.
Hmm, is there some easy way to avoid exposing that in the public API, then? Maybe a wrapper struct oe extension trait for serialization somehow? Seems like kinda a footgun for users, I think?
There was a problem hiding this comment.
Hmm, is there some easy way to avoid exposing that in the public API, then? Maybe a wrapper struct oe extension trait for serialization somehow? Seems like kinda a footgun for users, I think?
Not quite sure I understand the footgun? You mean because these types then have Writeable as well as Serialize implementations on them and users might wrongly pick Writeable when they use the types independently from/outside of lightning-liquidity?
There was a problem hiding this comment.
Sure, for example. Someone who uses serde presumably has some wrapper that serde-writes Writeable structs and suddenly their code could read/compile totally fine and be reading the wrong kind of thing. If they have some less-used codepaths (eg writing Events before they process them and then removing them again after) they might not find immediately.
There was a problem hiding this comment.
Sure, for example. Someone who uses serde presumably has some wrapper that serde-writes
Writeablestructs and suddenly their code could read/compile totally fine and be reading the wrong kind of thing.
I'm confused - Writeable is an LDK concept not connected to serde? Do you mean Serialize? But that also has completely separate API? So how would they trip up? You mean they'd confuse Writeable and Serialize?
| ) -> Pin<Box<dyn Future<Output = Result<(), lightning::io::Error>> + Send>> { | ||
| let outer_state_lock = self.per_peer_state.read().unwrap(); | ||
| let mut futures = Vec::new(); | ||
| for (counterparty_node_id, peer_state) in outer_state_lock.iter() { |
There was a problem hiding this comment.
Huh? Why would we ever want to do a single huge persist pass and write every peer's state at once? Shouldn't we be doing this iteratively? Same applies in the LSPS2 service.
There was a problem hiding this comment.
Yes, only persisting what's needed/changed will be part of the next PR as it ties into how we wake the BP to drive persistence (cf. "Avoid re-persisting peer states if no changes happened (needs_persist flag everywhere)" bullet over at #4058 (comment)).
There was a problem hiding this comment.
I'm confused why we're adding this method then? If its going to be removed in the next PR in the series we should just not add it in the first place.
There was a problem hiding this comment.
No, it's not gonna be removed, but extended: PeerState (here as well as in LSPS2) will gain a dirty/needs_persist flag and we'd simply skip persisting any entries that haven't been changed since the last persistence round.
There was a problem hiding this comment.
That seems like a weird design if we need to persist something immediately while its being operated on - we have the node in question why walk a whole peer list? Can you put up the followup code so we can see how its going to be used? Given this PR is mostly boilerplate I honestly wouldn't mind it being a bit bigger, as long as the code isn't too crazy.
There was a problem hiding this comment.
That seems like a weird design if we need to persist something immediately while its being operated on - we have the node in question why walk a whole peer list?
Yes, this is why persist_peer_state is a separate method - for inline persistence where we already hold the lock to the peer state we'd just call that. For the general/eventual persistence the background processor task calls LiquidityManager::persist which calls through to the respective LSPS*ServiceHandler::persist methods which then only persists the entries marked dirty since the last persistence round.
Can you put up the followup code so we can see how its going to be used? Given this PR is mostly boilerplate I honestly wouldn't mind it being a bit bigger, as long as the code isn't too crazy.
Sure will do as soon as it's ready an in a coherent state, although I had hoped to land this PR this week.
tnull
force-pushed
the
2025-01-liquidity-persistence
branch
from
f73146b to
2971982
Compare
|
Rebased to address minor conflict. |
We add `KVStore` to `LiquidityManager`, which will be used in the next commits. We also add a `LiquidityManagerSync` wrapper that wraps a the `LiquidityManager` interface which will soon become async due to usage of the async `KVStore`.
tnull
force-pushed
the
2025-01-liquidity-persistence
branch
from
2971982 to
93c234f
Compare
![graphite-app[bot]](https://avatars.githubusercontent.com/in/158384?s=60&v=4){kind=small}
| let mut peer_by_intercept_scid = new_hash_map(); | ||
| let mut peer_by_channel_id = new_hash_map(); | ||
| for (node_id, peer_state) in peer_states.iter() { | ||
| for (intercept_scid, _) in peer_state.outbound_channels_by_intercept_scid.iter() { | ||
| let res = peer_by_intercept_scid.insert(*intercept_scid, *node_id); | ||
| debug_assert!(res.is_none(), "Intercept SCIDs should never collide"); | ||
| } | ||
|
|
||
| for (channel_id, _) in peer_state.intercept_scid_by_channel_id.iter() { | ||
| let res = peer_by_channel_id.insert(*channel_id, *node_id); | ||
| debug_assert!(res.is_none(), "Channel IDs should never collide"); | ||
| } | ||
| } |
There was a problem hiding this comment.
The code uses debug_assert! to check for collisions in intercept_scid and channel_id mappings during state reconstruction. Since these assertions are removed in release builds, any actual collisions in persisted data (whether from corruption or malicious input) would result in silent overwrites of map entries. This could lead to incorrect routing behavior or state corruption.
Consider replacing these debug assertions with proper error handling that would be active in all build configurations. For example:
if peer_by_intercept_scid.insert(*intercept_scid, *node_id).is_some() {
return Err(io::Error::new(
io::ErrorKind::InvalidData,
"Corrupted state: Intercept SCID collision detected"
));
}This would ensure the integrity of the reconstructed state even in production environments.
| let mut peer_by_intercept_scid = new_hash_map(); | |
| let mut peer_by_channel_id = new_hash_map(); | |
| for (node_id, peer_state) in peer_states.iter() { | |
| for (intercept_scid, _) in peer_state.outbound_channels_by_intercept_scid.iter() { | |
| let res = peer_by_intercept_scid.insert(*intercept_scid, *node_id); | |
| debug_assert!(res.is_none(), "Intercept SCIDs should never collide"); | |
| } | |
| for (channel_id, _) in peer_state.intercept_scid_by_channel_id.iter() { | |
| let res = peer_by_channel_id.insert(*channel_id, *node_id); | |
| debug_assert!(res.is_none(), "Channel IDs should never collide"); | |
| } | |
| } | |
| let mut peer_by_intercept_scid = new_hash_map(); | |
| let mut peer_by_channel_id = new_hash_map(); | |
| for (node_id, peer_state) in peer_states.iter() { | |
| for (intercept_scid, _) in peer_state.outbound_channels_by_intercept_scid.iter() { | |
| if peer_by_intercept_scid.insert(*intercept_scid, *node_id).is_some() { | |
| return Err(io::Error::new( | |
| io::ErrorKind::InvalidData, | |
| "Corrupted state: Intercept SCID collision detected" | |
| )); | |
| } | |
| } | |
| for (channel_id, _) in peer_state.intercept_scid_by_channel_id.iter() { | |
| if peer_by_channel_id.insert(*channel_id, *node_id).is_some() { | |
| return Err(io::Error::new( | |
| io::ErrorKind::InvalidData, | |
| "Corrupted state: Channel ID collision detected" | |
| )); | |
| } | |
| } | |
| } |
Spotted by Diamond
Is this helpful? React 👍 or 👎 to let us know.
tnull
force-pushed
the
2025-01-liquidity-persistence
branch
from
93c234f to
716c06a
Compare
We add simple `persist` call to `LSPS2ServiceHandler` that sequentially persist all the peer states under a key that encodes their node id.
We add simple `persist` call to `LSPS5ServiceHandler` that sequentially persist all the peer states under a key that encodes their node id.
We add simple `persist` call to `EventQueue` that persists it under a `event_queue` key.
.. this is likely only temporary necessary as we can drop our own `dummy_waker` implementation once we bump MSRV.
We read any previously-persisted state upon construction of `LiquidityManager`.
We read any previously-persisted state upon construction of `LiquidityManager`.
We read any previously-persisted state upon construction of `LiquidityManager`.
tnull
force-pushed
the
2025-01-liquidity-persistence
branch
from
716c06a to
0367a27
Compare
This is the second PR in a series of PRs adding persistence to
lightning-liquidity(see #4058). As this is already >1000LoC, I now decided to put this up as an intermediary step instead of adding everything in one go.In this PR we add the serialization logic for for the LSPS2 and LSPS5 service handlers as well as for the event queue. We also have
LiquidityManagertake aKVStoretowards which it persists the respetive peer states keyed by the counterparty's node id.LiquidityManager::newnow also deserializes any previously-persisted state from that givenKVStore. Note that so far we don't actually persist anything, as wiring upBackgroundProcessorto drive persistence will be part of the next PR (which will also make further optimizations, such as only persisting when needed, and persisting some imporant things in-line).This also adds a bunch of boilerplate to account for both
KVStoreandKVStoreSyncvariants, following the approach we previously took withOutputSweeperetc.cc @martinsaposnic