[Key derivation V2] Attach a version byte to channel_keys_id

[tankyleo]

Early, broken draft of a PR that would allow us to understand which key derivation we used for a particular channel_keys_id.

After this PR, we would ship a new derivation for to_remote outputs such that they may be recovered solely from the seed in case of loss of all channel state (including the channel_keys_id of the channel - which currently is required to derive the to_remote key).

An alternative approach could assume a certain byte in channel_keys_id has never been used, and we would rely on that byte to understand which key derivation to use. I've heard this assumption would break some downstream users, so pushed this approach instead.

[ldk-reviews-bot]

👋 Thanks for assigning @tnull as a reviewer!
I'll wait for their review and will help manage the review process.
Once they submit their review, I'll check if a second reviewer would be helpful.

[tnull]

Seems this already needs a rebase :)

[tnull]

Concept ACK.

This is basically 1:1 my preferred approach from #3391, which @TheBlueMatt wasn't the biggest fan of though.

It might be good to include the actual v2 derivation in this PR, too, as it would be critical to see whether we can do the derivation in a backwards compatible manner everywhere. Also, would we necessarily break forwards compat everywhere with this?

Speaking off, please note that newly introduced fields need to have odd numbers to not automatically break forwards compatibility, as LDK follows the 'it's okay to be odd' rule for its serialization. This means that nodes would panic on downgrade if they encountered an even field that they aren't expecting.

[tnull]

This needs to be odd, as otherwise we'd break downgrades.

[tnull]

Same here, needs to be odd.

[tnull]

Same, here, new field needs to be odd to not break downgrades.

[tankyleo]

I was hoping with 'no_write_default' to allow backwards compatibility iff the version is set to 0. I may be missing something let me know.

[tnull]

Hmm, ah, not sure if we'd want to add another serialization path just for this. Maybe it would be easier to make the version an Option<u8>? This would also make the behavior explicit and you could actually do manual migration steps depending on whether it's set (without leaning on a magic '0' value).

[tankyleo]

If I recall correctly, for a None we would still write the even type, and make it impossible to downgrade ? Will check tomorrow :)

Or actually yes we could make it Option together with the option field type (not required)

[TheBlueMatt]

It does seem like its a nicer API if we always have the version, vs having an Optional version, no? I guess as a new field it is kinda an Option... FWIW, I don't think we have to break the reads out nor add a new write type here, eg the following. I guess I don't have a strong opinion on option-or-not.

+impl_writeable_tlv_based!(DelayedPaymentOutputDescriptor, {
+       (0, outpoint, required),
+       (2, per_commitment_point, required),
+       (4, to_self_delay, required),
+       (6, output, required),
+       (8, revocation_pubkey, required),
+       (10, channel_keys_id_id, (legacy, [u8; 32], |obj: &DelayedPaymentOutputDescriptor| Some(obj.channel_keys_id.id))),
+       (12, channel_value_satoshis, required),
+       (13, channel_transaction_parameters, (option: ReadableArgs, Some(channel_value_satoshis.0.unwrap()))),
+       (14, channel_keys_id_vers, (legacy, u8, |obj: &DelayedPaymentOutputDescriptor| Some(obj.channel_keys_id.version))),
+       (x, channel_keys_id, (static_value, ChannelKeysId {
+               id: channel_keys_id_id.ok_or(DecodeError::InvalidValue)?,
+               version: channel_keys_id_vers.ok_or(DecodeError::InvalidValue)?,
+       })),
+});

[tnull]

Same here.

[tnull]

Same here.

[tankyleo]

Concept ACK.

This is basically 1:1 my preferred approach from #3391, which @TheBlueMatt wasn't the biggest fan of though.

It might be good to include the actual v2 derivation in this PR, too, as it would be critical to see whether we can do the derivation in a backwards compatible manner everywhere. Also, would we necessarily break forwards compat everywhere with this?

Speaking off, please note that newly introduced fields need to have odd numbers to not automatically break forwards compatibility, as LDK follows the 'it's okay to be odd' rule for its serialization. This means that nodes would panic on downgrade if they encountered an even field that they aren't expecting.

Overall the idea is to allow backwards compat for version 0 and no backwards compat for anything else. For forward compat, when the field is not set, we assume it is 0.

[ldk-reviews-bot]

🔔 1st Reminder

Hey @TheBlueMatt! This PR has been waiting for your review.
Please take a look when you have a chance. If you're unable to review, please let us know so we can find another reviewer.

[tnull]

This unfortunately already needs a rebase.

[TheBlueMatt]

This is basically 1:1 my preferred approach from #3391, which @TheBlueMatt wasn't the biggest fan of though.

Indeed, I remain not a huge fan of the version approach - the channel keys id is a public thing that isn't restricted to just our KeysManager (even if most nodes probably do use our KeysManager), so changing the public API because of a limitation of our KeysManager feels incredibly weird - what is the "version" for a SignerProvider that doesn't need it?

In our KeysManager, on the other hand, we do have room for a few extra bytes...currently the first 4 bytes are a channel-open counter since start. Defining the top byte being 0xff as "version two derivation" seems perfectly reasonable to me, I'm really quite confident no one has opened 4 billion channels in a single LDK instance without restarting :).

[tnull]

In our KeysManager, on the other hand, we do have room for a few extra bytes...currently the first 4 bytes are a channel-open counter since start.

Although note we previously established that we can’t go that way either as some of our users deployed a custom channel_keys_id derivation, ie, we can’t lean on the first few bytes being available/unset.

[TheBlueMatt]

Ah, I'd forgotten that (also, why on earth did anyone do that?!). So, then, yea, I guess a version byte is the only real option? Was there any other option? (discussion on the previous PR seems to hint that we had some other path to explore)

[tnull]

Ah, I'd forgotten that (also, why on earth did anyone do that?!). So, then, yea, I guess a version byte is the only real option? Was there any other option? (discussion on the previous PR seems to hint that we had some other path to explore)

Well, one third option we discussed was to always derive both variants and always just check which one to use, i.e., never set an explicit version anywhere. I do however dislike that approach, if we can avoid it, also since introducing a version does allow us to change the derivation scheme in the future without facing the same challenges again.

[TheBlueMatt]

Ah, I mean hopefully we don't have to do that if we are more careful about the derivation scheme now? There's pretty little overhead to the extra derivation on construction and equality checks, so it shouldn't be too bad...