Draft HardForks.md

[nfrisby]

A document discussing how the node (mostly Ledger and Consensus) handle hard forks.

This is part of my going effort to justify simplifying the HFC and might also be useful for deriving some of the Cardano Blueprint.

[nfrisby]

Rendered: https://github.com/IntersectMBO/ouroboros-consensus/blob/nfrisby/doc-triggers/docs/website/contents/for-developers/HardForks.md

[nfrisby]

@lehins I'd appreciate your feedback/contributions on the entire Era section, but especially within this Which changes require a new era? subsection.

[nfrisby]

I had a sentence about "structural changes" specifically, but I seemed to have lost it 🤷

[lehins]

It's ultimately the judgment of the node developers

That is not entirely correct. As I've mentioned on slack, we cannot change serialization format of transactions without introduction of a new era.
Even if we were to change this behavior and allow serialization changes for any protocol version, changes to a transaction format require changes to plutus context, thus requiring a new plutus ledger version. We cannot affect execution of older plutus scripts, so for eg. addition of a field can only be visible to new plutus scripts. This means there is also a 1-to-1 mapping between eras and plutus ledger versions.

In other words we should document it as the current node implementation allows us, not a speculation of what could have been possible if we were in an alternative universe:

Suggested change

	It's ultimately the judgment of the node developers whether some changes should be achieved with an era transition rather than an intra-era hard fork.
	Any amendment to protocol that requires a change to the binary format of a transaction also requires a full era transition rather than an intra-era hard fork.

Moreover, in my personal opinion, this limitation of requiring a hard fork for serialization changes is also a feature. Because serialization changes have a very serious knock-on effect. All of the downstream tooling and hardware wallets are affected by serialization changes. For example we had to freeze all CDDL changes about 6 months before the Conway release, just so hardware wallets were ready for the hard fork.

[nfrisby]

In other words we should document it as the current node implementation allows us, not a speculation of what could have been possible if we were in an alternative universe:

My goal is to identify the requirements that lead to the current design/approach, so I must not take the current implementation for granted.

[nfrisby]

Moreover, in my personal opinion, this limitation of requiring a hard fork for serialization changes is also a feature. Because serialization changes have a very serious knock-on effect. All of the downstream tooling and hardware wallets are affected by serialization changes. For example we had to freeze all CDDL changes about 6 months before the Conway release, just so hardware wallets were ready for the hard fork.

Did you mean to write "requiring an era for serialization changes"? Otherwise, I'm not seeing how this explicitly relates to the question of whether some hard fork needs to be an era transition rather than "just" an intra-era hard fork.

changes to a transaction format require changes to plutus context, thus requiring a new plutus ledger version.

I didn't follow these statements about Plutus. But I'm pretty naive about it. Are you ultimately referring to these data structures?

• https://plutus.cardano.intersectmbo.org/haddock/latest/plutus-ledger-api/src/PlutusLedgerApi.V1.Contexts.html#TxInfo
• https://plutus.cardano.intersectmbo.org/haddock/latest/plutus-ledger-api/src/PlutusLedgerApi.V2.Contexts.html#TxInfo
• https://plutus.cardano.intersectmbo.org/haddock/latest/plutus-ledger-api/src/PlutusLedgerApi.V3.Contexts.html#TxInfo

I see how some eras have affected them (eg Conway obviously shows up in V3), but I'm not seeing how the CDDL details of a tx would affect these types, eg.

I chased from PlutusWithContext https://github.com/IntersectMBO/cardano-ledger/blob/9f98a9e15d8220ccc4940935efdce8dc69a63c22/libs/cardano-ledger-core/src/Cardano/Ledger/Plutus/Evaluate.hs#L94-L117 and those TxInfos seemed like the only ultimate data that was relevant. Maybe I missed a problematic hash or something?

Maybe the CostModel? Its Haddock does explicitly mention hard fork boundaries.

[lehins]

Did you mean to write "requiring an era for serialization changes"?

Yes. Sorry, that is exactly what I meant.

Are you ultimately referring to these data structures?

Correct

I'm not seeing how the CDDL details of a tx would affect these types, eg.

It is an indirect relationship that stems from change to Tx, rather than from CDDL. Change to Tx requires CDDL changes and it requires similar change to Plutus Context, thus new Plutus version. There can also be direct relationship, for example we are planning on making inputs an ordered Set, which will affect both CDDL and Plutus context in interesting ways.

Cost Model nowadays (starting with Conway) can be updated at any point in time, so that is not really related to any future hard fork anymore
See this file, this is where era + plutus context mapping happens for Conway (there is similar functionality for previous eras): https://github.com/IntersectMBO/cardano-ledger/blob/master/eras/conway/impl/src/Cardano/Ledger/Conway/TxInfo.hs

[lehins]

It is just my personal opinion, but the sense I felt from reading this document is that it downplays the role of HFC and overall importance of separation of logic by eras. In my mind it is an incredible achievement of the Cardano core team coming up with this type safe design that logically separates major upgrades.
Could have it been done differently? Of course! From what I understand, however, this document attempts to describe current state of affairs, as such it should emphasize the importance of separation by era, not the opposite.

[lehins]

It's ultimately the judgment of the node developers

That is not entirely correct. As I've mentioned on slack, we cannot change serialization format of transactions without introduction of a new era.
Even if we were to change this behavior and allow serialization changes for any protocol version, changes to a transaction format require changes to plutus context, thus requiring a new plutus ledger version. We cannot affect execution of older plutus scripts, so for eg. addition of a field can only be visible to new plutus scripts. This means there is also a 1-to-1 mapping between eras and plutus ledger versions.

In other words we should document it as the current node implementation allows us, not a speculation of what could have been possible if we were in an alternative universe:

Suggested change

	It's ultimately the judgment of the node developers whether some changes should be achieved with an era transition rather than an intra-era hard fork.
	Any amendment to protocol that requires a change to the binary format of a transaction also requires a full era transition rather than an intra-era hard fork.

Moreover, in my personal opinion, this limitation of requiring a hard fork for serialization changes is also a feature. Because serialization changes have a very serious knock-on effect. All of the downstream tooling and hardware wallets are affected by serialization changes. For example we had to freeze all CDDL changes about 6 months before the Conway release, just so hardware wallets were ready for the hard fork.

[lehins]

It is true, that most of the time it is complexity of the feature that drives the new era, but it is not a requirement. It is not necessarily the simplicity that drives this decision of including a change in an intra-era hard fork. The change could be as simple as prevent duplicates in a Set when it is being deserialized. But because of current limitations of our implementation we do require such change be introduced on a new era. This is a concrete example that we are planning for the next era that we can't introduce with an intra-era hard fork.
Another concrete example was Allegra era. The only change it had was introduction of a Timelock (two extra cases to a native script and a validity interval). It was a fairly simple change, but it required a new era.

[lehins]

Another reason for a new era is that sometimes it happens that new features that are being introduced require an initial value, which needs to be populated upon a hard fork. For example initial Constitution and the guardrails script in Conway. Such initial values are supplied through a Genesis file.