Draft HardForks.md and/or ProtocolVersioning.md #1401
Conversation
|
|
||
| ### Which changes require a new era? | ||
|
|
||
| It's ultimately the judgment of the node developers whether some changes should be achieved with an era transition rather than an intra-era hard fork. |
There was a problem hiding this comment.
@lehins I'd appreciate your feedback/contributions on the entire Era section, but especially within this Which changes require a new era? subsection.
There was a problem hiding this comment.
I had a sentence about "structural changes" specifically, but I seemed to have lost it 🤷
There was a problem hiding this comment.
It's ultimately the judgment of the node developers
That is not entirely correct. As I've mentioned on slack, we cannot change serialization format of transactions without introduction of a new era.
Even if we were to change this behavior and allow serialization changes for any protocol version, changes to a transaction format require changes to plutus context, thus requiring a new plutus ledger version. We cannot affect execution of older plutus scripts, so for eg. addition of a field can only be visible to new plutus scripts. This means there is also a 1-to-1 mapping between eras and plutus ledger versions.
In other words we should document it as the current node implementation allows us, not a speculation of what could have been possible if we were in an alternative universe:
| It's ultimately the judgment of the node developers whether some changes should be achieved with an era transition rather than an intra-era hard fork. | |
| Any amendment to protocol that requires a change to the binary format of a transaction also requires a full era transition rather than an intra-era hard fork. |
Moreover, in my personal opinion, this limitation of requiring a hard fork for serialization changes is also a feature. Because serialization changes have a very serious knock-on effect. All of the downstream tooling and hardware wallets are affected by serialization changes. For example we had to freeze all CDDL changes about 6 months before the Conway release, just so hardware wallets were ready for the hard fork.
There was a problem hiding this comment.
In other words we should document it as the current node implementation allows us, not a speculation of what could have been possible if we were in an alternative universe:
My goal is to identify the requirements that lead to the current design/approach, so I must not take the current implementation for granted.
There was a problem hiding this comment.
Moreover, in my personal opinion, this limitation of requiring a hard fork for serialization changes is also a feature. Because serialization changes have a very serious knock-on effect. All of the downstream tooling and hardware wallets are affected by serialization changes. For example we had to freeze all CDDL changes about 6 months before the Conway release, just so hardware wallets were ready for the hard fork.
Did you mean to write "requiring an era for serialization changes"? Otherwise, I'm not seeing how this explicitly relates to the question of whether some hard fork needs to be an era transition rather than "just" an intra-era hard fork.
changes to a transaction format require changes to plutus context, thus requiring a new plutus ledger version.
I didn't follow these statements about Plutus. But I'm pretty naive about it. Are you ultimately referring to these data structures?
- https://plutus.cardano.intersectmbo.org/haddock/latest/plutus-ledger-api/src/PlutusLedgerApi.V1.Contexts.html#TxInfo
- https://plutus.cardano.intersectmbo.org/haddock/latest/plutus-ledger-api/src/PlutusLedgerApi.V2.Contexts.html#TxInfo
- https://plutus.cardano.intersectmbo.org/haddock/latest/plutus-ledger-api/src/PlutusLedgerApi.V3.Contexts.html#TxInfo
I see how some eras have affected them (eg Conway obviously shows up in V3), but I'm not seeing how the CDDL details of a tx would affect these types, eg.
I chased from PlutusWithContext https://github.com/IntersectMBO/cardano-ledger/blob/9f98a9e15d8220ccc4940935efdce8dc69a63c22/libs/cardano-ledger-core/src/Cardano/Ledger/Plutus/Evaluate.hs#L94-L117 and those TxInfos seemed like the only ultimate data that was relevant. Maybe I missed a problematic hash or something?
Maybe the CostModel? Its Haddock does explicitly mention hard fork boundaries.
There was a problem hiding this comment.
Did you mean to write "requiring an era for serialization changes"?
Yes. Sorry, that is exactly what I meant.
Are you ultimately referring to these data structures?
Correct
I'm not seeing how the CDDL details of a tx would affect these types, eg.
It is an indirect relationship that stems from change to Tx, rather than from CDDL. Change to Tx requires CDDL changes and it requires similar change to Plutus Context, thus new Plutus version. There can also be direct relationship, for example we are planning on making inputs an ordered Set, which will affect both CDDL and Plutus context in interesting ways.
Cost Model nowadays (starting with Conway) can be updated at any point in time, so that is not really related to any future hard fork anymore
See this file, this is where era + plutus context mapping happens for Conway (there is similar functionality for previous eras): https://github.com/IntersectMBO/cardano-ledger/blob/master/eras/conway/impl/src/Cardano/Ledger/Conway/TxInfo.hs
There was a problem hiding this comment.
It is just my personal opinion, but the sense I felt from reading this document is that it downplays the role of HFC and overall importance of separation of logic by eras. In my mind it is an incredible achievement of the Cardano core team coming up with this type safe design that logically separates major upgrades.
Could have it been done differently? Of course! From what I understand, however, this document attempts to describe current state of affairs, as such it should emphasize the importance of separation by era, not the opposite.
|
|
||
| ### Which changes require a new era? | ||
|
|
||
| It's ultimately the judgment of the node developers whether some changes should be achieved with an era transition rather than an intra-era hard fork. |
There was a problem hiding this comment.
It's ultimately the judgment of the node developers
That is not entirely correct. As I've mentioned on slack, we cannot change serialization format of transactions without introduction of a new era.
Even if we were to change this behavior and allow serialization changes for any protocol version, changes to a transaction format require changes to plutus context, thus requiring a new plutus ledger version. We cannot affect execution of older plutus scripts, so for eg. addition of a field can only be visible to new plutus scripts. This means there is also a 1-to-1 mapping between eras and plutus ledger versions.
In other words we should document it as the current node implementation allows us, not a speculation of what could have been possible if we were in an alternative universe:
| It's ultimately the judgment of the node developers whether some changes should be achieved with an era transition rather than an intra-era hard fork. | |
| Any amendment to protocol that requires a change to the binary format of a transaction also requires a full era transition rather than an intra-era hard fork. |
Moreover, in my personal opinion, this limitation of requiring a hard fork for serialization changes is also a feature. Because serialization changes have a very serious knock-on effect. All of the downstream tooling and hardware wallets are affected by serialization changes. For example we had to freeze all CDDL changes about 6 months before the Conway release, just so hardware wallets were ready for the hard fork.
| A hard fork is the result of the nodes' developers introducing backwards-incompatible changes [^change-decisions] and the community ultimately choosing to adopt them by increasing the major protocol version parameter via on-chain governance [^frame-rule]. | ||
| When those developers design and implement such changes, they determine whether the changes are easy to maintain as variations within the existing code. | ||
| If so, then the hard fork is ultimately toggling some conditionals within the code. | ||
| If the changes are not so simple, the Cardano developers instead introduce a new _era_ within the code [^roadmap-disambiguation]. |
There was a problem hiding this comment.
It is true, that most of the time it is complexity of the feature that drives the new era, but it is not a requirement. It is not necessarily the simplicity that drives this decision of including a change in an intra-era hard fork. The change could be as simple as prevent duplicates in a Set when it is being deserialized. But because of current limitations of our implementation we do require such change be introduced on a new era. This is a concrete example that we are planning for the next era that we can't introduce with an intra-era hard fork.
Another concrete example was Allegra era. The only change it had was introduction of a Timelock (two extra cases to a native script and a validity interval). It was a fairly simple change, but it required a new era.
| Eras are a straight-forward mechanism that enables node developers to separate the concerns arising from features that are added to Cardano over time. | ||
| The separation amongst eras inherently simplifies the development of the new features, and it also helps ensure that the Cardano node can sync the entire historical chain [^mithril] --- as well as refuse invalid alternative histories. | ||
|
|
||
| ### Which changes require a new era? |
There was a problem hiding this comment.
Another reason for a new era is that sometimes it happens that new features that are being introduced require an initial value, which needs to be populated upon a hard fork. For example initial Constitution and the guardrails script in Conway. Such initial values are supplied through a Genesis file.
nfrisby
force-pushed
the
nfrisby/doc-triggers
branch
from
5e692b1 to
baf0e16
Compare
nfrisby
changed the title
|
I just pushed up a rewrite, added for now as a second file. There's probably some fat to trim still and maybe another dimension of the design to better emphasize, but I think it has a useful organization and flow. ProtocolVersioning.md |
nfrisby
force-pushed
the
nfrisby/doc-triggers
branch
from
baf0e16 to
ae95fa5
Compare
nfrisby
force-pushed
the
nfrisby/doc-triggers
branch
from
2813994 to
fe9dccc
Compare
| @@ -0,0 +1,155 @@ | |||
| # Introduction | |||
There was a problem hiding this comment.
Just a small detail, the outline of this MD document seems wrong, everything is a subsection of the introduction
| Two nodes won't even be able to exchange transactions, block headers, and/or blocks if they disagree on their identifying hashes. | ||
|
|
||
| The most common change in Cardano's protocol so far has been the addition of a new kind/variant of transaction, which triggers all of these except the Chain Order. | ||
| The Chain Order has only changed from Praos's length to Ouroboros Genesis's length-soon-after-the-intersection, which is slight. |
There was a problem hiding this comment.
| The Chain Order has only changed from Praos's length to Ouroboros Genesis's length-soon-after-the-intersection, which is slight. | |
| The Chain Order has only changed from Praos's length to Ouroboros Genesis's density-after-the-intersection, which is slight. |
I would not relate Genesis and length artificially. Genesis is about density.
There was a problem hiding this comment.
I'm hung up on this a little, since length-soon-after-the-intersection is density, assuming "soon" is a fixed value.
| For the sake of such implementations, the ledger introduces a new era whenever the Chain Data Model changes. | ||
| The [sequence of eras](https://github.com/cardano-foundation/CIPs/blob/master/CIP-0059/feature-table.md) at the time of writing are: Byron [^byron-halves], Shelley, Allegra, Mary, Alonzo, Babbage, and Conway. | ||
|
|
||
| A Cardano node implementation or other tooling implemented in a language prone to dynamic typing might find it convenient to ignore the era tags for the most part. |
There was a problem hiding this comment.
I appreciate the distinction between strongly-typed nodes and other nodes. Perhaps this could end up in the Blueprint
There was a problem hiding this comment.
Great to hear! The blueprint is certainly in the back of my mind as I'm writing this document.
|
In general the document looks good, and I think there explanation is quite clear and I appreciate the fact that no code is actually depicted here. |
Edit: Another option would be for blocks and headers to be tagged with the protocol version, but transactions to be tagged with a bespoke transaction version. The developers would increment this version anytime a new protocol version involves changes to the meaning the meaning of well-formed transactions of the previous transaction version. This eliminates the risk of the user submitting a transaction that is well formed in multiple protocol versions but has different semantics. TODO to what extent is the era tag different from such a transaction version? Maybe this is what @lehins meant before and it's taken me a while to grok it. |
|
@nfrisby I don't fully understand why transactions would include any such tag? This backwards compatibility is a nice property, because users could submit transactions and not worry about era transitions and HFCs, therefore it is unclear to me why during transaction submission today there is information about the era that the transaction was built for. We, in ledger, when we change transaction serialization, we try to do it in the most graceful matter possible for any one era transition, in order to make the transition to the new era smooth. The only reason why we would ever want to change serialization during an intra era hard fork would be to patch some bug, which would have no affect on normal legitimate transactions. If there is a desire from users to specify validity for protocol version, maybe we could provide ability for a user to supply an optional range of protocol versions that the transaction should be valid for. Considering such envelope information today wouldn't be signed it would be more of UX improvement/restriction, rather than some sort of guarantee, unless in the future era we make this protocol version range part of the TxBody, similarly to validity interval. How does this scenario work today?
Cause I'd assume when a node advances from era X to Y all transactions in the mempool are upgraded to that new era Y? If there is a selection of another chain that is still in era X then transactions in the mempool cannot be downgraded to the era X from era Y, at least there is no such mechanism provided by the Ledger. Is the whole mempool reconstructed by deserializing all transactions in era X from their binary representation or how does it work? |
Does this mean that transactions are also forward compatible in serialization? i.e. a transaction created in Shelley can be deserialized just fine as a Conway transaction? is there no special logic used when upgrading Transactions to later eras? Or maybe I'm misunderstanding your comment here. |
What is currently happening is that transaction in the mempool always are kept in the era that they were originally in. E.g. if you submit a Shelley transaction today on mainnet, it will be a Shelley transaction in the mempool. Every time we need to do something with it in the context of a specific ledger state (validate/revalidate), we will first, if possible, translate it into the corresponding era of the ledger state (today: Conway), and then invoke the ledger, but we don't modify the tx in the mempool. This way, we do not need to "un-translate" txs in this scenario. |
Yes, most of them will. There were some bug fixes and some aspects of a transaction since Shelley have been deprecated (like MIR Certs, etc), but majority of transactions will deserialize in Conway. This is not always going to be true. The goal is to keep as much backwards compatibility from one era to the next one that follows, but there is no real need to support all eras all the way down to Shelley.
Nope. Logic is take the bytes and deserialize that transaction in an era that you'd like to validate it in. |
|
|
||
| ## Introduction | ||
|
|
||
| This document details how the Cardano node handles a [hard fork](https://en.wikipedia.org/wiki/Fork_(blockchain)#Hard_fork), and why it handles hard forks that way. |
There was a problem hiding this comment.
Should this sentence mention "protocol versions" (eg "this document explains HF, why it handles HFs that way, and what the role of protocol versions supporting the HF mechanism is").
| Therefore, a new release of the software implements the new rules _and also_ the old rules. | ||
| That release continues to use the old rules --- even for fresh blocks --- until the community ultimately chooses to switch to the new rules by increasing the major protocol version parameter via on-chain governance [^frame-rule]. | ||
| - *Adoption*. | ||
| It is crucial that the community does not increment the major protocol version until a vast majority of (honest) stake has upgraded their nodes to the new software. |
There was a problem hiding this comment.
Perhaps it's worth clarifying or reprasing the sentence
increment the major protocol version
Does this mean: start producing blocks according to the rules of the new version?
| It is crucial that the community does not increment the major protocol version until a vast majority of (honest) stake has upgraded their nodes to the new software. | ||
| Stake pool operators should therefore not vote to increment before upgrading their nodes to the new software. | ||
|
|
||
| Because the protocol version is determined on-chain, it's perfectly plausible for divergent chains to increment the major version at different times. |
There was a problem hiding this comment.
Do you think we should mention in this section that protocol major-versions can only be incremented at the beginning of a new epoch?
| When Cardano developers design and implement the changes for a hard fork, they determine whether the changes are easy to maintain as variations within the existing code. | ||
| If so, the hard fork is ultimately implemented by some conditionals within the code that branch on the major protocol version. | ||
| If the changes are not so simple, the Cardano developers instead introduce a new _era_ within the code [^roadmap-disambiguation]. | ||
| (See "Which changes require a new era?" below --- this determination is done within the context of the existing implementation's architecture, not a vacuum.) |
There was a problem hiding this comment.
We can insert a Markdown reference here.
| When Cardano developers design and implement the changes for a hard fork, they determine whether the changes are easy to maintain as variations within the existing code. | ||
| If so, the hard fork is ultimately implemented by some conditionals within the code that branch on the major protocol version. | ||
| If the changes are not so simple, the Cardano developers instead introduce a new _era_ within the code [^roadmap-disambiguation]. | ||
| (See "Which changes require a new era?" below --- this determination is done within the context of the existing implementation's architecture, not a vacuum.) |
There was a problem hiding this comment.
this determination is done within the context of the existing implementation's architecture, not a vacuum.
Sorry, I don't get this sentence, or its relevance in this context.
| ### How wide-reaching are eras? | ||
|
|
||
| Eras are ultimately an implementation detail of the node, a consequence of architectural trade-offs. | ||
| The node's behaviors currently reflects the eras in only two ways. |
There was a problem hiding this comment.
I would expect these two ways to be mentioned after this sentence :)
|
|
||
| Specifically, the node must be able to forecast the data necessary to validate a useful prefix of some header chain from the intersection of that header chain and the node's current selection, including whether each header arrived too early. | ||
|
|
||
| - The familiar and intuitive notion of era is useful for organization in general, and so the eras' names might appear within parameter names in configuration files, specification documents, announcements, etc. |
There was a problem hiding this comment.
This bullet seems a bit disconnected from the preceding paragraph.
| ## Forecasting, forewarning | ||
|
|
||
| Hard forks can affect the validity of block headers, and so must be sufficiently forecastable. | ||
| For this reason, hard forks must not happen less than 36 hr (on mainnet) after the voting deadline, ie hard forks must have at least 36 hrs of forewarning --- just like other protocol parameter updates that affect headers (eg maximum block size). |
There was a problem hiding this comment.
Maybe clarify that 36 hours correspond to 3k?
| ## Forecasting, forewarning | ||
|
|
||
| Hard forks can affect the validity of block headers, and so must be sufficiently forecastable. | ||
| For this reason, hard forks must not happen less than 36 hr (on mainnet) after the voting deadline, ie hard forks must have at least 36 hrs of forewarning --- just like other protocol parameter updates that affect headers (eg maximum block size). |
There was a problem hiding this comment.
| For this reason, hard forks must not happen less than 36 hr (on mainnet) after the voting deadline, ie hard forks must have at least 36 hrs of forewarning --- just like other protocol parameter updates that affect headers (eg maximum block size). | |
| For this reason, hard forks must not happen until 36 hours (on mainnet) after the voting deadline has passed, ie hard forks must have at least 36 hours of forewarning --- just like other protocol parameter updates that affect headers (eg maximum block size). |
Say I get one of those deprecated MIR certs transactions, which I guess must be regarded as a Shelley transaction. It will succeed to deserialize as a Shelley transaction, and what happens then if I try to translate it to Conway? will it fail to be upgraded? will it be upgraded to something which yields always What I am looking to clarify is:
And also:
Are these two statements true? Edit: please consider writing a reply to this comment in this thread I made: https://github.com/IntersectMBO/ouroboros-consensus/pull/1401/files#r1993639143 |
|
|
||
| ## The Protocol Version | ||
|
|
||
| Each change to the Cardano protocol advances through the following steps before influencing the network's chain. |
There was a problem hiding this comment.
I'm not sure "influencing" is the right term.
| Each change to the Cardano protocol advances through the following steps before influencing the network's chain. | |
| Each change to the Cardano protocol advances through the following steps before impacting the network's blockchain. |
| - The Technical Steering Committee and Ouroboros researchers address the worth, safety, and future consequences of the proposed solution. | ||
| - Cardano developers prepare a software release that includes the changes but only enables them for chains on which the protocol version has been incremented. | ||
| - The community uses on-chain governance to increment the protocol version, thereby putting the change into effect. | ||
| (Stake pool operators must not vote to increment the protocol version before updating their software!) |
There was a problem hiding this comment.
Is it sufficiently clear for the reader which protocol version increment we're referring to? I guess this refers to the protocol version included in the headers, which denotes the maximum protocol version our node supports.
|
|
||
| It is crucial that the community does not increment the protocol version before enough nodes have upgraded their software. | ||
| Each protocol version increment corresponds to a _hard fork_, because the changes are backwards-incompatible: nodes running the old code will not be able to adopt blocks minted by the new code. | ||
| Even if some block content happened to be valid in both the old and the new protocol, each software release explicitly refuses any block that (extends a ledger state that) has a protocol version greater than that release was intended to implement — see the MaxMajorPV constant. |
There was a problem hiding this comment.
Is it worth adding a link to the location where MaxMajorPV is defined?
|
|
||
| Even software released well after mainnet increments the protocol version must implement the protocol both with and without that increment's changes. | ||
| The Cardano node must correctly process blocks with older protocol versions, eg when catching up to mainnet for the first time or after being offline for a long time. | ||
| So every protocol change must remain conditional within all future releases, instead of only within the particular release that mainnet used for that increment of the protocol version. |
There was a problem hiding this comment.
What does it mean for a protocol change to be conditional on a (future) release?
|
In the future, it would be nice to expand this document and describe the relation between protocol version, |
| @@ -0,0 +1,155 @@ | |||
| # Introduction | |||
|
|
|||
| The Cardano protocol will change over time. | |||
There was a problem hiding this comment.
GitHub does not allow threading of comments on the PR itself. But we've got a pretty big conversation happening there right now, so I'm going to arbitrarily choose this line in the diff in order to create a thread for that conversation.
@lehins wrote:
We, in ledger, when we change transaction serialization, we try to do it in the most graceful matter possible for any one era transition, in order to make the transition to the new era smooth. The only reason why we would ever want to change serialization during an intra era hard fork would be to patch some bug, which would have no affect on normal legitimate transactions.
and
This is not always going to be true. The goal is to keep as much backwards compatibility from one era to the next one that follows, but there is no real need to support all eras all the way down to Shelley.
Based on those two quotes, I'm thinking the tacit rule is that: Cardano users should only ever consider the latest protocol version, and maybe the two latest protocol versions when near a hard fork event.
And Alexey is saying that the ledger team tries, for each hard fork event, to minimize the disagreement between those two protocol versions regarding what some bytes might mean as a transaction.
This is a very helpful insight for me; I'll ponder it. I could very much see it simplifying my intuition about what's important here. On the other hand, there is some appeal in the user explicit declaring their intent via an explicit version field, as Alexey elaborated here:
If there is a desire from users to specify validity for protocol version, maybe we could provide ability for a user to supply an optional range of protocol versions that the transaction should be valid for. Considering such envelope information today wouldn't be signed it would be more of UX improvement/restriction, rather than some sort of guarantee, unless in the future era we make this protocol version range part of the TxBody, similarly to validity interval.
Ultimately, the Mempool needs to know the byte size/ExUnits size/ref script size/etc of a tx before validating that tx. And as of UTxO HD it'll need to identify the set of TxIn before validating the tx. Etc. Parsing the tx to project out that data currently requires choosing which era's codec to use.
That's the only reason why "a tx is just bytes" doesn't trivially work for the Mempool with the HFC.
However, maybe that idea remains workable if we store txs "as bytes" even when they're already in the Mempool. That is, whenever the Mempool does anything with a tx (eg measure its sizes, revalidate it, etc) it applies the codec for the era (/protocol version) of the node's current selection to the tx's bytes --- forgetting any parsing it might have done before (unless the era is the same as last time, eg).
There was a problem hiding this comment.
However, maybe that idea remains workable if we store txs "as bytes" even when they're already in the Mempool.
This is already possible. You can always get the original bytes of any transaction with this originalBytes function if you ever need to re-deserialize it another era.
|
@jasagredo It is much simpler than you think. When you look at CDDL of a transaction, there is no indication of which era it was built for. Sometimes it can be inferred, eg. if a transaction contains votes than it surely was build for Conway. So, the way translation works in ledger is we take the bytes of the transaction and try to deserialize it in the new era. In other words, there is no "translation" happens. We simply deserialize the submitted bytes in the era that it needs to be validated in and we retain those bytes. If that transaction is incompatible with this era you will get deserialization error. The overlap of compatibility of transactions for all eras can be inferred form the CDDL specification. If two different eras have the same specification for the parts of the transaction, then it will deserialize for those two eras. Whether it validate or not that is a different story, because there are plenty of things that can affect validity of a transaction, eg. if cost models protocol parameter were updated then it would effectively invalidate all transactions that use older cost model for the plutus version that was updated. |
A document discussing how the node (mostly Ledger and Consensus) handle hard forks.
This is part of my going effort to justify simplifying the HFC and might also be useful for deriving some of the Cardano Blueprint.