Miden node architecture

[bobbinth]

To better frame discussions in #121 and #126, I think it would make sense to talk through higher-level architecture of Miden node. Below is my first attempt at describing it.

Fist, here is a high-level diagram for Miden node. It includes most components that we need for the testnet, but it doesn't include components that go beyond that (i.e., consensus, P2P etc.).

Let's go through these components one-by-one

RPC provider

This component is responsible for handling external requests into Miden node. The request could be broadly categorized into two categories:

• Requests to process new transactions (submitted via submit_transaction or execute_transaction).
• Requests to retrieve some data from the node (e.g., get_block, check_nullifiers, get_notes_by_tag etc.).

RPC provider is also responsible to for handling things like initial request validation, rate limiting (maybe something PoW-based) etc.

Transaction compiler/executor

These components are responsible for receiving raw (un-proven) transactions from clients, compiling them, and proving their execution (see here for more detail). We don't need these components for the private testnet - so, I won't describe them in detail now (we will need them for the public testnet though).

Tx compiler and executor will need to get data about the current state of the chain to read account states, note data etc. Once compiled and executed, transactions end up in the unproven tx pool and await further processing.

Proof validator

This component is responsible for verifying transaction proofs received from clients. It needs to read the chain state to perform such checks as:

• Transaction updates the most recent account states.
• Notes consumed by the transaction haven't been consumed previously (via checking the nullifier DB).

After transaction proofs are verified, transactions end up in the proven tx pool.

Mempool

This component contains unproven and provent transactions mentioned previously, as well as transaction batches. For private testnent we won't have the unproven tx pool.

Mempool is responsible for coordinating progression of transactions from un-proven to proven to transaction batches, which eventually get included in blocks. My current idea of how it works is as follows:

• There is a set of tx provers which would probably run on separate machines. These tx provers periodically make requests to the mempool to see if there are any unproven transactions. If there are such transactions, a tx prover takes it and proves it.
  • An important thing to note is that tx provers should be "stateless" - that is, all info required to prove a transaction should already be in the mempool and thus, tx provers don't need to access chain state.
• There is a set of tx aggregators which would probably also run on separate machines. These tx aggregators would grab a set of proven transactions from the mempool, aggregate them them into batches, and return these batches back the the mempool.
  • Similar to tx provers, tx aggregators should be "stateless" and all the info needed to generate transaction batches should already be in the mempool.

Mempool will need to handle timeout, retry etc. logic in case one tx prover/aggregators takes transactions but then fails to prove or aggregate them in a reasonable amount of time. Most of the interfaces the mempool exposes would probably be asyncronous.

Block producer

This component would be responsible for taking a set of batches from the mempool and building the next block out of them.

To build a block, the block producer would need to read current chain state. For example, for each batch, the block producer will get a list of updated accounts. So, it will request Merkle paths from each updated account form the store. It would then put these paths into a Merkle store and run Miden VM block production program. Similar thing will be done for nullifiers as well.

Once the block is built, the block producer will send it to the store to update the chain state, and after that will start building the next block. To make this process more efficient, we could implement sophisticated caching strategies in the block producer, but this is not something we need for the testnet.

Store

This component will maintain the latest chain state. A more detailed description of every database inside the store is provided here. The store will expose a set of external interfaces which could be grouped into two categories:

• Interfaces for reading data from the store. These will likely be all asynchronous.
• An interface for updating the data in the store via apply_block. This would be the only way to update the data inside the store, and it would be performed atomically. That is, applying changes for a specific block will be done all at once, and no read requests would receive responses with partially updated data.

Internally, the store will maintain a persistent database as well as in-memory structures required to facilitate the requests.

[hackaugusto]

Awesome draft! I have a few questions:

• What do the arrows in the picture represent? Are they function calls? Queues? RPC Requests? For example, is the tx compiler called in the RPC thread? How is the data provided by the store to the RPC?
• How is the store handling concurrent access? I suppose the architecture won't be single threaded, so it is possible that a block producer is writing to the store at the same time a proof validator is reading from it.
• Will there be a different architecture to run a local client? Or will it be the same thing for a user doing local proofs?

I also think it makes sense to plan for concurrent write to the DB, since in the future with federation there will be writes coming from the RPC layer, and not just the block producer. It would also make sense to think about how parallelism will be done for individual accounts, i.e. how do we use the mailbox concept for each account's actor.

Here are a few ideas:

• One txpool per account. In the case of network accounts that allows proofs for these to be produced concurrently.
• The txpool should also serialize the data, for error recovery on the node without losing received transactions. Otherwise the RPC should block until the transaction is confirmed prior to returning an RPC endpoint. Another alternative is to have endpoints to request the status of a given transaction, so a client can tell when a given transaction disappeared and it has to send it again.
• The RPC should be a thin thread, that receives a request and pushes it to a queue. The queue should have a maximum size and block if that is reached (or return an error), for back pressure propagation.
  • A control plane RPC should be in a separate thread. Since managing the node should not be blocked if the main thread is handling pressure issues (i.e. a control/data plane split)
• The provers should write the result to the DB as soon as possible. This is useful for crash recovery, effectively it caches the result of proving a transaction, and it doesn't have a lot of work to do on restarts.

[bobbinth]

What do the arrows in the picture represent?

These are general directions of request/data flows (and now that I look at it, I've used them quite inconsistently). So, for example: RPC provider would make a call to the TX compiler (this would probably be an async call). Then, TX compiler would request some data from the store (could be in-process async call - or something else). Once TX compiler compiles the transaction, it would make a call to the TX executor and pass the compiled transaction to it. I think many of the exact mechanisms here are still TBD (we have a few options).

How is the store handling concurrent access? I suppose the architecture won't be single threaded, so it is possible that a block producer is writing to the store at the same time a proof validator is reading from it.

Yes, the architecture will be multi-threaded. In fact, running the store in a different process (on the same or different machine) could be an option (though, not sure if that's a good idea yet).

Regarding concurrency, I think it would be up to the store implementation. At this point, the contract of the store is:

• It serves data retrieval requests asynchronously (i.e., multiple read requests can be processed at the same time).
• The only way to update the store is via apply_block call, which will perform the update atomically. And this will probably be a synchronous call.

Will there be a different architecture to run a local client? Or will it be the same thing for a user doing local proofs?

Yes, there will be a different architecture for the client (some initial discussions started here).

I also think it makes sense to plan for concurrent write to the DB, since in the future with federation there will be writes coming from the RPC layer, and not just the block producer.

I think we might be able to handle these via the same apply_block call. Basically, whether the block is received from the network or from the local block producer, there would be the same way to update the store with it. There might be some issues with this as I haven't really spent a lot of time thinking about it - but if we can get away with no concurrent writes, I'd love to do that.

One txpool per account. In the case of network accounts that allows proofs for these to be produced concurrently.

This could work, but this is relevant only for public accounts. For private testnet we'd have only private accounts (public accounts would be introduced in the first public testnet) - so, for now, I haven't given too much thought to the exact mechanics of public transactions.

The txpool should also serialize the data, for error recovery on the node without losing received transactions. Otherwise the RPC should block until the transaction is confirmed prior to returning an RPC endpoint. Another alternative is to have endpoints to request the status of a given transaction, so a client can tell when a given transaction disappeared and it has to send it again.

Yep, these would be good enhancements for the mempool in general. I think these could be added incrementally after the foundational components are in place (e.g., we might not have mempool data serialization at first, but would add it later).

The RPC should be a thin thread, that receives a request and pushes it to a queue. The queue should have a maximum size and block if that is reached (or return an error), for back pressure propagation.

Agreed - though I am hoping that most of this will be handled for us by the framework we choose so that we don't have to implement any of it ourselves.

The provers should write the result to the DB as soon as possible. This is useful for crash recovery, effectively it caches the result of proving a transaction, and it doesn't have a lot of work to do on restarts.

I'd like to keep the provers completely stateless. As soon as they are done with a proof, they'll send it back to the mempool and it would be on the mempool to make sure it can recover those in case of a crush.

[bobbinth]

In fact, running the store in a different process (on the same or different machine) could be an option (though, not sure if that's a good idea yet).

Thinking about this a bit more, it might actually be a good idea. We are not sending too much data between different components - so, data transfers shouldn't be too much of a concern. On the other hand, we gain better encapsulation and option to run things on one or multiple machines. Probably, would be easier to develop too. If I were to break the diagram above into separate processes, I'd probably do it like so:

1. RPC provider.
2. Mempool + TX validator.
3. Block producer.
4. Store.
5. TX/batch provers.

(not sure about where to put + TX compiler/executor - but we can think about these later).

Looking at this as client/server relationships, we could group them like so:

1. Store is a server with RPC provider, block producer, proof validator being clients which make requests to the store.
2. Mempool is a server with RPC provider, block producer, and TX/batch provers being clients which make requests to the mempool.