feat: user batch support by Mirko-von-Leipzig · Pull Request #1846 · 0xMiden/node

Mirko-von-Leipzig · 2026-03-26T16:25:14Z

This PR is the third and final part of the mempool refactoring PR stack. Part 1 (#1820) performs the broad mempool refactoring to simplify this PR. Builds on part 2 (#1832).

Batch submissions must include their transaction inputs since we currently require this for the validator to verify them before inclusion in a block. This PR abuses this by treating the batch as a set of normal transactions at the mempool level. This simplifies the mempool implementation, which is currently built around a DAG of transactions - so having to insert a batch directly would be more complex. This will need to change once we stop requiring transaction inputs as part of the validator; but it won't be too bad.

The way this is implemented here, is that the transaction DAG tracks user batches and ensures that when a batch is selected, that transactions from user batches are not mixed with conventional transactions. That is, select_batch outputs either a user batch, or a conventional batch.

Effectively, the transaction DAG internally ensures that the user batch's transactions remain coherent even though the batch has been deconstructed into individual transactions. The benefit is that this doesn't require any major structural changes to the mempool. The rest of the mempool then treats the user batch as per normal.

Closes #1112 and closes #1859

proto/proto/types/transaction.proto

Mirko-von-Leipzig · 2026-03-26T16:36:08Z

crates/rpc/src/server/api.rs

@PhilippGackstatter if you could throw an eye on the process here to ensure I'm checking the correct things.

The state itself is checked in the mempool, so here we really just want to ensure that the batch and its transactions are valid and the reference block is correct iiuc.

Looks good to me. I left another comment re batch expiration, but your call where/if to do that.

crates/rpc/src/server/api.rs

crates/block-producer/src/mempool/graph/transaction.rs

sergerad · 2026-03-27T01:10:06Z

crates/block-producer/src/mempool/mod.rs

+        &mut self,
+        txs: &[Arc<AuthenticatedTransaction>],
+    ) -> Result<BlockNumber, MempoolSubmissionError> {
+        assert!(!txs.is_empty(), "Cannot have a batch with no transactions");


Just checking we want to crash here instead of return error

I assumed that one cannot build a ProvenBatch without one, so this would indicate an internal bug somewhere. But maybe that's a poor assumption.

sergerad · 2026-03-27T01:11:38Z

crates/block-producer/src/mempool/graph/transaction.rs

+    }
+
+    pub fn select_batch(&mut self, budget: BatchBudget) -> Option<SelectedBatch> {
+        self.select_user_batch().or_else(|| self.select_conventional_batch(budget))


Might want some doc comments to make it clear that budget is intended to only relevant for conventional batches.

Also are we OK with user batches always taking priority over conventional here?

Also wondering if we need to prevent user batches of size 1 (or some other limit). Unsure if that is relevant to this PR just a general thought.

Also are we OK with user batches always taking priority over conventional here?

I'm unsure, but at the moment it doesn't matter much. If its a concern we can make it random -- I was thinking maybe that's best.

Also wondering if we need to prevent user batches of size 1 (or some other limit). Unsure if that is relevant to this PR just a general thought.

Good question. I'm unsure 😬 I wonder if that makes some user loop more difficult i.e. they always submit user batches, but sometimes they don't have many transactions to bundle..

Probably we would want some limit even in the future? cc @bobbinth

Also are we OK with user batches always taking priority over conventional here?

Would having fees solve this?

It would give us a strategy to use, so yes fees solve this imo. Though it will be potentially complex to implement.

I think it is OK to have batches of 1 transactions as people are unlikely to to do them on purpose (they would first need to prove TXs and then prove batches - so, it is non-negligible cost). The fees would also address this as some of the tx-related fees would get amortized over large batches.

Regarding batch selection strategy - I think this is fine for now, but it'll need to change once we have fees so that we prioritize based on some value metric. But that's a whole other discussion.

And agreed that adding some doc comments would be nice.

crates/block-producer/src/mempool/mod.rs

crates/block-producer/src/server/mod.rs

igamigo

LGTM! I left some mostly minor, non-blocking comments. Not sure if you tested already but jsut in case, I'm going to run the client integration tests and report back.

crates/block-producer/src/mempool/graph/batch.rs

crates/block-producer/src/mempool/graph/dag.rs

igamigo · 2026-03-27T18:30:42Z

crates/block-producer/src/mempool/graph/transaction.rs

+    }
+
+    pub fn select_batch(&mut self, budget: BatchBudget) -> Option<SelectedBatch> {
+        self.select_user_batch().or_else(|| self.select_conventional_batch(budget))


Also are we OK with user batches always taking priority over conventional here?

Would having fees solve this?

igamigo · 2026-03-27T18:41:56Z

crates/block-producer/src/mempool/graph/transaction.rs

+                .expect("bi-directional mapping should be coherent");
+
+            for tx in txs {
+                let Some(tx) = self.inner.selection_candidates().get(&tx).copied() else {


Maybe this calls for a get_selection_candidate()? So that you avoid re-allocating the map every iteration through selection_candidates()

I'll refactor this in a follow-up PR.

crates/block-producer/src/mempool/graph/transaction.rs

crates/block-producer/src/server/mod.rs

crates/rpc/src/server/api.rs

Mirko-von-Leipzig · 2026-03-30T12:52:25Z

crates/rpc/src/server/api.rs

+        // Verify batch transaction proofs.
+        //
+        // Need to do this because ProvenBatch has no real kernel yet, so we can only
+        // really check that the calculated proof matches the one given in the request.
+        let expected_proof = LocalBatchProver::new(MIN_PROOF_SECURITY_LEVEL)
+            .prove(proposed_batch.clone())
+            .map_err(|err| {
+                Status::invalid_argument(err.as_report_context("proposed block proof failed"))
+            })?;
+
+        if expected_proof != proven_batch {
+            return Err(Status::invalid_argument("batch proof did not match proposed batch"));
+        }


Is this the idea? I'm unsure on how else to align the proof with the batch.. unless I also compare headers and then just assume?

Can a proof differ based on some other variables e.g. time, rng? Or is this safe to do.

cc @PhilippGackstatter

That seems fine and sufficient. The "proof" (which is not a cryptographic one now anyway) should be deterministic based on the input (the proposed batch), since it's really just destructuring the proposed batch, verifying transaction proofs and constructing the proven batch.

crates/rpc/src/server/api.rs

crates/block-producer/src/server/mod.rs

crates/rpc/src/server/api.rs

PhilippGackstatter · 2026-03-31T15:23:49Z

crates/rpc/src/server/api.rs

+        }
+


Is the batch expiration deliberately not checked? This would also be a pretty cheap check, assuming the latest block is easily retrievable here. E.g. if the batch expiration is already older than the latest block, we should be able to discard it directly.

I think this is something we should add in a follow-up PR, potentially together with keeping a cache of most recent block headers that I mentioned in another comment.

PhilippGackstatter · 2026-03-31T15:24:53Z

crates/rpc/src/server/api.rs

Looks good to me. I left another comment re batch expiration, but your call where/if to do that.

Co-authored-by: Philipp Gackstatter <PhilippGackstatter@users.noreply.github.com>

bobbinth

Looks good! Thank you! I left some comments inline - but I think these can be addressed in follow-up PRs. So, I'll merge this as is.

bobbinth · 2026-03-31T22:30:55Z

proto/proto/types/transaction.proto

+  // The inputs used for the transaction proof.
+  //
+  // Encoded using [miden_protocol::transaction::TransactionInputs::to_bytes].
+  optional bytes transaction_inputs = 2;


I would maybe expand on the comment here to explain why we need to provide transaction_inputs.

bobbinth · 2026-03-31T22:36:12Z

proto/proto/types/transaction.proto

+  // The batch contents of the given proof.
+  //
+  // Encoded using [miden_protocol::batch::ProposedBatch::to_bytes].
+  optional bytes proposed_batch = 2;
+  // The transaction inputs for each transaction in the batch.
+  //
+  // Must match the transaction ordering in the batch.
+  //
+  // Encoded using [miden_protocol::transaction::TransactionInputs::to_bytes].
+  repeated bytes transaction_inputs = 3;


Similar comment as above: would be great to expand the comments to explain how these fields are used.

Separately, I do think eventually this should be a single field that is either a separate structured message or a single serialized struct that contains all the data that we need to re-execute the batch.

bobbinth · 2026-03-31T22:39:51Z

crates/block-producer/src/mempool/graph/batch.rs

+        // We only revert transactions which are _not_ included in batches.
+        let mut to_revert = self.inner.expired(chain_tip);
+        to_revert.retain(|batch| !self.inner.is_selected(batch));


Is the comment here correct? Should it be "We only revert batches which are not included in blocks"?

bobbinth · 2026-03-31T23:13:37Z

crates/rpc/src/server/api.rs

+        // Verify the reference header matches the canonical chain.
+        let reference_header = self
+            .get_block_header_by_number(Request::new(proto::rpc::BlockHeaderByNumberRequest {
+                block_num: expected_proof.reference_block_num().as_u32().into(),
+                include_mmr_proof: false.into(),
+            }))
+            .await?
+            .into_inner()
+            .block_header
+            .map(BlockHeader::try_from)
+            .transpose()?
+            .ok_or_else(|| {
+                Status::invalid_argument(format!(
+                    "unknown reference block {}",
+                    expected_proof.reference_block_num()
+                ))
+            })?;


We don't do a similar check for transactions - i.e., we don't check if the transaction's reference block is actually in the chain (this is done by the block producer). So, if we do want to keep it here, probably makes to add a similar check for transactions.

But also, going to the store from here introduces non-trivial amount of work. So, maybe we should cache some number of recent block headers (e.g,. 1 million) in the RPC so that for most transactions/batches, we don't need to go to the store.

Separately, I wonder if we should keep some kind of an LRU cache for invalid batches/transactions. The attack vector here is that someone could generate a single proven tx/batch that references some invalid block, and then submit it many times. This way, they attacker pays the cost of generating the proof only once, but the node would be forced to verify it many times (maybe thousands) which may also include going to the store etc.

Let's create issues for these.

bobbinth · 2026-03-31T23:14:31Z

crates/rpc/src/server/api.rs

+        let expected_proof = LocalBatchProver::new(MIN_PROOF_SECURITY_LEVEL)
+            .prove(proposed_batch.clone())
+            .map_err(|err| {
+                Status::invalid_argument(err.as_report_context("proposed block proof failed"))
+            })?;


Should we create an issue for this so that we come back and implement it properly once batch kernel proofs are done?

bobbinth · 2026-04-01T00:05:17Z

crates/block-producer/src/mempool/graph/transaction.rs

+    }
+
+    pub fn select_batch(&mut self, budget: BatchBudget) -> Option<SelectedBatch> {
+        self.select_user_batch().or_else(|| self.select_conventional_batch(budget))


I think it is OK to have batches of 1 transactions as people are unlikely to to do them on purpose (they would first need to prove TXs and then prove batches - so, it is non-negligible cost). The fees would also address this as some of the tx-related fees would get amortized over large batches.

Regarding batch selection strategy - I think this is fine for now, but it'll need to change once we have fees so that we prioritize based on some value metric. But that's a whole other discussion.

And agreed that adding some doc comments would be nice.

bobbinth · 2026-04-01T00:13:16Z

crates/block-producer/src/mempool/graph/transaction.rs

+                    // Rollback this batch selection since it cannot complete.
+                    for tx in selected.txs.into_iter().rev() {
+                        self.inner.deselect(tx.id());
+                    }


Would be good to explain why a batch may not be able to complete here (maybe we can do this once we break this code out into a separate function).

bobbinth · 2026-04-01T00:13:39Z

crates/block-producer/src/mempool/graph/transaction.rs

+        None
+    }
+
+    fn select_conventional_batch(&mut self, mut budget: BatchBudget) -> Option<SelectedBatch> {


nit: maybe select_internal_batch()?

bobbinth · 2026-04-01T00:20:09Z

crates/block-producer/src/mempool/graph/transaction.rs

+        // We only revert transactions which are _not_ included in batches.
+        let mut to_revert = self.inner.expired(chain_tip);
+        to_revert.retain(|tx| !self.inner.is_selected(tx));


Is the comment still correct here? Wouldn't this revert transactions in user-defined batches (and the whole batches themselves)?

bobbinth · 2026-04-01T00:24:59Z

crates/block-producer/src/mempool/graph/transaction.rs

+                if let Some(batch) = self.txs_user_batch.remove(&tx.id()) {
+                    if let Some(batch) = self.user_batch_txs.remove(&batch) {
+                        to_revert.extend(batch);
+                    }


nit: using batch in both Some() expressions here is a bit confusing. I'd maybe use batch_id in the outer one, and batch_txs in the inner one.

Mirko-von-Leipzig force-pushed the mirko/mempool-user-batches branch 2 times, most recently from 51e74e4 to 6dd5f53 Compare March 26, 2026 16:31

Mirko-von-Leipzig commented Mar 26, 2026

View reviewed changes

proto/proto/types/transaction.proto Show resolved Hide resolved

Mirko-von-Leipzig marked this pull request as ready for review March 26, 2026 16:34

Mirko-von-Leipzig requested review from PhilippGackstatter, bobbinth, igamigo and sergerad March 26, 2026 16:34

Mirko-von-Leipzig commented Mar 26, 2026

View reviewed changes

crates/rpc/src/server/api.rs Outdated Show resolved Hide resolved

sergerad reviewed Mar 27, 2026

View reviewed changes

crates/block-producer/src/mempool/graph/transaction.rs Outdated Show resolved Hide resolved

sergerad reviewed Mar 27, 2026

View reviewed changes

crates/block-producer/src/mempool/mod.rs Outdated Show resolved Hide resolved

sergerad reviewed Mar 27, 2026

View reviewed changes

crates/block-producer/src/server/mod.rs Show resolved Hide resolved

igamigo approved these changes Mar 27, 2026

View reviewed changes

Base automatically changed from mirko/mempool-tx-reverting to next March 29, 2026 09:02

Merge next

a474113

Mirko-von-Leipzig force-pushed the mirko/mempool-user-batches branch from 681eb87 to a474113 Compare March 30, 2026 08:39

Mirko-von-Leipzig mentioned this pull request Mar 30, 2026

feat(mempool): revert transactions after 3 failures #1832

Merged

Mirko-von-Leipzig added 2 commits March 30, 2026 10:58

Improve comments & address review

2892014

Fix reference header commitment check

149f3ab

Mirko-von-Leipzig commented Mar 30, 2026

View reviewed changes

Require ProvenBatch in gRPC

e528296

Mirko-von-Leipzig force-pushed the mirko/mempool-user-batches branch from 1e09f16 to e528296 Compare March 30, 2026 13:21

sergerad reviewed Mar 31, 2026

View reviewed changes

crates/rpc/src/server/api.rs Outdated Show resolved Hide resolved

sergerad reviewed Mar 31, 2026

View reviewed changes

crates/block-producer/src/server/mod.rs Show resolved Hide resolved

Fix typo

7f2bce1

PhilippGackstatter reviewed Mar 31, 2026

View reviewed changes

Merge branch 'next' into mirko/mempool-user-batches

64bd875

Apply suggestions from code review

74b397a

Co-authored-by: Philipp Gackstatter <PhilippGackstatter@users.noreply.github.com>

bobbinth approved these changes Apr 1, 2026

View reviewed changes

bobbinth merged commit 168cd1d into next Apr 1, 2026
18 checks passed

bobbinth deleted the mirko/mempool-user-batches branch April 1, 2026 00:26

Conversation

Mirko-von-Leipzig commented Mar 26, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

igamigo left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

bobbinth left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Mirko-von-Leipzig commented Mar 26, 2026 •

edited

Loading