Discussion: Benchmark-Driven MVP

[hero78119]

Hi! The purpose of this lengthy issue is to initiate the discussion on benchmarking and to gain insight into the plans you already have in mind :)

The proposed MVP operates under a benchmark-driven approach, aiming to assess performance across various dimensions. This involves

1. conducting vertical benchmarks to scrutinize individual stage of Singer tackle any bottlenecks hindering critical path. e.g. use in place evaluation instead of clone #50
2. horizontal benchmarks on established crypto-primitives. => @han0110 has previously conducted a similar benchmark on Keccak256, the results of which can be found here. we can iterate on the Keccak256-singer version and compare its performance against benchmark listed results.
3. Furthermore, horizontal benchmarks target real-world applications of zkVM, providing insight into its feasibility and estimate the appropriate computational resources required.

Given the comprehensive coverage of aspects 1 and 2, this issue will allocate more focus to 3, delving into the practical applications of zkVM. Ideally, any performance enhancement in 1 and 2 should also translate into improvements in 3. However, there might be two scenarios where the necessity of focusing on 3 becomes evident:

• Even if performance in 1 and 2 is suboptimal, showcasing promising potential in 3 demonstrates good capability.
• conversely, even if excel in 1 and 2, if 3 remains impractical for real-world use, it signifies a significant gap that needs to be addressed.

zkVM + Simulating Workload

In line with the essence of the MVP, conduct benchmark the prover's performance on a practical zkVM candidate to gauge our current standing.

During the early stages, we could center on benchmarking the single prover runtime and memory consumption of GKR-zkVM. To simulate the workload encompassing DP + TS + CHIP, we construct a mock circuit and supply it with a realistic mock workload. This includes parameters such as the number of opcodes and the average length of instances for each opcode, and average length bytecode. A typical workflow involves fetching the latest X blocks from the blockchain, interpreting them into witnesses, and statistically analyzing opcode frequency and average counts etc. With this data in hand, we can conduct a mock benchmark consisting of the GKR-zkVM mock circuit and an asymmetric prover to verify one or more blocks, each with its distinct characteristics.

Given the benchmarking of a single prover, it's natural to infer a rough estimation for multi-prover scenarios, considering the high concurrency inherent in GKR-zkVM.

w.l.o.g., focusing on benchmarking a selected subset of implemented opcodes and substituting incomplete opcodes can render results representative with minimal effort. This approach enables us to conduct benchmarks at an early stage, even before achieving implementation completeness, which typically demands more time.

zkVM Options

One option involves selecting revm as the interpreter. This entails sampling numerous transactions from the blockchain, interpreting them into witnesses, and gathering statistical results. Based on these statistics, we construct a mock GKR circuit and benchmark it with mock payloads.

Alternatively, we could opt for a pure memory machine that seamlessly integrates with the Singer framework, with RISC-V presenting a promising choice. RISC-V operates as a memory machine, with its registers reserved at specific memory addresses, simplifying memory operations. To initiate this approach, we begin by selecting the latest X blocks from the blockchain and replaying them via revm compiled in RISC-V. Here, the RISC-V memory space initially lays out a single (or multiple) block byte alongside revm bytecode. Upon setting the initial program counter (PC) and executing until halt, the witness encompasses the entire execution trace. By repeating this process with different blocks, we can statistically derive the RISC-V opcode distribution necessary to prove a block.

In the case of the revm interpreter, GKR-zkVM will adopt a slightly narrower focus in several scope. Firstly, it concentrates on a transaction-based approach rather than operating at the block level, necessitating some dispatch/aggregation proof work outside of GKR-zkVM. Additionally, merkle proof verification will also be delegated outside of GKR-zkVM. Conversely, Revm-on-riscv simplifies the design of GKR-zkVM while covering a broader scope. It can prove blocks without encapsulating stack/calldata/storage computations within pure memory operations.

[zhenfeizhang]

In general I believe we are aligned on this approach. It is always our intention to bench over risc-v. It's just it takes time to build all the foundations and we are still at that stage.