MIMD 0006 Redesign of RPC/Geyser subsystem in the validator

[bmuddha]

Request for Comments: Decoupling and Modernizing the MagicBlock Validator RPC Subsystem

Table of Contents

• 1. Abstract
• 2. Assessment of Current Architecture
  • 2.1 Tight Coupling
  • 2.2 Outdated Technology Stack
  • 2.3 Inefficient Request Handling
    • 2.3.1 HTTP Requests
    • 2.3.2 WebSocket Subscriptions
  • 2.4 Unnecessary Overhead
• 3. Proposed Architecture
  • 3.1 Core Architectural Changes
  • 3.2 Technical Implementation Details
    • 3.2.1 Shared State
    • 3.2.2 Communication Protocol
    • 3.2.3 Validator-RPC Interface Design
    • 3.2.4 Geyser Plugin Support
  • 3.3 Deployment Model
• 4. Benefits
  • 4.1 Separation of Concerns
  • 4.2 Independent Evolution
  • 4.3 Performance Improvements
  • 4.4 Improved Maintainability
• 5. Risks and Mitigations
  • 5.1 Increased Deployment Complexity
  • 5.2 Consistency Guarantees
• 6. Implementation Plan
• 7. Conclusion
• 8. Current Blockers

1. Abstract

This RFC proposes a comprehensive redesign of the MagicBlock validator’s RPC subsystem. The core objective is to decouple RPC functionality from the validator core, implementing it as an independent service that communicates with the validator via high-performance shared memory channels. This architectural separation is intended to improve maintainability, performance, and scalability, while enabling both components to evolve independently.

2. Assessment of Current Architecture

2.1 Tight Coupling

Currently, the RPC subsystem is tightly integrated with the validator core, resulting in a monolithic architecture with complex interdependencies. This tight coupling complicates maintenance, testing, and extensibility, and increases the risk of introducing bugs during development.

2.2 Outdated Technology Stack

The existing system relies on the deprecated jsonrpc library. While this library facilitated rapid prototyping, it introduces significant abstraction that obscures underlying mechanisms and limits opportunities for performance optimization.

2.3 Inefficient Request Handling

2.3.1 HTTP Requests

HTTP request handling, though straightforward, is constrained by the limitations of the underlying libraries, resulting in suboptimal performance.

2.3.2 WebSocket Subscriptions

The current WebSocket (PubSub) implementation is particularly inefficient:

• Each subscription spawns an asynchronous task that listens to all events on a broadcast channel.
• Every task receives every event, regardless of relevance.
• Each task must independently filter events for client interest.
• The vast majority of events (estimated at 99.99% in some scenarios) are discarded as irrelevant, resulting in wasted computational resources.

2.4 Unnecessary Overhead

All updates are routed through the Geyser plugin subsystem, despite the absence of active plugins. This introduces unnecessary computational overhead and latency without delivering tangible benefits.

3. Proposed Architecture

3.1 Core Architectural Changes

This RFC proposes a complete decoupling of the RPC subsystem from the validator core:

1. Service Separation: Move all user-facing components (HTTP RPC, WebSocket RPC, and Geyser plugin API) into a dedicated project with its own codebase and dependencies.
2. Validator Simplification: Refocus the validator on its core responsibilities: transaction execution and state persistence, significantly reducing complexity and maintenance burden.
3. Independent Deployment: Compile and deploy the RPC subsystem as a standalone process, co-located with the validator.
4. Inter-Process Communication: Establish communication between the validator and RPC processes via high-performance shared memory channels.

3.2 Technical Implementation Details

3.2.1 Shared State

Both the validator and RPC service will access the same underlying databases: accountsdb and the ledger. As these databases are memory-mapped, they can be opened by multiple processes:

• accountsdb: Designed for concurrent access, with the validator having read-write access and the RPC service operating in read-only mode.
• Ledger (RocksDB): While technically possible to open in shared mode, RocksDB readers only see a static snapshot at the time of opening. Any updates require reopening the database, which is impractical under high load. A potential rewrite or simplification of the ledger storage layer may be necessary to fully realize the benefits of this architecture.

3.2.2 Communication Protocol

Communication between the validator and RPC service will utilize a shared memory queue, implemented as a fixed-length array with OS-provided synchronization. The shmemq crate provides an initial implementation. Each message type will have its own dedicated channel for efficiency. The primary message types are:

1. Account Update: (pubkey) Validator → RPC
2. Signature Update: (signature) Validator → RPC
3. Block Update: (block object) Validator → RPC
4. Transaction Execution/Simulation Request: RPC → Validator
5. Transaction Execution/Simulation Result: Validator → RPC

3.2.3 Validator-RPC Interface Design

For each system event, unique information will be communicated between the validator and RPC service:

1. Account Update: When an account is updated, the pubkey is sent to the RPC service, which checks for interested consumers. If any exist, the account is read from accountsdb and delivered directly to the consumer (e.g., a WebSocket subscription or Geyser plugin). Otherwise, no further action is taken.
2. Transaction Execution: Upon transaction execution, the signature and a success flag are sent to the RPC service. The RPC service checks for registered consumers:
   • If a WebSocket subscription exists and the transaction succeeded, the client is notified immediately.
   • If the transaction failed, the RPC service retrieves the error message from the database and notifies the client.
   • If a Geyser plugin is registered, the transaction data is read from the database and delivered to the plugin.
3. Block Production: When a new block is produced, the RPC service checks for blockSubscribe or slotSubscribe subscriptions and notifies interested consumers directly, same applies for any registered geyser plugin.
4. Transaction Request: For sendTransaction or simulateTransaction requests, the RPC service parses and decodes the transaction, then sends it to the validator for execution or simulation.
5. Simulation Result: Simulation results, including logs and error messages, are sent back to the RPC service and delivered to the client, without querying the database.

Additional Notes:

• For database reads, required pages are likely to be in memory due to recent writes by the validator due to transaction execution, minimizing latency.
• For frequently used requests like getSignatureStatuses, the RPC service will cache recent transaction results and serve them directly, bypassing the database.

3.2.4 Geyser Plugin Support

Solana provides several crates for Geyser plugin integration. The proposed RPC service will only require the interface crate, allowing it to dynamically load shared object plugins and invoke their methods as needed. If no plugins are registered, the RPC service incurs no additional overhead.

3.3 Deployment Model

The RPC service will be deployed alongside the validator on the same host, ensuring low-latency communication while maintaining process isolation.

4. Benefits

4.1 Separation of Concerns

This architectural split enforces a clear separation of responsibilities:

• Validator: Transaction execution and state management
• RPC Service: Client communication and API exposure

4.2 Independent Evolution

Decoupling the codebases allows each to evolve independently, avoiding dependency conflicts and enabling targeted optimizations. This addresses previous challenges such as dependency management issues caused by tight coupling.

4.3 Performance Improvements

The proposed architecture enables:

• Optimized, purpose-built RPC implementations unconstrained by validator requirements
• Efficient event filtering and targeted distribution
• Reduced memory pressure on the validator process
• Potential for specialized scaling strategies for RPC workloads

4.4 Improved Maintainability

Smaller, focused codebases are easier to understand, test, and maintain. This separation reduces cognitive load for developers and accelerates iteration.

5. Risks and Mitigations

5.1 Increased Deployment Complexity

Risk: Operating two processes instead of one increases operational complexity.
Mitigation: Develop comprehensive deployment tooling and documentation to ensure correct configuration and co-location of the processes.

5.2 Consistency Guarantees

Risk: In accountsdb, it is theoretically possible for an account to be modified by the validator while being read by the RPC service.
Mitigation: Employ a synchronization primitive such as SeqLock (already integrated into the accountsdb storage format) to allow readers to retry reads if concurrent modifications are detected.

6. Implementation Plan

The implementation will proceed in three primary stages:

1. Standalone RPC Service: Develop the RPC service as an independent project, leveraging its independence from the validator's codebases for rapid development.
2. Validator Refactoring: Remove all RPC and Geyser-related functionality from the validator, simplifying the Bank and TransactionProcessor implementations.
3. Integration: Establish the necessary shared memory channels and startup logic to integrate the validator and RPC service.

The development effort is estimated at approximately three developer-weeks, with an additional two weeks for review and feedback.

7. Current Blockers

As noted, RocksDB’s limitations in shared mode present a significant blocker. A rewrite or replacement of the ledger storage layer is required before the RPC service can be fully implemented.

8. Conclusion

Decoupling the RPC subsystem from the MagicBlock validator core represents a significant architectural improvement. This approach will deliver better separation of concerns, improved maintainability, and enhanced performance, aligning with modern software engineering best practices and positioning the system for sustainable long-term evolution.

[taco-paco]

I will go over this in more detail this week.
But I think we need to define: what problem are we solving with shared memory approach

1. Is that slow RPC implementation
2. some issue with geyser plugin?

Slow RPC is due to inoptimial implementation of it, switching to shared memory approach won't help here, the RPC implementation needs to be changed itself.

Is problem we trying to solve is a geyser plugin? I have real doubt that it's, it is just a function call with extra jump.

If our goal is to make RPC faster I would suggest fixing suboptimal RPC implementation itself.

1. It would be less dramatic change
2. We can switch easily to shared memory after. Just need to abstract transport layer of how updates get to us, via dll call or via msg in queue

trait IncomingUpdatesProvider {
    async fn get_account_updates() -> Result<Vec<(ReplicaAccountInfoVersions, Slot, bool)>>
    async  fn get_slot_status_updates() -> Result<Vec<(Slot, Option<u64>, SlotStatus)>>
    /// so on...
}

// or push version
trait Updater {
    fn update_account(
        &self,
        account: ReplicaAccountInfoVersions,
        slot: Slot,
        is_startup: bool,
    ) -> Result<()>;

    #[allow(unused_variables)]
    fn update_slot_status(
        &self,
        slot: Slot,
        parent: Option<u64>,
        status: &SlotStatus,
    ) -> Result<()>;
}

[bmuddha]

Most and foremost it is about an isolation of RPC codebase along with its complicated dependency graph from the validator core. It's just a matter of better maintainability, at the same time not sacrificing performance. Admittedly we can improve the RPC without moving it out into separate codebase, but the whole process is going to be more dev-time intensive and won't solve maintainability issues.

[taco-paco]

If this is for sake of isolation, imo isolating it as regular geyser plugin, compiling to dll is a much cheaper endeavor. It would be then loaded following regular plugin workflow

[bmuddha]

There're tons of reasons for why this won't work, e.g. lack of bidirectional communication, ABI incompatibility issues, extreme difficulty of performance tuning, strict need to conform to the geyser interface and I can go on, but that's definitely not a direction that we should take.

[taco-paco]

The communication shall be one directional, that service shall not have any affect on validator. The service just needs to get updates and manipulate data to give it to user.
ABI compatibility shouldn't be a big issue since both libs are on Rust. It's just a matter of CI setup that will detect if your rust version has incompatible ABI. I also see that most of structs use C ABI with #[repr(C)], tho functions defined without extern "C"
Confomring to an interface is a good thing imo.

There're issue with shared memory approach:
While shared memory is fast for data transfer, in queue implementation synchronization costs could negate these benefits. I also have a gut feeling that dll calls are faster.
Cache coherency issues between multiple processes for same memory regions.
Complex memory management.
Risk of memory leaks if one process crashes.
Debugging shared memory issues is notoriously difficult
More complex deployment.
Error handling.