Multi-staged sync & import-db (aka "node versions manager")

[andreibancioiu]

These scripts allow one to use multiple versions of the MultiversX node, in sequence, to sync (from the deep past) or run import-db flows.

Primary reason: at some point, we'd like to drop legacy code from the newest (current) versions of the Node. Though, we have to be able to still process blocks from the deep past within some flows such as import-db, sync from genesis etc.

Configuration files:

• build configuration
• driver configuration (lanes and stages)

Notes:

• generally speaking, a processing lane would correspond to a given network shard (this is not a constraint, though). A lane has more stages - each stage references a specific configuration & node binary. A stage runs until a specific condition (configured) is met. Then, the next stage on the lane follows and so on.
• in order to enable ES indexing, pass a custom prefs.toml using the node arguments, in the driver configuration file.

Also see: README.

[Copilot]

Pull Request Overview

This PR adds a multi-stage synchronization and import pipeline for MultiversX nodes, enabling sequential use of different node versions.

• Introduces Python controllers and utilities to manage node stages and lanes
• Provides CLI scripts (build.py, driver.py) with sample JSON configs for build and testnet flows
• Updates project linting (pyrightconfig.json, .flake8) and documentation (README.md)

Reviewed Changes

Copilot reviewed 14 out of 16 changed files in this pull request and generated 2 comments.

Show a summary per file

File	Description
pyrightconfig.json	Setup for Pyright type checking in the multistage dir
.flake8	Flake8 config to ignore E501 (line-length)
multistage/config.py	Defines BuildConfigEntry, DriverConfig, LaneConfig, StageConfig schemas
multistage/constants.py	Adds shared constants (process limits, file modes, etc.)
multistage/errors.py	Custom exception hierarchy for controlled failures
multistage/shared.py	fetch_archive utility to download and unpack archives
multistage/stage_controller.py	StageController to configure, start/stop, and monitor a node stage
multistage/lane_controller.py	LaneController to sequence through multiple stages
multistage/golang.py	Go environment acquisition and build helper functions
multistage/driver.py	CLI driver to run a lane using a driver JSON config
multistage/build.py	CLI tool to build node binaries from Go sources
multistage/driver.testnet.json	Sample testnet lane/driver configuration
multistage/build.json	Sample build definitions for different node versions
multistage/README.md	Instructions for setup, build, and running the driver

Comments suppressed due to low confidence (3)

multistage/config.py:114

• The parameter name bin shadows Python's built-in bin() function. Consider renaming it to something like bin_path for clarity.

                 bin: str,

multistage/shared.py:13

• fetch_archive implements critical I/O and archive unpacking logic but has no corresponding unit tests. Consider adding tests for success paths and error scenarios.

def fetch_archive(archive_url: str, destination_path: Path):

multistage/constants.py:3

• METACHAIN_ID is introduced but not referenced in this module or elsewhere. Remove it or add a usage example to avoid dead code.

METACHAIN_ID = 4294967295

[andreibancioiu]

See multiversx/mx-chain-multistage-scripts#1.