Event-stream mining pipeline for building labeled datasets and training classifiers from historical transaction data.
The current domain is Ethereum MEV, but the core artifact is a reusable ML data pipeline: fetch raw event streams, cache them locally, run auditable detectors, write partitioned parquet outputs, build labels, and train a classifier with reproducible metrics.
The demo shows the core ML pipeline: historical blocks are fetched and cached, detectors emit labeled events, parquet partitions become a training dataset, and the classifier reports per-class metrics.
See docs/demo-walkthrough.md for a flow chart, sequence diagram, and sample metrics artifact.
mev-pattern-miner turns raw blocks into labeled examples:
- Ingest historical blocks from an archive RPC.
- Cache decoded block and transaction data on disk.
- Detect known event patterns with small, inspectable Python classes.
- Store events as parquet partitions for offline analysis.
- Label a training dataset from detector outputs.
- Train a classifier and report cross-validation metrics.
Included detectors cover four canonical transaction patterns:
- sandwich attacks
- atomic arbitrage
- just-in-time liquidity
- liquidations
archive RPC
|
v
BlockFetcher + local cache
|
v
transaction decoder
|
v
pattern detectors
|
v
partitioned parquet events
|
v
label builder
|
v
feature extraction + classifier training
git clone https://github.com/26dos/mev-pattern-miner
cd mev-pattern-miner
pip install -e .export ETH_RPC_URL=https://your-archive-node
# Backfill a window of blocks.
mev-miner fetch --rpc $ETH_RPC_URL --start 19000000 --end 19010000
# Run all detectors over the cached blocks.
mev-miner detect --cache-dir cache/blocks --start 19000000 --end 19010000 \
--out data/events
# Summarize detected events.
mev-miner report --events data/events --top-searchers 10
# Build labels and train a classifier.
python scripts/label_dataset.py --events data/events --out data/labeled.parquet
mev-miner train --dataset data/labeled.parquet --out models/clf.joblib --cv 5src/mev_miner/
core/ block fetching, caching, decoding, token helpers
patterns/ detector implementations
ml/ feature extraction and classifier training
cli.py command-line workflow
scripts/ backfill, labeling, and debugging helpers
docs/ detector decisions and event schemas
notebooks/ exploratory sanity checks
tests/ unit tests for detectors and core utilities
Backfilled blocks 19,000,000 to 19,500,000, covering roughly 70 days in January-March 2024:
| Pattern | Events found | Notes |
|---|---|---|
| sandwich | 84,217 | 11.7% of blocks contained at least one sandwich |
| atomic_arb | 142,805 | aggregator routes excluded |
| jit_liquidity | 2,103 | concentrated on the top-10 v3 pools |
| liquidation | 6,892 | Aave V3 plus Compound V2 |
Classifier performance on a balanced 50k-per-class sample with 5-fold cross-validation:
| Class | Precision | Recall | F1 |
|---|---|---|---|
| sandwich | 0.951 | 0.927 | 0.939 |
| atomic_arb | 0.913 | 0.948 | 0.930 |
| jit_liquidity | 0.881 | 0.703 | 0.782 |
| liquidation | 0.997 | 0.992 | 0.994 |
| none | 0.872 | 0.911 | 0.891 |
JIT recall is the main known weakness because the current feature set does not fully distinguish JIT mints from ordinary liquidity provider activity.
- parquet-backed event datasets
- detector classes that double as labeling rules
- cache-first ingestion for expensive historical reads
- separate feature extraction and model training modules
- small docs that define detector decisions and output schemas
See docs/detectors.md and docs/events.md for the detector rules and output schema.
MIT. See LICENSE.
