Skip to content

About

Event-stream mining pipeline for labeled parquet datasets, detector-based labels, and classifier training.

Topics

Resources

Stars

2 stars

Watchers

0 watching

Forks

Repository files navigation

mev-pattern-miner

Event-stream mining pipeline for building labeled datasets and training classifiers from historical transaction data.

The current domain is Ethereum MEV, but the core artifact is a reusable ML data pipeline: fetch raw event streams, cache them locally, run auditable detectors, write partitioned parquet outputs, build labels, and train a classifier with reproducible metrics.

Demo Snapshot

MEV event mining pipeline demo

The demo shows the core ML pipeline: historical blocks are fetched and cached, detectors emit labeled events, parquet partitions become a training dataset, and the classifier reports per-class metrics.

See docs/demo-walkthrough.md for a flow chart, sequence diagram, and sample metrics artifact.

What It Does

mev-pattern-miner turns raw blocks into labeled examples:

  1. Ingest historical blocks from an archive RPC.
  2. Cache decoded block and transaction data on disk.
  3. Detect known event patterns with small, inspectable Python classes.
  4. Store events as parquet partitions for offline analysis.
  5. Label a training dataset from detector outputs.
  6. Train a classifier and report cross-validation metrics.

Included detectors cover four canonical transaction patterns:

  • sandwich attacks
  • atomic arbitrage
  • just-in-time liquidity
  • liquidations

Pipeline

archive RPC
    |
    v
BlockFetcher + local cache
    |
    v
transaction decoder
    |
    v
pattern detectors
    |
    v
partitioned parquet events
    |
    v
label builder
    |
    v
feature extraction + classifier training

Quick Start

git clone https://github.com/26dos/mev-pattern-miner
cd mev-pattern-miner
pip install -e .
export ETH_RPC_URL=https://your-archive-node

# Backfill a window of blocks.
mev-miner fetch --rpc $ETH_RPC_URL --start 19000000 --end 19010000

# Run all detectors over the cached blocks.
mev-miner detect --cache-dir cache/blocks --start 19000000 --end 19010000 \
                 --out data/events

# Summarize detected events.
mev-miner report --events data/events --top-searchers 10

# Build labels and train a classifier.
python scripts/label_dataset.py --events data/events --out data/labeled.parquet
mev-miner train --dataset data/labeled.parquet --out models/clf.joblib --cv 5

Repository Layout

src/mev_miner/
  core/          block fetching, caching, decoding, token helpers
  patterns/      detector implementations
  ml/            feature extraction and classifier training
  cli.py         command-line workflow

scripts/         backfill, labeling, and debugging helpers
docs/            detector decisions and event schemas
notebooks/       exploratory sanity checks
tests/           unit tests for detectors and core utilities

Sample Results

Backfilled blocks 19,000,000 to 19,500,000, covering roughly 70 days in January-March 2024:

Pattern Events found Notes
sandwich 84,217 11.7% of blocks contained at least one sandwich
atomic_arb 142,805 aggregator routes excluded
jit_liquidity 2,103 concentrated on the top-10 v3 pools
liquidation 6,892 Aave V3 plus Compound V2

Classifier performance on a balanced 50k-per-class sample with 5-fold cross-validation:

Class Precision Recall F1
sandwich 0.951 0.927 0.939
atomic_arb 0.913 0.948 0.930
jit_liquidity 0.881 0.703 0.782
liquidation 0.997 0.992 0.994
none 0.872 0.911 0.891

JIT recall is the main known weakness because the current feature set does not fully distinguish JIT mints from ordinary liquidity provider activity.

Reusable Engineering Patterns

  • parquet-backed event datasets
  • detector classes that double as labeling rules
  • cache-first ingestion for expensive historical reads
  • separate feature extraction and model training modules
  • small docs that define detector decisions and output schemas

See docs/detectors.md and docs/events.md for the detector rules and output schema.

License

MIT. See LICENSE.

About

Event-stream mining pipeline for labeled parquet datasets, detector-based labels, and classifier training.

Topics

Resources

Stars

2 stars

Watchers

0 watching

Forks

Releases

Packages

Contributors

Languages