Retro Funding Season Data and Algorithms

Quickstart

Clone the repo:

gh repo clone ethereum-optimism/Retro-Funding

Navigate to the repo:

cd Retro-Funding

Install the dependencies:

poetry install

Activate the virtual environment:

poetry env activate

Test the pipeline:

./run_pipeline.sh 7 M6 --skip-fetch

Run the pipeline with data fetching:

./run_pipeline.sh 8 M7

Simulation Pipeline

For each season (S7, S8) and measurement period (M1, M2, etc.), the maintainers of this repository fetch a snapshot of the data, review the model weights, and run the scoring algorithm.

Directory Structure

The project directory is organized as follows:

├── docs/                    # Documentation
├── eval-algos/              # Core evaluation algorithms
│   ├── core/
│   │   ├── models/          # Model implementations
|   |   |   ├── allocator.py 
|   |   |   ├── devtooling.py
|   |   |   └── onchain_builders.py
│   │   └── utils/           # Utility functions and scripts
│   └── queries/             # Season-specific query files
|       ├── s7_queries.py 
|       └── s8_queries.py
├── results/                 
│   ├── S7/                  # Season 7 results
│   │   ├── M1/              # February 2025
│   │   │   ├── data/        # Input data snapshots
│   │   │   ├── weights/     # Model YAML config files
│   │   │   └── outputs/     # Output results
│   │   └── ...
│   └── S8/                  # Season 8 results
│       ├── M7/              # August 2025
│       │   ├── data/        
│       │   ├── weights/     
│       │   └── outputs/     
│       └── ...
└── run_pipeline.sh          # Main pipeline orchestration script

Data Snapshots

We add raw data snapshots in the data directory associated with each measurement period. The following files are included for each measurement period:

• onchain__project_metadata.csv: Metadata about each project that applied to the Onchain Builders round.
• onchain__metrics_by_project.csv: Chain-level, month-by-month metrics about each project in the Onchain Builders round.
• devtooling__project_metadata.csv: Metadata about each project that applied to the Devtooling round.
• devtooling__onchain_metadata.csv: Metadata about each onchain project that was linked to a devtooling project.
• devtooling__dependency_graph.csv: A graph of the dependencies between devtooling and onchain projects.
• devtooling__developer_graph.csv: A graph of the developers who have committed code to onchain projects and engaged with devtooling projects.
• devtooling__raw_metrics: A summary of the raw (graph-level) metrics for each devtooling project, in JSON format.

Note that the project_id for each project is generated by OSO and is not the same as the project_id used by OP Atlas. To get the OP Atlas identifier for a project, query the project_name field in the project_metadata tables. The display_name is the name of the project as it will appear on OP Atlas's website.

Important

The reason for using snapshots instead of live data is because the underlying mappings between projects and their artifacts (i.e., contracts, repos, packages, etc.) are continuously changing. Therefore, each snapshot represents the state of which projects are enrolled in the round, which artifacts are associated with those projects, and which metrics are publicly available about those artifacts.

You can also query Retro Funding data directly from OSO by following the guides here.

There are test datasets as well for testing the simulation pipeline.

Fetching Data

To fetch the latest data for a specific measurement period, you need to:

1. Add your OSO API key to a .env file in the root directory:

   OSO_API_KEY=your_api_key_here
   

2. Run the data_fetcher.py script with the desired season and measurement period. For example:

   poetry run python -m eval-algos.core.utils.data_fetcher --season 8 --period M7

This will download the data from OSO and save it to the results/S8/M7/data/ directory. The script will fetch all the necessary CSV files for both the onchain builders and devtooling rounds.

Weights

The weights for each model are stored in the results/S8/<measurement_period>/weights directory.

Be sure to use the right data snapshot and simulation period for your simulation.

data_snapshot:
  data_dir: 'results/S8/M7/data/'
  projects_file: 'onchain__project_metadata.csv'
  metrics_file: 'onchain__metrics_by_project.csv'

simulation:
  periods:
    previous: 'Jul 2025'
    current: 'Aug 2025'

You can also modify the weights for each model as you see fit. Note that new metrics may be added throughout the season. For example, the onchain__metrics_by_project.csv in M2 includes new metrics related to Worldchain activity.

  metrics:
    amortized_contract_invocations_monthly: 0.25
    gas_fees_monthly: 0.25
    average_tvl_monthly: 0.25
    active_farcaster_users_monthly: 0.25
    
    qualified_addresses_monthly: 0.0
    active_addresses_monthly: 0.0
    contract_invocations_monthly: 0.0
  
  # Variant weights must sum to 1.0
  metric_variants:
    adoption: 0.20    # Current period value
    growth: 0.20      # Change from previous period
    retention: 0.60   # Minimum of current and previous

Allocation Settings

Each model configuration file includes allocation settings that determine how funding is distributed:

allocation:
  budget: 1333333.3333333333  # In OP tokens
  min_amount_per_project: 200 # In OP tokens
  max_share_per_project: 0.05 # Percentage of the total budget

These settings control:

• The total budget available for allocation
• The minimum amount each project can receive
• The maximum share of the budget any single project can receive

Run the Pipeline

The Retro Funding pipeline follows this workflow:

1. Data Fetching: Fetch raw data from OSO for the specified season and measurement period
2. Model Execution: Run the devtooling and/or onchain models with specified weights
3. Reward Allocation: Apply the allocation algorithm to distribute rewards based on model scores
4. Consolidation: Combine results from multiple models into a single file
5. Serialization: Create JSON files for OP Atlas integration

The main pipeline can be run using the run_pipeline.sh script, which orchestrates the entire process:

# Run both models for a season/period
./run_pipeline.sh 7 M6 --skip-fetch

# Run a specific model
./run_pipeline.sh 7 M6 --skip-fetch --algo devtooling --weights devtooling__arcturus

# Run with data fetching (default)
./run_pipeline.sh 8 M7 

Note: S7 models have been disabled, so you can only use the snapshot datasets.

Alternatively, you can run individual models directly:

# Run devtooling model
poetry run python -m eval-algos.core.utils.process_models --algo devtooling --weights devtooling__arcturus --season 7 --period M6

# Run onchain model
poetry run python -m eval-algos.core.utils.process_models --algo onchain --weights onchain__goldilocks --season 7 --period M6

This will:

1. Load the model configuration from results/S7/M6/weights/<model>.yaml
2. Process the data according to the model's algorithm
3. Allocate rewards using the allocation algorithm
4. Save the results to results/S7/M6/outputs/<model>_rewards.csv

Consolidating and Serializing Results

After running the simulation pipeline, you can consolidate and serialize the results using the following utilities:

Consolidating Rewards

The consolidate_rewards.py script combines all rewards files from a measurement period into a single CSV file:

poetry run python -m eval-algos.core.utils.consolidate_rewards --season 7 --period M6

This will:

1. Find all rewards CSV files in the results/S7/M6/outputs/ directory
2. Standardize the format (ensuring consistent column names)
3. Add filename, season, and period columns
4. Save the consolidated data to results/S7/M6/outputs/M6_consolidated_rewards.csv

Serializing Results

The serialize.py script creates JSON files that combine metrics and rewards data:

poetry run python -m eval-algos.core.utils.serialize --season 7 --period M6

This will:

1. Use the consolidated rewards file (or create it if it doesn't exist)
2. Create devtooling__results.json with devtooling metrics merged with rewards
3. Create onchain__results.json with onchain metrics merged with rewards

These JSON files are used for displaying results on OP Atlas and for further analysis.

Model Details

Onchain Builders

The Onchain Builders model analyzes and scores active Superchain projects based on their onchain activity. It processes raw metrics through a multi-step pipeline to produce normalized, weighted scores that reflect project performance across different chains and time periods.

Pipeline Steps

1. Filter and Pivot Raw Metrics

• Takes raw metrics data with non-zero weights for specified measurement periods
• Pivots data by chain and metric to create a structured view
• Groups by project_id, project_name, display_name, and chain

2. Sum and Weight by Chain

• Applies chain-specific weights (e.g., different weights for OP Mainnet vs other chains)
• Sums weighted metrics across all chains for each project
• Preserves project metadata in the aggregation

3. Calculate Metric Variants

For each metric, computes three variants:

• Adoption: Current period value
• Growth: Positive difference between current and previous period values
• Retention: Minimum value between current and previous periods

4. Normalize Metric Variants

• Applies min-max normalization to each metric variant
• Scales values to [0,1] range while preserving null values (e.g., TVL for non-DeFi projects)
• Optionally caps values at a specified percentile (e.g., 97th percentile)
• Uses fallback center value (0.5) when range is zero

5. Apply Weights

• Multiplies each normalized metric variant by:
  • Its metric-specific weight
  • Its variant-specific weight (adoption/growth/retention)

6. Aggregate Final Scores

• Combines weighted variants using power mean (p=2)
• Normalizes final scores to sum to 1.0

7. Prepare Results

• Flattens multi-level columns for readability
• Merges intermediate results for transparency
• Sorts projects by final weighted score

Allocation Algorithm

The allocation algorithm (allocator.py) distributes rewards based on project scores while respecting constraints:

• Budget: Total amount available for distribution
• Minimum Amount: Minimum reward per project
• Maximum Share: Maximum percentage of budget any single project can receive

The algorithm uses a constrained optimization approach to ensure fair and efficient distribution of rewards.

Devtooling OpenRank

The Devtooling model evaluates open-source developer tools by analyzing their relationships with onchain projects through package dependencies and developer contributions. It uses an EigenTrust-based algorithm released by OpenRank to propagate value through the graph.

Pipeline Steps

1. Build Unweighted Graph

Constructs a directed graph with three types of edges:

• Package Dependencies: Onchain projects → Devtooling projects
• Commit Events: Onchain projects → Developers
• GitHub Engagement: Developers → Devtooling projects

Removes duplicate edges when an onchain project is also a devtooling project.

2. Compute Onchain Project Pretrust

• Uses economic metrics from onchain projects
• Applies log transformation and min-max scaling
• Combines metrics using configured weights
• Normalizes scores to sum to 1.0

3. Compute Devtooling Project Pretrust

• Uses GitHub metrics (num packages, stars, forks, etc.)
• Applies log transformation and min-max scaling
• Combines metrics using configured weights
• Normalizes scores to sum to 1.0

4. Compute Developer Reputation

• Developers are pre-filtered to focus on active developers committing code to onchain project repos in Rust, Solidity, TypeScript, and Vyper
• Distributes onchain project pretrust to developers based on commit activity

5. Weight Edges

Applies weights based on:

• Link type (package dependency, commit, GitHub engagement)
• Event type (NPM, CARGO, COMMIT_CODE)
• Time decay for non-static relationships
• More recent interactions contribute more weight in most cases

6. Apply EigenTrust

• Combines pretrust scores from all sources
• Runs EigenTrust propagation on weighted graph
• Computes final trust scores for each node

7. Rank and Evaluate Projects

• Ranks devtooling projects by final EigenTrust scores
• Applies eligibility criteria:
  • Minimum package dependency count
  • Minimum developer link count
• Normalizes scores among eligible projects

8. Serialize Value Flow

• Uses iterative proportional fitting (IPF)
• Creates detailed value flow attribution
• Ensures contribution sums match:
  • Per devtool: Sum equals its overall score
  • Per onchain project: Sum equals its pretrust