modestbench

“A full-ass benchmarking framework for Node.js”
by @boneskull

Features

• Fast & Accurate: High-precision timing with statistical analysis
• Multiple Output Formats: Human-readable, JSON, and CSV reports (at the same time!!)
• Historical Tracking: Store and compare benchmark results over time
• Tagging System: Organize and filter benchmarks by categories
• CLI & API: Command-line interface and programmatic API
• TypeScript Support: Full type safety

In summary, modestbench wraps tinybench and enhances it with a bunch of crap so you don't have to think.

Quick Start

Installation

The usual suspects:

npm install --save-dev modestbench

Optional: Initialize a Project

The modestbench CLI provides a init command. This command:

1. Generates a configuration file in a format of your choosing
2. Creates an example benchmark file
3. Appends .modestbench/ to .gitignore to exclude historical data from version control

# Initialize with examples and configuration
modestbench init

# Or specify project type and config format
modestbench init advanced --config-type typescript

Project Types:

• basic - Simple setup for small projects (100 iterations, human reporter)
• advanced - Feature-rich with multiple reporters and structured output (1000 iterations, warmup, human + JSON reporters)
• library - Optimized for library performance testing (5000 iterations, high warmup, human + JSON reporters, organized suite structure)

My First Benchmark

PRO TIP: The convention for modestbench benchmark files is to use the .bench.js or .bench.ts extension.

modestbench supports two formats for defining benchmarks:

Simplified Format (Recommended for Simple Cases)

For quick benchmarks with just a few tasks, you can use the simplified format:

// benchmarks/example.bench.js
export default {
  'Array.push()': () => {
    const arr = [];
    for (let i = 0; i < 1000; i++) {
      arr.push(i);
    }
    return arr;
  },

  'Array spread': () => {
    let arr = [];
    for (let i = 0; i < 1000; i++) {
      arr = [...arr, i];
    }
    return arr;
  },
};

Suite-Based Format (For Complex Projects)

When you need to organize benchmarks into groups with setup/teardown hooks:

// benchmarks/example.bench.js
export default {
  suites: {
    'Array Operations': {
      benchmarks: {
        'Array.push()': () => {
          const arr = [];
          for (let i = 0; i < 1000; i++) {
            arr.push(i);
          }
          return arr;
        },

        'Array spread': () => {
          let arr = [];
          for (let i = 0; i < 1000; i++) {
            arr = [...arr, i];
          }
          return arr;
        },
      },
    },
  },
};

When to use each format:

• Simplified format: Quick benchmarks, single file with related tasks, no setup/teardown needed
• Suite format: Complex projects, multiple groups of benchmarks, need setup/teardown hooks, or want explicit organization

Running Your First Benchmarks

# Run all benchmarks
modestbench

# Run with specific options
modestbench --iterations 5000 --reporters human,json

View Results

Getting Started

Jump to:

• Quick Start - Basic concepts and your first benchmark
• Configuration - Project and runtime configuration options
• Advanced Features - Multiple suites, async operations, and tagging
• Integration Examples - CI/CD integration and performance monitoring
• Programmatic API - Using modestbench programmatically

See the examples directory for additional guides and sample code.

CLI Commands

Run Benchmarks

Run benchmarks with sensible defaults:

# Run benchmarks in current directory and bench/ (top-level only)
modestbench

# Run all benchmarks in a directory (searches recursively)
modestbench benchmarks/

# Run benchmarks from multiple directories
modestbench src/perf/ tests/benchmarks/

# Run specific files
modestbench benchmarks/critical.bench.js

# Mix files, directories, and glob patterns
modestbench file.bench.js benchmarks/ "tests/**/*.bench.ts"

# With options
modestbench \
  --config ./config.json \
  --iterations 2000 \
  --reporters human,json,csv \
  --output ./results \
  --tags performance,algorithm \
  --concurrent

Supported file extensions:

• JavaScript: .js, .mjs, .cjs
• TypeScript: .ts, .mts, .cts

Controlling Benchmark Limits

The --limit-by flag controls whether benchmarks are limited by time, iteration count, or both:

# Limit by iteration count (fast, predictable sample size)
modestbench --iterations 100

# Limit by time budget (ensures consistent time investment)
modestbench --time 5000

# Limit by whichever comes first (safety bounds)
modestbench --iterations 1000 --time 10000

# Explicit control (overrides smart defaults)
modestbench --iterations 500 --time 5000 --limit-by time

# Require both thresholds (rare, for statistical rigor)
modestbench --iterations 100 --time 2000 --limit-by all

Smart Defaults:

• Only --iterations provided → limits by iteration count (fast)
• Only --time provided → limits by time budget
• Both provided → stops at whichever comes first (any mode)
• Neither provided → uses default iterations (100) with iterations mode

Modes:

• iterations: Stop after N samples (time budget set to 1ms)
• time: Run for T milliseconds (collect as many samples as possible)
• any: Stop when either threshold is reached (defaults to iterations behavior for fast completion)
• all: Require both time AND iterations thresholds (tinybench default behavior)

Filtering by Tags

Run specific subsets of benchmarks using tags:

# Run only benchmarks tagged with 'fast'
modestbench --tags fast

# Run benchmarks with multiple tags (OR logic - matches ANY)
modestbench --tags string,array,algorithm

# Exclude specific benchmarks
modestbench --exclude-tags slow,experimental

# Combine: run fast benchmarks except experimental ones
modestbench --tags fast --exclude-tags experimental

Key Features:

• Tags cascade from file → suite → task levels
• --tags uses OR logic (matches ANY specified tag)
• --exclude-tags takes precedence over --tags
• Suite setup/teardown only runs if at least one task matches

See Tagging and Filtering for detailed examples.

Output Options

Control where and how benchmark results are saved:

# Write to a directory (creates results.json, results.csv, etc.)
modestbench --reporters json,csv --output ./results

# Custom filename for single reporter
modestbench --reporters json --output-file my-benchmarks.json

# Custom filename in specific directory
modestbench --reporters json --output ./results --output-file benchmarks-2024.json

# Custom filename with absolute path
modestbench --reporters json --output-file /tmp/my-benchmarks.json

# With subdirectories
modestbench --reporters csv --output ./results --output-file reports/performance.csv

# Short flag alias
modestbench --reporters json --of custom.json

Key Options:

• --output <dir>, -o <dir> - Directory to write output files (default: stdout)
• --output-file <filename>, --of <filename> - Custom filename for output
  • Works with absolute or relative paths
  • Requires exactly one reporter (e.g., --reporters json)
  • When used with --output, the filename is relative to that directory
  • When used alone, the path is relative to current working directory

Limitations:

• --output-file only works with a single reporter
• For multiple reporters, use --output <dir> (defaults to results.json, results.csv, etc.)

History Management

modestbench automatically tracks benchmark results over time in a local .modestbench/ directory. This history enables you to:

• Track performance trends - See how your code's performance changes across commits
• Detect regressions - Compare current results against previous runs to catch slowdowns
• Analyze improvements - Validate that optimizations actually improve performance
• Document progress - Export historical data for reports and analysis

# List recent runs
modestbench history list

# Show detailed results
modestbench history show <run-id>

# Compare two runs
modestbench history compare <run-id-1> <run-id-2>

# Export historical data
modestbench history export --format csv --output results.csv

# Clean old data
modestbench history clean --older-than 30d

Configuration

Project Configuration

Create modestbench.config.json:

{
  "bail": false, // Stop execution on first failure
  "exclude": ["node_modules/**"], // Patterns to exclude from discovery
  "excludeTags": ["slow", "experimental"], // Tags to exclude from execution
  "iterations": 1000, // Number of samples per benchmark
  "limitBy": "iterations", // Limit mode: 'iterations', 'time', 'any', 'all'
  "outputDir": "./benchmark-results", // Directory for results and reports
  "pattern": "benchmarks/**/*.bench.{js,ts}", // Glob pattern to discover benchmark files
  "quiet": false, // Minimal output mode
  "reporters": ["human", "json"], // Output reporters to use
  "tags": ["fast", "critical"], // Tags to include (if empty, all benchmarks run)
  "time": 5000, // Time budget in ms per benchmark
  "timeout": 30000, // Task timeout in ms
  "verbose": false, // Detailed output with debugging info
  "warmup": 50, // Warmup iterations before measurement
}

Configuration Options:

• pattern - Glob pattern(s) to discover benchmark files (can be string or array)
• exclude - Glob patterns for files/directories to exclude from discovery
• excludeTags - Array of tags to exclude from execution; benchmarks with ANY of these tags will be skipped (default: [])
• iterations - Number of samples to collect per benchmark task (default: 100)
• time - Time budget in milliseconds per benchmark task (default: 1000)
• limitBy - How to limit benchmarks: "iterations" (sample count), "time" (time budget), "any" (whichever comes first), or "all" (both thresholds required)
• warmup - Number of warmup iterations before measurement begins (default: 0)
• timeout - Maximum time in milliseconds for a single task before timing out (default: 30000)
• bail - Stop execution on first benchmark failure (default: false)
• reporters - Array of reporter names to use for output (available: "human", "json", "csv")
• outputDir - Directory path for saving benchmark results and reports
• quiet - Minimal output mode, suppresses non-essential messages (default: false)
• tags - Array of tags to include; if non-empty, only benchmarks with ANY of these tags will run (default: [])
• verbose - Detailed output mode with additional debugging information (default: false)

Note: Smart defaults apply for limitBy based on which options you provide. See Controlling Benchmark Limits for details.

Configuration File Support

modestbench supports multiple configuration file formats, powered by cosmiconfig:

• JSON: modestbench.config.json, .modestbenchrc.json, .modestbenchrc
• YAML: modestbench.config.yaml, modestbench.config.yml, .modestbenchrc.yaml, .modestbenchrc.yml
• JavaScript: modestbench.config.js, modestbench.config.mjs, .modestbenchrc.js, .modestbenchrc.mjs
• TypeScript: modestbench.config.ts
• package.json: Use a "modestbench" field

Generate a configuration file using:

modestbench init --config-type json   # JSON format
modestbench init --config-type yaml   # YAML format
modestbench init --config-type js     # JavaScript format
modestbench init --config-type ts     # TypeScript format

Configuration Discovery: modestbench automatically searches for configuration files in the current directory and parent directories, following standard conventions.

Output Formats

Human-Readable (Default)

Real-time progress bars with color-coded results and performance summaries.

JSON Output

Structured data perfect for programmatic analysis and integration:

{
  "results": [
    {
      "file": "example.bench.js",
      "suite": "Array Operations",
      "task": "Array.push()",
      "hz": 1234567.89,
      "stats": {
        "mean": 0.00081,
        "stdDev": 0.00002,
        "marginOfError": 2.45
      }
    }
  ],
  "run": {
    "id": "run-2025-10-07-001",
    "timestamp": "2025-10-07T10:30:00.000Z",
    "duration": 15420,
    "status": "completed"
  }
}

CSV Export

Tabular data for spreadsheet analysis and historical tracking.

Advanced Features

Multiple Suites

const state = {
  data: [],
  sortedData: [],
};

export default {
  suites: {
    Sorting: {
      setup() {
        state.data = generateTestData(1000);
      },
      benchmarks: {
        // Shorthand syntax for simple benchmarks
        'Quick Sort': () => quickSort(state.data),
        'Merge Sort': () => mergeSort(state.data),
      },
    },

    Searching: {
      setup() {
        state.sortedData = generateSortedData(10000);
      },
      benchmarks: {
        'Binary Search': () => binarySearch(state.sortedData, 5000),
        'Linear Search': () => linearSearch(state.sortedData, 5000),
      },
    },
  },
};

Async Operations

export default {
  suites: {
    'Async Performance': {
      benchmarks: {
        // Shorthand syntax works with async functions too
        'Promise.resolve()': async () => {
          return await Promise.resolve('test');
        },

        // Full syntax when you need config, tags, or metadata
        'Fetch Simulation': {
          async fn() {
            const response = await simulateApiCall();
            return response.json();
          },
          config: {
            iterations: 100, // Fewer iterations for slow operations
          },
        },
      },
    },
  },
};

Tagging and Filtering

modestbench supports a powerful tagging system that lets you organize and selectively run benchmarks. Tags can be applied at three levels: file, suite, and task. Tags automatically cascade from parent to child, so tasks inherit tags from their suite and file.

Adding Tags

Tags can be added at any level:

export default {
  // File-level tags (inherited by all suites and tasks)
  tags: ['performance', 'core'],

  suites: {
    'String Operations': {
      // Suite-level tags (inherited by all tasks in this suite)
      tags: ['string', 'fast'],

      benchmarks: {
        // Task inherits: ['performance', 'core', 'string', 'fast', 'regex']
        'RegExp Test': {
          fn: () => /pattern/.test(str),
          tags: ['regex'], // Task-specific tags
        },

        // Task inherits: ['performance', 'core', 'string', 'fast']
        'String Includes': () => str.includes('pattern'),
      },
    },
  },
};

Filtering Benchmarks

Use --tags to include only benchmarks with specific tags (OR logic - matches ANY tag):

# Run only fast algorithms
modestbench run --tags fast

# Run benchmarks tagged with 'string' OR 'array'
modestbench run --tags string,array

# Multiple tags can be space-separated too
modestbench run --tags fast optimized

Use --exclude-tags to skip benchmarks with specific tags:

# Exclude slow benchmarks
modestbench run --exclude-tags slow

# Exclude experimental and unstable benchmarks
modestbench run --exclude-tags experimental,unstable

Combine both to fine-tune your benchmark runs (exclusion takes precedence):

# Run fast benchmarks, but exclude experimental ones
modestbench run --tags fast --exclude-tags experimental

# Run algorithm benchmarks but skip slow reference implementations
modestbench run --tags algorithm --exclude-tags slow,reference

Tag Cascading Example

export default {
  tags: ['file-level'], // All tasks get this tag

  suites: {
    'Fast Suite': {
      tags: ['fast'], // Tasks get: ['file-level', 'fast']
      benchmarks: {
        'Task A': {
          fn: () => {},
          tags: ['math'], // This task has: ['file-level', 'fast', 'math']
        },
        'Task B': () => {}, // This task has: ['file-level', 'fast']
      },
    },
  },
};

Filtering Behavior:

• --tags math → Runs only Task A (matches 'math')
• --tags fast → Runs both Task A and Task B (both have 'fast')
• --tags file-level → Runs both tasks (both inherit 'file-level')
• --exclude-tags math → Runs only Task B (Task A excluded)

Suite Lifecycle with Filtering

Suite setup() and teardown() only run if at least one task in the suite matches the filter. This prevents unnecessary setup work for filtered-out suites.

Integration Examples

GitHub Actions

name: Performance Tests
on: [push, pull_request]

jobs:
  benchmark:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v3
      - uses: actions/setup-node@v3
        with:
          node-version: 18

      - run: npm ci
      - run: npm run build

      - name: Run Benchmarks
        run: |
          modestbench \
            --reporters json,csv \
            --output ./results

      - name: Upload Results
        uses: actions/upload-artifact@v3
        with:
          name: benchmark-results
          path: ./results/

Performance Regression Detection

// scripts/check-regression.js
import { execSync } from 'child_process';
import { readFileSync } from 'fs';

// Run current benchmarks
execSync('modestbench --reporters json --output ./current');
const current = JSON.parse(readFileSync('./current/results.json'));

// Load baseline results
const baseline = JSON.parse(readFileSync('./baseline/results.json'));

// Check for significant regressions
for (const result of current.results) {
  const baselineResult = baseline.results.find(
    (r) => r.file === result.file && r.task === result.task,
  );

  if (baselineResult) {
    const regression = (baselineResult.hz - result.hz) / baselineResult.hz;
    if (regression > 0.1) {
      // 10% regression threshold
      console.error(
        `Performance regression detected in ${result.task}: ${(regression * 100).toFixed(1)}%`,
      );
      process.exit(1);
    }
  }
}

console.log('No performance regressions detected ✅');

Programmatic API

import { modestbench, HumanReporter } from 'modestbench';

// initialize the engine
const engine = modestbench();

engine.registerReporter('human', new HumanReporter());

// Execute benchmarks
const result = await engine.execute({
  pattern: '**/*.bench.js',
  iterations: 1000,
});

Contributing

We welcome contributions! Please see our Contributing Guide for details.

Development Setup

# Clone the repository
git clone https://github.com/boneskull/modestbench.git
cd modestbench

# Install dependencies
npm install

# Run tests
npm test

# Build the project
npm run build

# Run examples
npm run examples

Acknowledgments

• Built on top of the small-but-mighty benchmarking library, tinybench
• Interface inspired by good ol' Benchmark.js
• Built with zshy for dual ESM/CJS modules
• AccurateEngine statistical analysis inspired by the excellent work of bench-node

Resources

• Issue Tracker
• Discussions

License

Copyright © 2025 Christopher Hiller. Licensed under the Blue Oak Model License 1.0.0.