MOSAIC Scalability Experiment

Measures MOSAIC middleware scalability by increasing robot count (N = 1, 2, 4, 6, 8, 10) and recording network latency, FPS, jitter, and RTT under two deployment conditions (co-located and distributed).

---

Repository Structure

mosaic-scalability-experiment/
├── terraform/          # AWS EC2 infrastructure provisioning
├── robot_configs/      # Per-robot MOSAIC config files (YAML, Grafana JSON)
├── configs/            # Docker Compose template
├── compose/            # Compose file generation script
├── scripts/            # Experiment start/stop scripts
└── analysis/           # Analysis pipeline: preprocess → merge → plot
    ├── raw-results/    # Raw measurement CSVs (collected after experiment)
    └── output/         # All generated files (gitignored)
        ├── results_preprocessed/   # Preprocessed CSVs
        ├── merged_tail/            # Pooled percentile aggregation
        ├── merged_worst/           # Per-robot worst-case aggregation
        └── figures_combined/       # Combined plots (pooled + worst-case overlaid)

---

1. Running the Experiment

Prerequisites

• AWS account and EC2 key pair configured
• Fill in terraform/terraform.tfvars (see terraform.tfvars.example)
• Place robot YAML configs in robot_configs/

Start

# Launch N robots on EC2 (provisions instances + starts Docker Compose)
./scripts/start_experiment.sh <num_robots> [key_pair_name]

# Or run locally
./scripts/start_local_experiment.sh <num_robots>

Stop and Collect Results

./scripts/stop_experiment.sh
# or locally
./scripts/stop_local_experiment.sh

Collected CSVs go into analysis/raw-results/ with the following layout:

analysis/raw-results/
└── {deployment}/           # co-located or distributed
    └── {N}/                # number of robots (1, 2, 4, 6, 8, 10)
        ├── husky-sim-0_delay_stats.csv
        ├── husky-sim-0_large_delay_stats.csv
        ├── husky-sim-0_stream_stats.csv
        └── ...

File	Content	Key columns
delay_stats.csv	Small message (30 B) round-trip delay	latency_ms
large_delay_stats.csv	Large message (250 KB) round-trip delay	latency_ms
stream_stats.csv	Stream quality	fps, jitter, rtt_ms

---

2. Analysis and Visualization

All commands are run from the repo root.

Pipeline

Pooled percentile (P50/P95) and per-robot worst-case (median/extreme) are overlaid in a single graph, with one file per deployment (co-located / distributed).

Output: analysis/output/figures_combined/

Quick Start (intermediate results already exist)

python analysis/run_scalability_pipeline.py --skip-preprocess --skip-merge-pooled --skip-merge-worst

Full Run from Scratch

python analysis/run_scalability_pipeline.py

Use --skip-merge-pooled --skip-merge-worst to skip re-aggregation when only the plots need to be regenerated.

Common Options

Option	Description	Default
--raw	Raw data directory	analysis/raw-results
--dpi	PNG resolution	300
--delay-mode	small / large / both	both
--log-latency	Log scale for latency axis	(linear)
--legend-in-each	Embed legend in each figure	(separate legend file)
--figsize W H	Figure size in inches	4.0 3.0
--fps-ylim YMIN YMAX	Fix FPS y-axis range	(auto)

Use --skip-preprocess, --skip-merge-pooled, --skip-merge-worst to skip completed stages.

Aggregation Methods

Two aggregations are computed and overlaid in each plot:

	Pooled (merge_stats_pooled)	Per-robot worst-case (merge_stats_worst)
Method	Concatenate all robot samples, then compute percentiles	Compute per-robot worst percentile, then aggregate across robots
High-is-bad (latency, jitter, rtt)	P50, P95	Per-robot P99 → median(P99_i), max(P99_i)
Low-is-bad (fps)	P05, P50	Per-robot P01 → median(P01_i), min(P01_i)
Characteristic	Individual robot extremes can be diluted	Directly exposes per-robot worst-case behavior

---

3. Analysis Code Structure

analysis/
├── preprocess.py                            # Preprocessing (120-row window, fps=0 filter)
├── merge_stats_pooled.py                    # Pooled percentile aggregation
├── merge_stats_worst.py                     # Per-robot worst-case aggregation
├── plot_results_combined_per_deployment.py  # Plot renderer
└── run_scalability_pipeline.py              # Pipeline runner