[P3][Phase-3][document] Issue 5.1: Computational Cost Not Reported

[matercomus]

Metadata

• Priority: P3-Minor
• Phase: Phase 3 (Week 3)
• Feasibility: document
• Category: documentation
• Effort: 1 hour
• Dependencies: None

Problem Statement

No computational cost analysis reported. Training time, inference time, memory usage not documented.

Evidence

Documentation: Performance metrics only, no computational cost
Missing: Time benchmarks, resource usage

Required Changes

1. Benchmark training time per epoch
2. Benchmark inference time per trajectory
3. Document memory usage
4. Compare student vs teacher computational costs
5. Document cost savings from distillation

Validation Steps

• Training time benchmarked
• Inference time benchmarked
• Memory usage documented
• Cost comparison added
• Efficiency gains quantified

Files to Modify

• Performance documentation
• Efficiency section (create if needed)
• README
• Results

---

Reference: Additional minor issue from peer review
Roadmap: Phase 3 - Week 3 (1 hour)

[matercomus]

Data Collection from Overnight A* Evaluation Runs

Data Already Available

Computational cost data has been collected throughout all evaluation runs and is stored in performance files:

Performance Metrics Files:

• gene/*/TIMESTAMP_*_perf.json - Generation performance data
• Contains:
  • Total generation time
  • Trajectories per second
  • Average time per trajectory
  • Model type and configuration

Overnight Runs Will Provide Additional Data

Tonight's A evaluations* (Issues #19, #44) will generate comprehensive performance data for:

Beijing (6 models × 2 OD sources = 12 runs):

• distilled, distilled_seed44, vanilla
• train + test OD sources
• A* search method (5000 trajectories each)

Porto (9 models × 2 OD sources = 18 runs):

• vanilla, distill_phase1, distill_phase2 (3 seeds each)
• train + test OD sources
• A* search method (5000 trajectories each)

Metrics to Extract and Document

1. Training Time (from training logs/WandB):

• Time per epoch
• Total training time
• Vanilla vs distilled comparison

2. Inference Time (from *_perf.json):

• Generation speed (trajectories/second)
• Time per trajectory
• A* vs Beam search comparison
• Vanilla vs distilled comparison

3. Memory Usage:

• Model size comparison (vanilla vs distilled)
• Peak memory during generation
• Memory efficiency gains

4. Computational Efficiency:

• Speedup from distillation (where applicable)
• Search method impact on speed
• Cross-dataset consistency

Documentation Plan

After overnight runs complete:

1. Extract performance data from all *_perf.json files

2. Create performance summary tables:

   • Beijing A* vs Beam (both search methods now available)
   • Porto A* vs Beam (both search methods now available)
   • Vanilla vs distilled across datasets
   • Phase 1 vs Phase 2 distillation (Porto)

3. Update documentation:

   • Add computational cost section to main README
   • Create docs/COMPUTATIONAL_EFFICIENCY.md
   • Add to docs/EVALUATION_COMPARISON.md

4. Report efficiency metrics:

   • Training cost savings (distilled model size)
   • Inference speed comparisons
   • Total computational budget per experiment

Cross-References

• Issue [P2][Phase-3][document] Issue 3.1: Sample Size Not Justified #19: Sample size justification (overnight runs)
• Issue [P2][Phase-3][experimental] Porto Dataset Beam Search Ablation Study #44: Porto ablation study (performance comparison)
• Issue [P0][Phase-2][document] Issue 1.8: Beam Search Evaluation Dependence #8: Beijing beam ablation (Beam search performance baseline)

Next Steps

1. Wait for overnight runs to complete (~6-8 hours)
2. Extract all performance metrics from JSON files
3. Create comprehensive computational cost documentation
4. Close this issue with findings

Status: Data collection in progress (overnight run)
Note: This is a documentation task using existing data - no new experiments needed