[P2][Phase-3][experimental] Porto Dataset Beam Search Ablation Study

[matercomus]

Metadata

• Priority: P2-Moderate
• Phase: Phase 3 (Future Work)
• Feasibility: experimental
• Category: validation
• Effort: 10-40 hours (depending on scope)
• Dependencies: Issue [P0][Phase-2][document] Issue 1.8: Beam Search Evaluation Dependence #8 (Beijing ablation complete)

Problem Statement

Issue #8 demonstrated that distillation benefits are search-method dependent on Beijing dataset. However, Porto dataset shows fundamentally different baseline characteristics (vanilla succeeds with 88-89% OD match vs Beijing's 12-18% failure), raising questions about cross-dataset generalizability of the search-method interaction.

Evidence

Beijing Results (Issue #8):

• A* Search: Distilled 6.0x faster than vanilla (0.30 vs 0.05 traj/s)
• Beam Search: Vanilla 1.4x faster than distilled (2.46 vs 1.79 traj/s)
• Conclusion: Search method determines whether distillation helps or hurts speed

Porto Characteristics (Different from Beijing):

| Characteristic          | Beijing      | Porto        | Impact                    |
|------------------------|--------------|--------------|---------------------------|
| Vanilla OD Match (Beam)| 19%          | 88-89%       | Porto vanilla succeeds!   |
| Average trip distance  | 5.16 km      | 3.66 km      | 29% shorter trips         |
| Distillation benefit   | +486% OD     | -2% OD       | Minimal advantage         |
| Road network           | Grid         | Complex      | Different topology        |
| Models available       | 6 (2×3)      | 9 (3×3)      | More comprehensive        |

Key Question: Does the search-method interaction:

• Only manifest when vanilla baseline fails (Beijing-specific)?
• Show different patterns with successful vanilla baseline (Porto)?
• Depend on dataset characteristics (trip length, network topology)?

Proposed Experiment

Option A: Selective Ablation (Recommended - 10 hours)

Models to test (2 models only):

• vanilla_25epoch_seed42.pth (best Porto baseline: 91.7% train, 89.6% test)
• distill_phase2_seed42.pth (optimized hyperparameters: λ=0.00598, τ=2.515)

Configuration:

• Search methods: A* + Beam (width=4)
• OD sources: train + test
• Trajectories: 1,000 per run
• Total runs: 2 models × 2 OD × 2 search = 8 runs
• Estimated time: ~10 hours

Rationale:

• Tests key hypotheses with 75% less computation than full study
• Phase 2 hyperparameters optimized for Porto (unlike Phase 1 used in existing eval)
• Best vanilla baseline for fair comparison
• Sufficient for cross-dataset validation

Option B: Full Ablation (Thorough - 40 hours)

All models (9 total):

• 3× vanilla (seeds 42, 43, 44)
• 3× distill_phase1 (seeds 42, 43, 44)
• 3× distill_phase2 (seeds 42, 43, 44)

Configuration:

• Search methods: A* + Beam (width=4)
• OD sources: train + test
• Trajectories: 1,000 per run
• Total runs: 9 models × 2 OD × 2 search = 36 runs
• Estimated time: ~40 hours

Benefits:

• Comprehensive cross-dataset validation
• Compares Phase 1 vs Phase 2 hyperparameter impact
• Tests seed robustness (3 seeds per model type)
• Publication-quality rigor

Expected Outcomes

Hypothesis A: No Interaction (Most Likely)

Since Porto vanilla already succeeds:

• Both A* and Beam show similar performance
• Distillation benefit remains minimal (~-2% to +1%)
• Implication: Search-method interaction is Beijing-specific (occurs when vanilla fails)

Hypothesis B: Reversed Interaction

Porto's shorter trips might show opposite pattern:

• Vanilla might excel with A* (simpler navigation)
• Distilled might benefit more from beam (exploration helps in complex topology)
• Implication: Interaction pattern depends on task complexity

Hypothesis C: Phase 2 Reveals Benefits

Optimized hyperparameters might show:

• Phase 2 distilled performs better than Phase 1
• Search-method interaction appears with proper hyperparameters
• Implication: Hyperparameter tuning quality affects interaction

Validation Steps

• Experiment designed (Option A or B selected)
• Script adapted for Porto dataset
• A* generation complete (if not already existing)
• Beam generation complete
• A* evaluation with normalized metrics
• Beam evaluation with normalized metrics
• Results analyzed and compared to Beijing
• Cross-dataset conclusions documented

Scientific Value

Publication Impact

Strengthens Paper:

• ✅ Cross-dataset validation (2 cities vs 1)
• ✅ Tests generalizability of search-method interaction
• ✅ Shows dataset-dependent vs universal effects
• ✅ Demonstrates when distillation benefits apply

Negative Results Also Valuable:

• "No interaction when vanilla succeeds" is a finding
• Shows interaction is context-dependent, not universal
• Provides deployment guidance based on baseline performance

Research Questions Answered

1. Is search-method interaction universal?

   • Beijing: Yes, dramatic interaction
   • Porto: To be determined

2. Does interaction depend on baseline performance?

   • When vanilla fails (Beijing): Strong interaction
   • When vanilla succeeds (Porto): ?

3. Do optimized hyperparameters change interaction?

   • Phase 1 (suboptimal): Minimal benefit on Porto
   • Phase 2 (optimized): To be tested

Files to Generate

Trajectories (if not already exist):

• hoser-distill-optuna-porto-eval-*/gene/Porto/seed42/*_astar_*.csv (A* search)
• Beam search trajectories already exist from previous eval

Evaluations (with normalized metrics):

• eval/2025-11-*/results.json (with Hausdorff_norm, DTW_norm)
• Performance metrics: *_perf.json

Documentation:

• Comparison report: Porto vs Beijing search-method interaction
• Update to docs/EVALUATION_COMPARISON.md

Coordination with Other Issues

Dependencies:

• ✅ Issue [P0][Phase-2][document] Issue 1.8: Beam Search Evaluation Dependence #8 (Beijing ablation) - Complete
• ✅ Issue [P1][Phase-2][fix] Issue 2.6: Local Metrics Not Normalized #14 (Normalized metrics) - Merged
• ✅ Issue [P2][Phase-3][document] Issue 3.4: Coefficient of Variation Misuse #22 (CV handling) - Merged

Blocks:

• None (independent validation study)

Relates to:

• Issue [P3][Phase-3][document] Issue 5.3: Cross-Model Comparisons Limited #34: Cross-Model Comparisons Limited
• Issue [P3][Phase-3][document] Issue 4.4: Porto Dataset Discussion Weak #30: Porto Dataset Discussion Weak

Execution Plan

Immediate (for Issue #8 closure):

• Document as future work in Issue [P0][Phase-2][document] Issue 1.8: Beam Search Evaluation Dependence #8
• Create this issue for tracking
• Close Issue [P0][Phase-2][document] Issue 1.8: Beam Search Evaluation Dependence #8 with Beijing results

Short-term (if pursuing Option A):

1. Verify A* trajectories exist for vanilla + distill_phase2 seed42
2. Generate missing A* trajectories if needed (~5 hours)
3. Re-evaluate with normalized metrics (~1 hour)
4. Generate beam trajectories (~4 hours)
5. Evaluate beam with normalized metrics (~1 hour)
6. Analyze and document results (~1 hour)

Long-term (if pursuing Option B):

1. Full A* generation for all 9 models (~20 hours)
2. Full beam generation for all 9 models (~20 hours)
3. Comprehensive analysis (~2 hours)

Success Criteria

Minimum (Option A):

• 8 evaluation runs completed with normalized metrics
• Clear comparison: Porto vs Beijing interaction patterns
• Documented: When distillation benefits depend on search method

Comprehensive (Option B):

• 36 evaluation runs completed across all models
• Phase 1 vs Phase 2 impact on search-method interaction
• Seed robustness analysis
• Cross-dataset comparison table

Notes

From Beijing Ablation (Issue #8):

• Normalized metrics (Hausdorff_norm, DTW_norm) essential for fair comparison
• Speed-quality tradeoffs must be considered
• OD match rate is critical differentiator

Porto-Specific Considerations:

• Shorter trips (3.66 km) may reduce search depth impact
• Higher road network connectivity might minimize A* vs Beam differences
• Vanilla baseline strength reduces distillation's relative advantage

---

Reference: Issue #8 (Beijing Beam Search Ablation)
Related: hoser-distill-optuna-porto-eval-eb0e88ab-20251026_152732/EVALUATION_ANALYSIS_PHASE1.md
Status: Future work - to be scheduled after P1-Major issues complete
Recommendation: Start with Option A (selective ablation) for efficient cross-validation

[matercomus]

Sample Size Update from Beijing Paired Tests (Related to Issue #19)

Empirical Evidence: 1000 vs 5000 Trajectories

We just ran paired statistical tests on Beijing with 1000 trajectory samples. The results provide concrete evidence for sample size decisions in this Porto ablation study.

Beijing Results (1000 Trajectories)

Model Performance:

• Distilled: 85.6% OD pair match rate (856/1000)
• Vanilla: 19.4% OD pair match rate (187/1000)

Paired Analysis:

• Matched pairs for comparison: 61-776 (depending on model pair)
• Distilled vs Vanilla: 61 matched pairs
• Statistical power achieved: 80-95% for medium-large effects (d≥0.5)
• All key metrics showed significant differences (p<0.01)

Implications for Porto Ablation (Issue #44)

Option A Proposal (1000 trajectories):

• ✅ Sufficient for Porto if vanilla succeeds (88-89% OD match)
• Expected matched pairs: ~880-890 pairs (vs Beijing's 61-187)
• Power: >95% for all effect sizes
• Verdict: 1000 is adequate for Option A

Why Porto is Better Positioned:

• Porto vanilla: 88-89% OD match (vs Beijing's 19%)
• More matched pairs available for comparison
• Higher statistical power per trajectory generated
• Less affected by OD matching bottleneck

Scaling Expectations

If using 5000 trajectories (tonight's A* runs):

• Beijing: 61 → ~305 matched pairs (5× improvement)
• Porto: ~880 → ~4,400 matched pairs (5× improvement)

Recommendation for Issue #44:

• Option A (1000 traj): Statistically adequate, especially for Porto ✅
• Option B (5000 traj): Provides robust estimates but 40+ hour cost
• Pragmatic approach: Use tonight's A* runs (5000) for comprehensive analysis

Tonight's A* Runs Address Both Issues

The overnight evaluations will:

1. ✅ Generate 5000 A* trajectories for both Beijing and Porto
2. ✅ Provide optimal sample sizes for Issue [P2][Phase-3][document] Issue 3.1: Sample Size Not Justified #19 justification
3. ✅ Enable Porto ablation study (Issue [P2][Phase-3][experimental] Porto Dataset Beam Search Ablation Study #44) with robust statistics
4. ✅ Allow cross-dataset comparison with matched sample sizes

Updated Recommendation

For Issue #44 Porto Ablation:

• Wait for tonight's A* runs to complete
• Use 5000 trajectory results for both A* and Beam
• Compare with existing Porto beam results (already have 5000)
• This gives you publication-quality sample sizes at no extra cost

Cross-reference: See Issue #19 for detailed sample size analysis and power calculations.

---

Status: Overnight A* runs will provide optimal data for both Porto ablation (Issue #44) and sample size justification (Issue #19).

[matercomus]

Overnight A* Evaluation - Porto Ablation Data Collection

Decision: Pursuing Full Evaluation (Option B+)

Based on sample size analysis from Issue #19, proceeding with full A evaluation across all Porto models* using existing infrastructure. This provides comprehensive ablation data at no extra development cost.

Timeline

Tonight (Nov 6-7, 2025): Full A* evaluation run for Porto dataset

Command Executed

cd /home/matt/Dev/HOSER && ./scripts/run_astar_evaluation.sh hoser-distill-optuna-porto-eval-eb0e88ab-20251026_152732

Models Included

All 9 Porto models will be evaluated with A* search:

• 3× vanilla (seeds 42, 43, 44)
• 3× distill_phase1 (seeds 42, 43, 44)
• 3× distill_phase2 (seeds 42, 43, 44)

Configuration:

• OD sources: train + test
• Trajectories: 5000 per run (optimal sample size from Issue [P2][Phase-3][document] Issue 3.1: Sample Size Not Justified #19)
• Search method: A* (new data to compare with existing Beam results)
• Metrics: All normalized metrics (Hausdorff_norm, DTW_norm) included

Data to be Collected

Quality Metrics:

• Normalized trajectory similarity (Hausdorff_norm, DTW_norm, EDR)
• OD match rates (origin, destination, pair, both correct)
• Trajectory-level metrics for paired statistical tests
• Distribution metrics (Distance JSD, Duration JSD, Radius JSD)

Performance Metrics:

• Generation speed (trajectories/second)
• Total generation time
• Memory usage
• Comparison: vanilla vs distilled across phases

Statistical Analysis:

• Paired comparisons between all model pairs
• Effect sizes (Cohen's d) for each metric
• Significance tests with FDR correction
• Cross-seed robustness analysis

Porto vs Beijing Comparison

Porto Advantages for Ablation:

• High vanilla baseline: 88-89% OD match (vs Beijing 19%)
• More matched pairs: ~4,400 expected (vs Beijing ~305)
• Higher statistical power: >95% for all effect sizes
• Better model coverage: 9 models vs Beijing's 6

Research Questions to Answer:

1. Does search-method interaction exist when vanilla succeeds?
2. Do Phase 2 hyperparameters improve distillation benefits?
3. Is the interaction pattern dataset-dependent or universal?
4. What role does trip length play (3.66 km Porto vs 5.16 km Beijing)?

Expected Outcomes

Hypothesis Testing:

• H1: No search-method interaction (vanilla succeeds with both A* and Beam)
• H2: Phase 2 hyperparameters reveal distillation benefits not seen in Phase 1
• H3: Seed robustness confirmed across 3 random seeds per model type

Files Generated:

• 18 evaluation runs: 9 models × 2 OD sources
• trajectory_metrics.json for each run (~5000 trajectories each)
• Paired analysis results for all 36 model pair combinations
• Performance benchmarks for computational cost analysis

Cross-References

• Issue [P2][Phase-3][document] Issue 3.1: Sample Size Not Justified #19: Sample size justification (5000 trajectories validated)
• Issue [P0][Phase-2][document] Issue 1.8: Beam Search Evaluation Dependence #8: Beijing beam ablation (comparison baseline)
• Issue Integrate paired statistical tests into evaluation pipeline #51: Paired statistical tests (methodology)
• Issue [P3][Phase-3][document] Issue 5.1: Computational Cost Not Reported #32: Computational cost reporting (perf data collected)

Next Steps After Completion

1. Verify data collection (~6-8 hours completion time)

   • Check all 18 trajectory_metrics.json files
   • Verify paired analysis completed successfully
   • Validate performance metrics (*_perf.json)

2. Compare A vs Beam* (existing Beam results already available)

   • Speed comparison: vanilla A* vs Beam
   • Speed comparison: distilled A* vs Beam
   • Quality comparison: normalized metrics across search methods

3. Cross-dataset analysis (Porto vs Beijing)

   • Search-method interaction patterns
   • Dataset characteristic effects (trip length, network topology)
   • Generalizability conclusions

4. Documentation updates

   • Update docs/EVALUATION_COMPARISON.md with Porto ablation
   • Create Porto-specific ablation summary
   • Add to docs/SEARCH_METHOD_GUIDANCE.md

5. Close Issue [P2][Phase-3][experimental] Porto Dataset Beam Search Ablation Study #44 with comprehensive findings

Status: Data collection in progress (overnight run)
Related: Issue #19 (sample size), Issue #8 (Beijing baseline)

[matercomus]

Porto A* Evaluation Started - Nov 6, 2025

Status: IN PROGRESS (8% Complete)

Current execution:

• Model: distilled_25epoch_seed42
• Progress: 422/5000 trajectories (8%)
• Speed: 5.45 seconds/trajectory
• Estimated remaining: ~6.5 hours for this model

Total Porto evaluation:

• 9 models × 2 OD sources × 5000 trajectories
• Estimated total time: ~14 hours
• Expected completion: Nov 7, ~10:30

Parallel Operations

Beijing training (running in parallel):

• Training new baseline models (Issue [P1][Phase-3][experimental] Beijing Baseline Validation & 3×3 Model Grid Completion #55)
• No interference with Porto evaluation
• Independent GPU usage

Data Collection in Progress

Quality metrics being generated:

• Normalized trajectory similarity (Hausdorff_norm, DTW_norm, EDR)
• OD match rates (origin, destination, pair, both correct)
• Trajectory-level metrics for paired tests
• Distribution metrics (Distance JSD, Duration JSD, Radius JSD)

Performance metrics:

• Generation speed (trajectories/second)
• Total generation time per model
• Memory usage

Statistical analysis (automatic after generation):

• Paired comparisons between all model pairs
• Effect sizes (Cohen's d) for each metric
• Significance tests with FDR correction
• Cross-seed robustness analysis

---

Next update: When Porto evaluation completes (~14 hours from start)

[matercomus]

Update: Fixed File Detection & Added Resume Support

Fixed two issues that were blocking Porto A* evaluation runs:

1. Fixed File Detection Bug

The _check_existing_results() method in python_pipeline.py was using an incorrect glob pattern (*{model_type}*{od_source}od*.csv) that didn't match actual file names. Files are named like {timestamp}_{model_type}_{od_source}.csv (e.g., 2025-11-07_00-13-07_distill_phase1_train.csv).

Fix: Updated to try multiple patterns supporting both Porto and Beijing naming conventions:

• *{model_type}*{od_source}.csv (primary pattern)
• *{model_type}*{od_source}od*.csv (backward compatibility)
• {model_type}*{od_source}*.csv (Beijing: model_type first)

The pipeline now correctly detects and automatically skips already-generated trajectories, preventing unnecessary regeneration.

2. Added --skip-backup Flag

Added --skip-backup flag to scripts/run_astar_evaluation.sh to allow resuming interrupted runs without backing up existing eval directories.

Usage:

./scripts/run_astar_evaluation.sh porto --skip-backup

This is especially useful when:

• Resuming interrupted A* generation runs
• Only 2 of 9 models completed (e.g., distill_phase1, distill_phase1_seed43)
• Want to continue from where it left off without regenerating completed work

Impact on This Issue

These fixes directly support the execution plan:

• ✅ Step 1: "Verify A* trajectories exist" - Now works correctly
• ✅ Step 2: "Generate missing A* trajectories if needed" - Will automatically skip existing ones
• ✅ Can resume interrupted runs without losing progress

The pipeline will now:

1. Detect existing A* trajectories (e.g., distill_phase1, distill_phase1_seed43)
2. Skip generation for those models
3. Continue generating missing trajectories (distill_phase1_seed44 and remaining models)
4. Proceed with evaluation automatically

Files changed:

• python_pipeline.py - Fixed _check_existing_results() method
• scripts/run_astar_evaluation.sh - Added --skip-backup flag and conditional backup logic

[matercomus]

Status Update: Pipeline Running

The Porto A* evaluation pipeline is currently running successfully!

Started: 2025-11-07 14:33:57
Log file: /home/matt/Dev/HOSER/hoser-distill-optuna-porto-eval-eb0e88ab-20251026_152732/logs/pipeline_20251107_143258.log

Fixes Working Correctly ✅

The file detection fix is working as expected:

• ✅ Successfully detecting existing generated files for distill_phase1, distill_phase1_seed43, and distill_phase1_seed44
• ✅ Automatically skipping regeneration for models with existing trajectories
• ✅ Pipeline continuing with remaining models that need generation

Current status from log:

• Processing 9 models total (distill_phase1, distill_phase1_seed43, distill_phase1_seed44, distill_phase2, distill_phase2_seed43, distill_phase2_seed44, vanilla, vanilla_seed43, vanilla_seed44)
• Using A* search (beam_search: False)
• Currently in evaluation phase (calculating radii for trajectory metrics)

The pipeline will automatically:

1. Skip generation for models with existing trajectories
2. Generate missing trajectories for remaining models
3. Proceed with full evaluation, paired analysis, cross-dataset, abnormality detection, and scenario analysis

This validates that the fixes are working correctly and the pipeline can now resume interrupted runs efficiently.

[matercomus]

Critical Fix: A* Detection Now Working Correctly

Problem Identified

The previous evaluation runs (mentioned in earlier comments) were not actually using A search*. The pipeline was detecting and using beam search files instead of A* files, which means the A* ablation study data was not being collected correctly.

Root Cause: The _check_existing_results() method in python_pipeline.py was finding generated CSV files but not checking whether they were A or beam search*. It would pick the newest file regardless of search method, so beam search files (which were generated earlier) were being selected over A* files.

What We Fixed

1. A File Detection Logic*

• Modified _check_existing_results() to check *_perf.json files for beam_search_enabled: false
• Only returns files confirmed to be A* search (skips beam search files)
• If only beam search files exist, returns None to trigger A* generation

2. Evaluation Phase A Filtering*

• Updated run_base_eval() to filter for A* files only
• Prevents evaluation from using beam search trajectories when A* files are expected

3. Automatic Detection Per Model

• Pipeline now automatically detects which models have A* files
• Uses existing A* files where available (e.g., distill_phase1, distill_phase1_seed43)
• Generates missing A* trajectories for remaining models (e.g., distill_phase1_seed44, all distill_phase2, all vanilla)

Current Status

Running now with fixes applied (2025-11-07 15:54:59):

cd hoser-distill-optuna-porto-eval-eb0e88ab-20251026_152732
uv run python ../python_pipeline.py --use-astar

Pipeline behavior:

• ✅ Detected existing A* files: distill_phase1 (train/test), distill_phase1_seed43 (train/test)
• ✅ Skipping generation for models with A* files
• ✅ Generating A* trajectories for missing models: distill_phase1_seed44, all distill_phase2 (3 seeds), all vanilla (3 seeds)
• ✅ Using A* search (beam_search: False confirmed in logs)
• ✅ Will evaluate all 9 models with A* trajectories only

Expected outcome:

• 18 evaluation runs total (9 models × 2 OD sources)
• All using confirmed A search trajectories* (verified via *_perf.json)
• Clean comparison with existing beam search results
• Valid data for Porto ablation study

Impact on Issue #44

This fix ensures that:

1. ✅ A trajectories are actually being used* (not beam search)
2. ✅ Automatic detection prevents manual file management
3. ✅ Resume capability - can safely restart without regenerating existing A* files
4. ✅ Clean data for comparing A* vs Beam search methods

The previous runs that appeared to be "A* evaluation" were actually using beam search files. This run is the first true A evaluation* for the Porto ablation study.

Files changed:

• python_pipeline.py - _check_existing_results() and run_base_eval() methods
• Removed scripts/run_astar_evaluation.sh (redundant wrapper)

Verification: Check logs for "Found existing A* generated file" and "beam_search: False" in configuration to confirm A* search is being used.

[matercomus]

✅ Porto A* Analysis Completed Successfully

The Porto A* evaluation pipeline has finished running successfully!

Completion Summary

Status: All phases completed without errors

Results Generated:

• ✅ All 18 evaluations completed (9 models × 2 OD sources)
• ✅ All evaluations verified to use A* generated trajectories (confirmed via _perf.json files with beam_search_enabled: false)
• ✅ All phases completed: generation, base_eval, paired_analysis, cross_dataset, road_network_translate, abnormal, abnormal_od_extract, abnormal_od_generate, abnormal_od_evaluate, scenarios
• ✅ WandB sync: All 32 runs synced successfully

Models Evaluated (all with A* search):

• distill_phase1 (train/test)
• distill_phase1_seed43 (train/test)
• distill_phase1_seed44 (train/test)
• distill_phase2 (train/test)
• distill_phase2_seed43 (train/test)
• distill_phase2_seed44 (train/test)
• vanilla (train/test)
• vanilla_seed43 (train/test)
• vanilla_seed44 (train/test)

Data Available:

• 18 evaluation directories with results.json and trajectory_metrics.json
• 18 A* trajectory CSV files (5000 trajectories each)
• Paired analysis results
• Scenario analysis results
• Cross-model scenario aggregation

Next Steps

Now analyzing the results to:

1. Compare A* vs Beam search performance (Porto ablation study)
2. Evaluate search-method interaction patterns
3. Cross-dataset comparison with Beijing results
4. Document findings for Issue [P2][Phase-3][experimental] Porto Dataset Beam Search Ablation Study #44

Status: Analysis in progress - will update with findings when complete.