Jimmy/behavior-research-app - #30
Merged
Merged
Conversation
…airs (verified against DataExporter.swift's second-precision timestamps)
…so existed in Task 6's fusion CLI code
…full known data _run_pooled_and_louo was evaluating pooled models on each participant's FULL known-label set, which structurally includes the frames those same participants' 80% train split was trained on. Pooled eval now uses each participant's local 20% held-out split (previously computed and discarded) instead. LOUO's "100% unseen" semantics are unchanged. Adds an inline assertion guarding against this exact regression.
…pens, not just mtime non-decrease The old assertion (mtime_after >= mtime_before) would pass even if refresh=True silently no-op'd, since an unchanged mtime still satisfies >=. Now asserts extract_features() call counts across a miss, a hit, and a refresh, so a regression that ignores refresh=True is caught.
…ocess log Review flagged that 'Key facts for future agents' cited a windowed-acc number (1.000/1.000) without stating it came from fusion_pooled_train.py's separately-trained LOUO IMU-only comparison model, distinct from train_hand_classifier.py's own LOUO model reported earlier in the same document (1.000/0.999). Both numbers are real, independently-trained models (stochastic training, no fixed seed) -- not a contradiction, just missing attribution that could confuse a future reader skimming the summary section.
Both were stale test assumptions, unrelated to the pooled/LOUO/fusion work (flagged as a known follow-up by that work's final review): - hand_manifest_combined.csv now legitimately has an imu_relative_path column (from the earlier per-condition IMU-capture work on this branch) -- the test asserted it must NOT have one. Manifests are now told whether they're expected to have the column via a new expects_imu_column parameter, so old-format exports (Tran, Jimmy_Chen) still get their original stale-format assertions. - hand_manifest_Jimmy_Chen.csv references 549 images that aren't present in this checkout's gitignored Model-Training-Test/hand_images/, so every row gets skipped at load and 0 records come back -- a local data-completeness gap, not a parsing regression. Now skipped (like the existing "manifest file itself missing" case) instead of failed when 0 records load. tests/test_hand_pipeline.py: 76 passed/2 failed/1 skipped -> 77 passed/0 failed/2 skipped.
The image projection (Dense(128) off a 25088-d frozen VGG16 input, ~3.2M params) and the post-concatenation fusion head had no regularization, unlike the IMU branch (Dropout(0.5)) and unlike train_hand_classifier._train_handynet's head (same rate) -- flagged in code review as the likely cause of fusion's poor cross-user (LOUO) generalization in the 2026-07-20 real-data run. Testing that hypothesis directly: added Dropout(0.5) in both spots, matching the rate already used everywhere else in this codebase. Real-data result (see .claude/process/2026-07-21-fusion-dropout- experiment.md): LOUO windowed-accuracy gap vs IMU-only closed from 0.267/0.065 down to 0.045/0.000 -- confirms the overfitting hypothesis.
Two real bugs hit merging a 16,947-frame export (Jimmy re-captured with the new 30fps app build), both verified fixed against an isolated sandbox before landing (never touching real data during testing): - `cp SRC/* DEST/` shell-globs every filename onto one command line, which exceeds ARG_MAX well before 30fps-scale exports (broke at ~16,900 files; a 2fps-era export like the original ~1,000-3,000 frame ones never hit this). Switched to `cp -R SRC/. DEST/`, which copies directory contents as a single argument. - The "already in exports/" resume path did a literal string match against the absolute $EXPORTS path, so passing a RELATIVE path to an existing exports/<...> folder fell through to the "new export" branch and made a redundant full copy instead of resuming. SRC is now normalized to an absolute path before that comparison. Verified: an isolated fake repo + 2500-file synthetic export exercises both fixes (no ARG_MAX failure; a relative-path repeat run correctly hits the existing-merge guard instead of duplicating).
…n — measured slower on CPU) Added train_hand_classifier.segment_batch()/extract_features_batch() (additive, existing single-image functions untouched) and fusion_pooled_train.cache_images_batch(), aimed at speeding up the fusion feature cache's cold-start cost (~61min projected for the 20,236-frame dataset after the Jimmy re-capture merge). A fair warm-process A/B on real images found batching ~2x SLOWER on this CPU-only dev hardware (233 img/min single-image vs 115 img/min batched, batch_size=32) -- no GPU to keep fed, and the larger per-batch working set likely blows past cache into memory-bandwidth-bound territory. eligible_records() is therefore reverted to its original one-image-at-a-time loop (verified identical to pre-batching behavior, all tests unchanged). The batched functions are kept as tested, correct, documented-but-unused utilities for if this pipeline ever runs on GPU-accelerated hardware. See .claude/process/2026-07-21-batched-caching-negative-result.md.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Improved model features + added app for user behavior research