TypingResearch

TypingResearch has two parts:

• an iPhone app for running typing studies
• a scripts/ folder for cleaning exports, rendering visuals, and running post-study analyses

A companion iPhone app, FreeTypeRecorder, runs free-typing posture sessions with screen, camera-silhouette, IMU, and keystroke recording, and uploads them to Google Drive. See FreeTypeRecorder/README.md for setup and how to run a study.

iOS App

The app runs timed iPhone typing sessions and records:

• touch coordinates and key geometry
• timing, correctness, and correction behavior
• session and study metadata

Automatic trial and session outputs

The iOS app automatically turns collected trial data into these review outputs:

• session summaries with accuracy, WPM, and backspace behavior
• cleaned-data summaries with normalized tap positions and outlier flags
• tap-distribution keyboard views for raw and cleaned data
• per-session Gaussian boundary review pages showing how the boundary evolves over sessions
• a final Gaussian ground-truth boundary built from the full classic training data
• ground-truth loss charts showing how many trials are needed before the model stabilizes

In practice, this means the app gives you both the raw study data and the built-in review artifacts without needing to run Python scripts.

Keyboard modes:

• classic: normal fixed rectangular key regions
• gaussian: adaptive probabilistic key regions

Study designs:

• classic + adaptive: first half of sessions use the classic keyboard; second half use the Gaussian keyboard, using a model trained from the classic-session data
• classic only: every session uses the classic keyboard

Main in-app exports:

• raw keystroke CSV
• cleaned keystroke CSV
• tap-distribution PDF
• Gaussian boundary PDF
• ground-truth loss PDF
• holding-hand manifest CSV + captured images

Holding-hand classification (HandyTrak)

The app can collect holding-hand data for offline classification following the HandyTrak approach (Lim et al., UIST '21). After each typing session, a sheet captures a front-camera upper-body photo and a self-reported holding-hand label (Left / Right / Both / Unknown). The label defaults from the participant's stated dominant hand but is always editable. Photo capture is optional — label-only records are supported. Captured images are stored under Documents/hand_images/ and exported via the "Hand data" button in the summary screen alongside a manifest CSV. See scripts/README_hand.md for the offline training pipeline.

ML model training environment

The classifier follows the paper-faithful path (FCN-ResNet101 segmentation + VGG16 features + a small HandyNet head), which needs TensorFlow/Keras and torch. These are kept in an isolated .venv-ml/ at the repo root so they don't disturb the analysis venv/ or the anaconda base.

One-time setup (creates .venv-ml/, installs TensorFlow, prefetches the ~700 MB FCN-ResNet101 + VGG16 weights):

bash scripts/setup_ml_env.sh --prefetch-weights

Train a model (run from the repo root, using .venv-ml):

.venv-ml/bin/python scripts/train_hand_classifier.py \
    Model-Training-Test/hand_manifest_Jimmy_Chen.csv \
    --images-root Model-Training-Test/ \
    --out Model-Training-Test/models/ \
    --mode both --epochs 2

The two arguments that matter are the manifest CSV path and --images-root, which must point at the folder that contains hand_images/. A [PAPER-FAITHFUL] banner at startup confirms torch + TensorFlow are both available; segmentation is the slow stage (~10–20 min on CPU). Outputs land in --out: <participant>/hand_model.keras, <participant>/labels.json, and summary.json with the accuracy numbers.

Quick checks (no real data or heavy deps needed):

.venv-ml/bin/python scripts/train_hand_classifier.py --demo   # end-to-end plumbing
python3 scripts/train_hand_classifier.py --mode centroid ...  # zero-training baseline

Useful knobs: --epochs N (train longer), --mode handynet (skip the centroid baseline), --mode centroid (baseline only, runs in seconds). See Model-Training-Test/README.md for the step-by-step training guide, Model-Training-Test/model.md for the results log, and scripts/README_hand.md for full pipeline and manifest-schema details.

Open the app with:

open TypingResearch.xcodeproj

Script Summary

• scripts/clean_keystrokes.py: adds normalized coordinates and outlier flags
• scripts/keystrokes_to_pdf.py: renders tap-distribution PDFs
• scripts/gaussian_keyboard_pdf.py: renders one full-dataset Gaussian boundary as PDF or SVG
• scripts/session_overlap_visualization.py: renders a baseline keyboard boundary plus cumulative per-session Gaussian boundaries and summary CSVs
• scripts/key_session_boundary_video.py: ranks per-key session-vs-ground-truth boundary overlap changes and exports one-key-at-a-time session frame sequences
• scripts/plot_cleansing_subset.py: renders a side-by-side raw-vs-cleaned keyboard view for a chosen session range
• scripts/ground_truth_trial_loss.py: compares trial prefixes against all-trial ground truth
• scripts/numpy_analysis_utils.py: shared histogram and CSV helpers for the analysis scripts
• scripts/hand_dataset.py: reads the holding-hand manifest CSV + images; returns (image_paths, labels) for the training pipeline
• scripts/train_hand_classifier.py: HandyTrak pipeline (preprocess → segment → classify) for holding-hand classification; heavy DL deps optional with lightweight fallbacks

Offline Workflow

Run all commands from the repository root.

1. Clean an exported CSV

python3 scripts/clean_keystrokes.py <raw_keystrokes.csv>
python3 scripts/clean_keystrokes.py <raw_keystrokes.csv> <cleaned_keystrokes.csv>

If no output path is given, the script writes <input_stem>_cleaned.csv.

2. Render tap distributions

python3 scripts/keystrokes_to_pdf.py <cleaned_keystrokes.csv>
python3 scripts/keystrokes_to_pdf.py <cleaned_keystrokes.csv> <tap_distribution.pdf>

3. Render one overall Gaussian keyboard

python3 scripts/gaussian_keyboard_pdf.py <keystrokes.csv>
python3 scripts/gaussian_keyboard_pdf.py <keystrokes.csv> <gaussian_boundary.pdf>
python3 scripts/gaussian_keyboard_pdf.py <keystrokes.csv> <gaussian_boundary.svg>

4. Render per-session Gaussian boundaries

Default behavior writes both SVG and PDF outputs.

python3 scripts/session_overlap_visualization.py <cleaned_keystrokes.csv> --output-dir <output_dir>

Example:

python3 scripts/session_overlap_visualization.py /Users/jimmy2/Downloads/keystrokes_cleaned_Tran_.csv --output-dir /Users/jimmy2/Downloads/session_boundary_Tran_review

Useful options:

python3 scripts/session_overlap_visualization.py <cleaned_keystrokes.csv> --output-dir <output_dir> --format svg
python3 scripts/session_overlap_visualization.py <cleaned_keystrokes.csv> --output-dir <output_dir> --format pdf
python3 scripts/session_overlap_visualization.py <cleaned_keystrokes.csv> --output-dir <output_dir> --raster-step 3
python3 scripts/session_overlap_visualization.py --demo --output-dir /tmp/session-boundary-demo

Primary outputs:

• session_gaussian_boundaries_00.svg
• session_gaussian_boundaries_00.pdf
• session_gaussian_boundaries_XX.svg
• session_gaussian_boundaries_XX.pdf
• session_gaussian_boundaries_all_sessions.pdf
• final_gaussian_ground_truth_boundary.svg
• final_gaussian_ground_truth_boundary.pdf
• session_gaussian_boundaries_summary.csv
• session_gaussian_boundaries_by_key.csv

5. Run ground-truth trial loss

python3 scripts/ground_truth_trial_loss.py <cleaned_keystrokes.csv>

6. Export per-key boundary video frames

This is useful when you want to animate one key at a time over sessions, using the Gaussian boundary panel UI while also showing that key's Gaussian fade outside its winner boundary. Each exported frame now includes a Session N Trials X/Y label, and the script also writes a combined whole-keyboard overview sequence plus an MP4 when PNG frames are enabled and ffmpeg is installed.

python3 scripts/key_session_boundary_video.py <cleaned_keystrokes.csv> --output-dir <output_dir>

Useful options:

python3 scripts/key_session_boundary_video.py <cleaned_keystrokes.csv> --output-dir <output_dir> --keys q space delete
python3 scripts/key_session_boundary_video.py <cleaned_keystrokes.csv> --output-dir <output_dir> --format both
python3 scripts/key_session_boundary_video.py <cleaned_keystrokes.csv> --output-dir <output_dir> --fps 3
python3 scripts/key_session_boundary_video.py <cleaned_keystrokes.csv> --output-dir <output_dir> --skip-letter-overview
python3 scripts/key_session_boundary_video.py --demo --output-dir /tmp/key-session-boundary-demo

Primary outputs:

• key_boundary_overlap_summary.csv
• key_boundary_overlap_by_session.csv
• key_boundary_frame_manifest.csv
• largest_difference_key.txt
• frames/<key>/frame_XX_*.png
• frames/whole_keyboard/frame_XX_*.png
• videos/whole_keyboard_boundary.mp4

7. Generate a side-by-side cleansing check

python3 scripts/plot_cleansing_subset.py <raw_keystrokes.csv> <cleaned_keystrokes.csv> <output.pdf> --session-start 1 --session-end 5
python3 scripts/plot_cleansing_subset.py <raw_keystrokes.csv> <cleaned_keystrokes.csv> <output.pdf> --sessions 1 2 3 4 5