Skip to content

Log per-check grading outcomes as structured research data #128

Description

@dbosk

The grading scripts (modules/*/grading/grade.sh.nw) run per-repo/per-student checks but flatten them to P/F plus feedback text. For the vt-terminal/vt-git/vt-latex studies, each check failure is an observation of an undiscerned critical aspect — the grading rubric doubles as the coding scheme (full design: dbosk/vt-git#2).

To do:

  • Give every check a stable check-id, and map check-ids to the papers' critical aspects (the mapping lives in the papers; the ids live here).
  • After each grading run, append per-check outcomes to a structured log (CSV/JSONL): run timestamp, course, module, pseudonymous student id, check-id, outcome (pass/fail), attempt number.
  • Keep the log out of Canvas and out of git; store where the grading runs (server), pseudonymised.
  • Extraction tooling filters by the research-consent list at extraction time (grading itself runs for everyone); consent is collected by the papers' pre-quizzes ("my submissions in the course").
  • The existing results CSVs (e.g. modules/latex/grading/results-datintro20-LAB1.csv) already give attempt trajectories per student — keep them, they are the longitudinal backbone.

Fits naturally into the ongoing review-all-modules grading-script work. Related: #127, #50; dbosk/vt-terminal#3, dbosk/vt-git#2, dbosk/vt-latex#2.

Metadata

Metadata

Assignees

No one assigned

    Labels

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions