Skip to content

A staged-row source warning prints two times per run #1700

Closed

Description

@Muizzkolapo

Problem

Each staged-row warning about a source field prints two times in one run.

Reproduction (main at e8a1303, sample project copy)

Workflow ql_guard_probe (tools only): tag reads the staged rows, and check (below it) has guard: { condition: "source.scroe IS NULL", on_false: "filter" }. No staged row holds scroe.

agac run -a ql_guard_probe --fresh 2>&1 | grep -c "guards on source.scroe"
2

Both lines are the same:
Action 'check' guards on source.scroe, but no record staged from staging/rows.json holds 'scroe'. Records from that file will not have it.

Cause

validate_source_reads_below (input/preprocessing/staging/field_validation.py; renamed validate_staged_source_reads in #1686) runs in _check_and_prepare. The walk calls that function twice for each staged file: once from precheck_staged_file, before the first file is processed, and once from process_initial_stage, when the file is processed. Each call logs its warnings.

Proposed fix

Log each warning once per file per run. For example, the precheck logs the warnings and the processing pass does not, or the warnings are kept by file and logged once. A refusal still stops the run at the precheck.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't working

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions