[Feature]: Resilient bulk prediction — per-row error isolation, input caps, and reload-consistent version stamping

[pavsoss]

Summary

/bulk-predict and /bulk-predict/export (backend/bulk_predict.py) run inference over an uploaded CSV/TXT file, but the pipeline is all-or-nothing and disconnected from the hot-reloadable serving state. A single un-transformable batch aborts the entire request, there is no cap on the number of rows (only a 2 MB file-size check), results are fully buffered in memory before responding, and the classifier objects are read directly off current_app rather than from the versioned serving snapshot — so a mid-request /reload-model can silently pair a new model with a stale vectorizer.

Problems

• No per-row isolation. One malformed/oversized row raises inside _batch_predict and 500s the whole file; the caller loses every valid result.
• Not reload-consistent. Inference uses current_app.model / .vectorizer / .label_encoder instead of a coherent serving snapshot, defeating the atomicity guarantees added for /reload-model.
• No row/complexity guardrails. File size is bounded but row count and per-row length are not, allowing large-but-under-2 MB files to exhaust memory or time.
• No provenance. Bulk results carry no model version, so exported CSVs can't be traced to the model that produced them.

Proposed work

• Route bulk inference through the versioned serving snapshot so an entire request is scored against one consistent set of objects, and stamp the response/export with that version.
• Isolate failures per row: invalid or over-length rows are skipped with a typed reason and surfaced in a skipped/errors section, while valid rows still return.
• Enforce configurable caps (max rows, max per-row length) with clear typed errors via the existing error envelope.
• Add an opt-in NDJSON streaming response so very large files return incrementally instead of buffering all results.

Suggested PR split

Each part stands alone and leaves /bulk-predict fully functional.

Part A — Robust, reload-consistent inference
Route bulk inference through the versioned serving snapshot (fixing the mid-request reload hazard) and stamp responses/exports with the serving model version. Isolate failures per row so invalid/over-length rows are skipped with a typed reason and returned in a structured skipped/errors section while valid rows still succeed, and enforce configurable row-count and per-row-length caps through the error envelope.

Part B — NDJSON streaming for large files
Add an opt-in streaming response so very large uploads return results incrementally instead of buffering the full result set in memory.

Acceptance criteria

• A file mixing valid and invalid rows returns all valid predictions plus a structured list of skipped rows with reasons.
• Bulk responses and exports include the serving model version.
• Row-count and per-row-length caps are enforced and configurable.
• Regression coverage for mixed-validity input, cap enforcement, and version stamping.

[github-actions]

👋 Welcome back! Thanks for your continued contribution. Your issue/PR is received and waiting for maintainer review.

[github-actions]

The issue outlines limitations in the current /bulk-predict and /bulk-predict/export pipeline, including lack of per-row error isolation (one error cancels the entire batch), inconsistency with reloaded models, absence of input size restrictions, and missing model version tracing in outputs. The proposal suggests routing predictions through versioned serving snapshots for consistency, isolating per-row errors to allow partial successes, enforcing row/length caps to prevent resource exhaustion, and optionally switching to NDJSON streaming for large files to avoid memory issues. The work is divided into two parts: robust, consistent inference with error handling and NDJSON support for incremental responses.

[Userunknown84]

if file length problem done in backend then on frontend it solves? the 2mb size is less for csv sometimes data is large.