feat: add parallel ingestion consumers and logging

[peeeteeer]

Summary
This PR adds configurable parallel processing for batched document ingestion and improves observability around file embedding.

Why
Large file ingestion can be slow and hard to diagnose. The existing batching pipeline reduced memory pressure, but processed embedding/database batches with a single consumer. This change allows multiple async consumers per file so batches can be embedded and inserted in parallel, while adding detailed logs to understand file size, chunk count, route, user/file identifiers, elapsed time, and memory usage during ingestion.

What Changed

• Added PARALLEL_EXECUTION env config, defaulting to 2.
• Updated the async ingestion pipeline to run multiple batch consumers when batching is enabled.
• Preserved deterministic result ordering by collecting batch results by batch number.
• Added rollback tracking for successfully inserted batches.
• Added detailed ingestion logging for /embed, /embed-upload, and /local/embed.
• Added process memory details to ingestion logs.
• Documented the new env var in README.md.
• Added a regression test covering concurrent batch processing and ordered results.

Peter Rothlaender ginkgo.rothlaender@external.daimlertruck.com on behalf of Daimler Truck AG.
Provider & Legal Notice | Daimler Truck
Copyright (c) {2026} Daimler Truck AG

[peeeteeer]

@codex review

[chatgpt-codex-connector]

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: e71112ac5f

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

[peeeteeer]

@codex review

[chatgpt-codex-connector]

Codex Review: Didn't find any major issues. Can't wait for the next one!

Reviewed commit: 0ba7df0f21

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

[peeeteeer]

@codex review

[chatgpt-codex-connector]

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: e9b2623d75

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

[peeeteeer]

@codex review

[chatgpt-codex-connector]

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 65a6ef7424

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

[peeeteeer]

@codex review

[chatgpt-codex-connector]

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: ccd9537930

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

[peeeteeer]

@codex review

[chatgpt-codex-connector]

Codex Review: Didn't find any major issues. What shall we delve into next?

Reviewed commit: d8d20cfb8d

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

[danny-avila]

I completed an independent review of the current head (d8d20cf). The prior Codex findings are fixed, but I found one remaining cancellation edge case: cancelling an ingestion request bypasses the Exception handler while shielded inserts can still finish, so committed vectors may escape rollback.

I prepared and validated the fix in 0aca7be. It also corrects the touched README batching default (500, not 0) and restores consistent README line endings.

Validation: tests/test_batch_processing.py (32 passed), the full non-integration suite (199 passed, 6 skipped), and the Docker-backed integration suite (21 passed).

GitHub does not support the maintainer-edit toggle for this organization-owned fork, and the current account has read-only access there, so the direct push was rejected. Peter (or another Daimler fork collaborator), please cherry-pick 0aca7be798c1685fe969d671b4b3ba0467ede9c5 onto parallel-execution; alternatively, a Daimler repo admin can grant danny-avila write access. Once the PR head updates, I can finish the fresh Codex review cycle.

[danny-avila]

Superseded by #317, which preserves this PR’s full commit history and attribution and adds the independently reviewed cancellation-cleanup fix on a repository-owned branch. Closing this PR so CI and Codex review continue on the pushable replacement.