Skip to content

fix: validate document chunker input types - #60

Open
AkashKundu114 wants to merge 4 commits into
mainfrom
fix/document-chunk-input-validation
Open

AkashKundu114 wants to merge 4 commits into
mainfrom
fix/document-chunk-input-validation

Conversation

@AkashKundu114

@AkashKundu114 AkashKundu114 commented Nov 16, 2026 •

Copy link
Copy Markdown
Owner

Summary

  • Preserve the existing empty-input behavior for None and empty strings.
  • Reject non-string document content and non-integer chunk/overlap settings with clear TypeError messages.
  • Add focused tests for invalid types and the existing overlap-clamping behavior.

Validation

  • Added tests/core/test_document_indexer.py.
  • Tests have not been executed in this session; please review CI before merging.

Summary by Sourcery

Validate document chunker inputs while preserving existing empty-input and overlap behavior.

Bug Fixes:

  • Validate document chunker inputs and raise clear TypeError messages for non-string text and non-integer chunk settings while preserving empty-input handling.

Tests:

  • Add focused coverage for invalid chunker input types, empty inputs, and overlap clamping.

@sourcery-ai

sourcery-ai Bot commented Nov 16, 2026 •

Copy link
Copy Markdown

Reviewer's Guide

Implementation

Validate chunking arguments and document text before processing, while preserving empty-input and overlap-clamping behavior; the same file also has related formatting-only cleanup in its indexing functions.

  • backend/app/core/document_indexer.py L8-19 (+7 small edits)

Tests

Cover empty inputs, rejected text and parameter types, and the existing overlap-clamping boundaries.

  • tests/core/test_document_indexer.py L1-31

Tips and commands

Interacting with Sourcery

  • Trigger a new review: Comment @sourcery-ai review on the pull request.
  • Continue discussions: Reply directly to Sourcery's review comments.
  • Generate a GitHub issue from a review comment: Ask Sourcery to create an
    issue from a review comment by replying to it. You can also reply to a
    review comment with @sourcery-ai issue to create an issue from it.
  • Generate a pull request title: Write @sourcery-ai anywhere in the pull
    request title to generate a title at any time. You can also comment
    @sourcery-ai title on the pull request to (re-)generate the title at any time.
  • Generate a pull request summary: Write @sourcery-ai summary anywhere in
    the pull request body to generate a PR summary at any time exactly where you
    want it. You can also comment @sourcery-ai summary on the pull request to
    (re-)generate the summary at any time.
  • Generate reviewer's guide: Comment @sourcery-ai guide on the pull
    request to (re-)generate the reviewer's guide at any time.
  • Resolve all Sourcery comments: Comment @sourcery-ai resolve on the
    pull request to resolve all Sourcery comments. Useful if you've already
    addressed all the comments and don't want to see them anymore.
  • Dismiss all Sourcery reviews: Comment @sourcery-ai dismiss on the pull
    request to dismiss all existing Sourcery reviews. Especially useful if you
    want to start fresh with a new review - don't forget to comment
    @sourcery-ai review to trigger a new review!

Customizing Your Experience

Access your dashboard to:

  • Enable or disable review features such as the Sourcery-generated pull request
    summary, the reviewer's guide, and others.
  • Change the review language.
  • Add, remove or edit custom review instructions.
  • Adjust other review settings.

Getting Help

@sourcery-ai sourcery-ai Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hey - I've found 3 issues

Prompt for AI Agents
Please address the comments from this code review:

## Individual Comments

### Comment 1
<location path="backend/app/core/document_indexer.py" line_range="9" />
<code_context>

 def chunk_text(text: str, chunk_size: int = 1000, overlap: int = 200) -> list[str]:
-    if not text:
+    if text is None or text == "":
         return []
+    if not isinstance(text, str):
</code_context>
<issue_to_address>
**Invalid settings are accepted**

When text is empty or None and either chunking setting has an unsupported type, `chunk_text` returns before checking `chunk_size` and `overlap`, so callers receive an empty result instead of the promised `TypeError`.

Check `chunk_size` and `overlap` before returning for empty or `None` text.
</issue_to_address>

### Comment 2
<location path="backend/app/core/document_indexer.py" line_range="9" />
<code_context>

 def chunk_text(text: str, chunk_size: int = 1000, overlap: int = 200) -> list[str]:
-    if not text:
+    if text is None or text == "":
         return []
+    if not isinstance(text, str):
</code_context>
<issue_to_address>
**Array input raises ValueError**

When a multi-element NumPy array is passed as `text`, `text == ""` produces an array, so the `or` condition raises `ValueError` before the string-type guard and callers receive the wrong exception.

Handle `None` and validate that `text` is a string before comparing it with `""`.
</issue_to_address>

### Comment 3
<location path="tests/core/test_document_indexer.py" line_range="29-32" />
<code_context>
+        chunk_text("some content", overlap=overlap)
+
+
+def test_chunk_text_keeps_existing_overlap_clamping():
+    chunks = chunk_text("abcdefghijklmnopqrstuvwxyz", chunk_size=10, overlap=20)
+    assert chunks
+    assert all(len(chunk) <= 10 for chunk in chunks)
</code_context>
<issue_to_address>
**Overlap regression goes unnoticed**

When the implementation uses zero overlap instead of clamping the oversized overlap to `chunk_size - 1`, `test_chunk_text_keeps_existing_overlap_clamping` at lines 28–29 only checks that the result is nonempty and chunks are at most 10 characters long; with overlap clamping replaced by zero overlap, those assertions still pass and the regression goes unnoticed.

Assert the expected chunk boundaries or that adjacent chunks overlap by the clamped amount.
</issue_to_address>

Sourcery assessment

Approval pending. 3 findings to address first.

Blocking findings: backend/app/core/document_indexer.py:9, backend/app/core/document_indexer.py:9, tests/core/test_document_indexer.py:32


Sourcery is free for open source - if you like our reviews please consider sharing them ✨

Comment thread backend/app/core/document_indexer.py Resolved
Comment thread backend/app/core/document_indexer.py Resolved
Comment thread tests/core/test_document_indexer.py Resolved

@sourcery-ai sourcery-ai Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Sourcery assessment

Approved.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant