CSV upload fails with UnicodeDecodeError when chardet misdetects Shift-JIS file as utf-8 #291

Bug description

CSVLoader fails with UnicodeDecodeError for non-UTF-8 CSV files (e.g. Shift-JIS / CP932 from Japanese Excel) when the file has a mostly-ASCII header. The encoding detection added in #173 returns "utf-8" because chardet only inspects the first 4096 bytes, which contain too few non-ASCII characters to detect Shift-JIS reliably.

Reproduce

1. Create a CSV in Excel (Japanese locale) with English column headers and a few Japanese values. Save as default CSV (Comma delimited) → file is CP932.
2. POST it to /text (or /embed).
3. → 400, server logs UnicodeDecodeError: 'utf-8' codec can't decode bytes ... invalid continuation byte inside langchain_community/document_loaders/csv_loader.py.

Root cause

app/utils/document_loader.py get_loader() for csv:

• detect_file_encoding() reads only 4096 bytes → chardet returns "utf-8" for a Shift-JIS file with ASCII-heavy header.
• The if encoding != "utf-8" branch is skipped, CSVLoader opens the original file with default UTF-8 → crash on header.

Affected versions

Reproduced on v0.7.0 (production). The same logic is still present in v0.7.1, v0.7.2, v0.7.3, v0.8.0, and current main.

Workaround

Re-save the file as UTF-8 (in Excel: "Save As" → "CSV UTF-8 (Comma delimited)").

Possible fixes

A few directions, in order of invasiveness:

1. Retry fallback: on UnicodeDecodeError from CSVLoader.load(), retry with common non-UTF-8 encodings (cp932, shift_jis, gb18030, latin-1) before giving up.
2. Larger detection sample: increase the chardet sample from 4096 bytes to e.g. 64 KB or the full file (CSV files are typically small).
3. Validate the detected encoding: after chardet.detect(), try decoding the whole file with the detected encoding; if it fails, fall back to candidate encodings.

Happy to send a PR for whichever direction you prefer.

Environment

• rag_api v0.7.0
• Python 3.10 (official image)
• File: Japanese CSV (CP932) with ASCII column headers