Add incremental update mode to Box crawler

[adeelehsan]

Summary

• Adds incremental_update and hours_back config options to the Box crawler
• When enabled, only processes files modified within the hours_back window (single Box folder traversal)
• Detects files removed from Box and deletes them from Vectara, updating indexed.csv
• Uses the full indexing pipeline (docling, tables, OCR, permissions) — no shortcuts

Config

box_crawler:
  incremental_update: true
  hours_back: 8h  # supports: 6h, 24h, 2d

vectara:
  reindex: true  # needed for updated files

How it works

1. Scans all Box folders once → gets file IDs + modified_at
2. Filters to files modified within hours_back
3. Downloads and indexes only those files (new + updated)
4. Compares full Box file list vs indexed.csv → deletes orphaned docs from Vectara
5. If incremental_update is not set, crawler runs exactly as before (zero behavior change)

Test plan

• Run with incremental_update: true, hours_back: 24h on existing corpus with indexed.csv
• Verify only recently modified files are processed
• Verify deleted files are removed from Vectara
• Verify normal crawl mode still works when incremental_update is not set

🤖 Generated with Claude Code