Stability, performance, and UX hardening TODOs

[c4pt0r]

Stability, Performance, and UX Hardening TODOs

This issue consolidates the first whole-project AGFS diagnostic pass requested in Slock.

Ref inspected: origin/master @ cc9a09c5cd593ca661a36a9bfbd31bd37ee449d0 after:

• PR fix(agfs-shell): handle fd redirection tokens in lexer #19: lexer fd-redirection hang fix
• PR fix(queuefs): support SQLite queue DDL and regressions #20: QueueFS SQLite DDL/regression fix

Diagnostic inputs:

• Backend stability/performance: @Dev-1, Slock task feat: add gptfs plugin #6
• Shell/SDK/FUSE stability/performance: @dev-2, Slock task Fix the OpenHandle method of agfs-sdk #7
• User experience/install/docs/webapp: @dev-3, Slock task Add truncate functionality to AGFSNode, client, and server; implement… #8

P0 TODOs

• SQLFS2 identifier/destructive-operation hardening
  Harden dynamic identifiers and destructive paths in SQLFS2 (DROP DATABASE, DROP TABLE, table-qualified queries/inserts) with backend-specific validation/quoting and regression tests.

• HTTP/body limits and streaming write strategy
  Add request/body size limits, return 413 on overflow, and implement streaming/chunked write paths where supported. Current handlers and several plugins buffer whole bodies or objects.

• Fresh source/server startup must work
  Fix fresh-checkout startup so source docs and make dev do not require an absent config.yaml. Either default to config.example.yaml, generate config.yaml, or update all quickstarts consistently.

• Installer/systemd config path must be valid
  Make install.sh, installed agfs-server.service, and post-install instructions agree on config location. The service points at /etc/agfs.yaml, but installer does not create it.

• Integrated webapp missing-build detection
  Before starting agfs-shell --webapp, detect missing agfs-shell/webapp/dist/index.html and fail fast with an actionable build command instead of serving 404.

• SDK streaming progress timeouts
  Replace Python SDK timeout=None and Go SDK Timeout: 0 streaming behavior with configurable per-chunk progress timeouts so stalled servers cannot hang clients indefinitely.

• Bound FUSE cache memory and stop cleanup goroutine cleanly
  Add maxEntries or byte-budget eviction (LRU or equivalent) and a shutdown path/context for the cache cleanup goroutine.

• Narrow highest-risk broad exception swallowing in agfs-shell
  Audit the broad except Exception surface, starting with user-facing commands (upload, ls, llm, tail, completer). Re-raise or emit structured errors outside documented external boundaries.

P1 TODOs

• Server readiness and mount-status semantics
  /health can report healthy even when configured mounts fail asynchronously. Add readiness/mount status reporting and decide which configured mounts are startup blockers.

• QueueFS SQLite concurrent dequeue semantics
  Fix or explicitly guard/document SQLite single-consumer mode. Current SQLite dequeue can duplicate-deliver under concurrent consumers.

• VectorFS consistency/retry model
  Add status/outbox/retry/compensation behavior for partial failures across S3, metadata, chunk state, and async indexing.

• Unmount handle invalidation
  Ensure plugin unmount closes/invalidates open handles under that mount path and add post-unmount handle behavior tests.

• Webapp reproducibility and audit warnings
  Commit an npm lockfile; resolve or track Vite/esbuild and Monaco/DOMPurify warnings; keep dev server localhost-bound until resolved.

• Backend/plugin/handler regression coverage
  Add focused tests for handlers, SQLFS/SQLFS2, StreamFS, HeartbeatFS, KVFS, HTTPFS/ProxyFS, S3FS, and VectorFS, using fakes/mocks where external services are required.

• Python 3.14 shell arithmetic compatibility
  Replace deprecated ast.Num usage with ast.Constant in agfs_shell/expression.py and keep arithmetic tests green.

• Incrementally decompose Shell.execute() / shell.py
  Extract function-definition detection, control-flow dispatch, redirection prep, and pipeline build into smaller helpers across incremental PRs.

• Optimize StreamingInputStream.readline()
  Avoid byte-by-byte read(1) loops; buffer chunks and split on newline more efficiently.

• Python SDK lifecycle support
  Add AGFSClient.close(), __enter__, and __exit__ so the underlying requests.Session can be closed predictably.

• First-run docs matrix and CLI naming
  Add a first-run matrix for install script, Docker, source, webapp, FUSE, and MCP. Clarify agfs-server, agfs, agfs-shell, and agfs-fuse naming/aliases.

P2 TODOs

• Add FUSE troubleshooting and platform fallback docs.
• Add MCP preflight/smoke-test docs.
• Add webapp JS smoke tests (Vitest or equivalent).
• Add SQLFS benchmarks for path scans, tree operations, and range/blob reads.
• Add opt-in repeatable integration jobs for S3/TiDB/VectorFS using local services where possible.
• Expose backend pool/backpressure/timeout limits and mount/indexing status through config and observability surfaces.
• Add a top-level docs/troubleshooting structure for issue links, screenshots, runbooks, and known warnings.

Evidence Summary

Verified commands across the diagnostic pass included:

go test ./...
python3 scripts/run_failpoint_tests.py
go test -race ./pkg/mountablefs ./pkg/plugins/queuefs ./pkg/plugins/localfs ./pkg/plugins/memfs ./pkg/plugins/vectorfs
go run cmd/server/main.go -c config.example.yaml -addr :18081
curl -sf http://localhost:18081/api/v1/health
PYTHONPATH=. .venv/bin/python -m pytest agfs-shell/tests/ --timeout=15
npm install
npm audit --audit-level=moderate
npm run build

Notable evidence:

• source startup without explicit config fails on fresh checkout because config.yaml is absent
• integrated webapp mode serves 404 if webapp/dist is missing
• webapp install currently reports moderate npm audit findings and Vite CJS warning
• backend handler/plugin coverage is sparse, with several packages reporting no tests
• shell/SDK/FUSE risks include no streaming progress timeouts, unbounded FUSE cache, broad exception swallowing, and thin test coverage in core modules

Execution Ownership

Initial owner mapping:

• @Dev-1: backend/server/plugin/test/CI hardening items
• @dev-2: shell/SDK/FUSE/runtime hardening items
• @dev-3: docs/UX/webapp/install/release-readiness items
• @cindy: final review, PR coordination, and issue/task orchestration