fix(isolation): auto-detect overlayfs and fallback for Docker compatibility

[NikkeTryHard]

Summary

• Add is_overlayfs() detection using statfs with OVERLAYFS_SUPER_MAGIC
• Auto-skip overlay mounts when project root is on overlayfs (Docker overlay2)
• Keep namespace isolation, RO root, and tmpfs — only overlay mounts are skipped
• Add overlay filesystem check to tach self-test

Closes #83

How it works

Docker uses overlay2 storage driver → container root is already overlayfs → Linux kernel doesn't support nested overlay → tach's overlay mount fails with EINVAL.

Fix: At worker startup, setup_filesystem() checks if project_root is on overlayfs. If so, skip the overlay mount steps but keep all other isolation (mount namespace, RO root remount, tmpfs). Workers get fork-only isolation instead of crashing.

Verification

• Docker: [tach:isolation] Project root is on overlayfs (Docker detected). Overlay mounts disabled — using fork-only isolation. ✅
• Tests pass without --no-isolation inside Docker: 4/4 pass ✅
• tach self-test from overlayfs dir: [WARN] Overlay FS: Project root is on overlayfs ✅
• tach self-test from native fs: [PASS] Overlay FS: Native filesystem ✅
• 857/857 unit tests pass, fmt ✅, clippy ✅

Summary by CodeRabbit

Release Notes

• New Features
  • Added overlayfs filesystem detection to the diagnostics pipeline, warning users when projects reside on overlayfs with remediation guidance.
  • Automatic isolation adaptation: system now detects overlayfs environments and switches to fork-only isolation mode, bypassing overlay mounting on affected filesystems.

[coderabbitai]

Caution

Review failed

The pull request is closed.

📝 Walkthrough

Walkthrough

The changes implement auto-detection of overlayfs filesystems and graceful fallback to fork-only isolation when nested overlays are detected. New public diagnostic checks expose overlayfs detection capabilities, and the isolation setup conditionally disables overlay mounting when the project root resides on overlayfs, preventing kernel errors in Docker environments.

Changes

Cohort / File(s)	Summary
Diagnostics Integration src/core/diagnostics.rs	Added public check_overlay_filesystem() function that detects overlayfs on the project root and emits diagnostic warnings with optional remediation details. Integrates into the main diagnostic pipeline and output display.
Module Re-exports src/isolation/mod.rs	Updated public exports to expose is_overlayfs alongside setup_filesystem, making overlayfs detection available throughout the codebase.
Overlayfs Detection and Fallback src/isolation/namespace.rs	Added overlayfs detection via is_overlayfs() using libc statfs calls and OVERLAYFS_SUPER_MAGIC constant. Modified setup_filesystem() to skip conditional overlay mounting when overlayfs is detected. Introduced public helper functions (worker_base_dir, tmp_overlay_options, project_overlay_options, is_isolation_disabled) and tests for overlay option formatting and isolation flag handling.

Estimated code review effort

🎯 3 (Moderate) | ⏱️ ~20 minutes

Poem

🐰 A rabbit hops through Docker's nested streams,
Detecting overlays in all their schemes. ✨
When layers stack too high, the kernel frowns,
So we fall back graceful without breaking crowns! 🏰
Fork-only isolation saves the day,
Tests can now run! Hip-hip-hooray! 🎉

✨ Finishing touches

• 📝 Generate docstrings

🧪 Generate unit tests (beta)

• Create PR with unit tests
• Post copyable unit tests in a comment
• Commit unit tests in branch fix/issue-83-overlay-detection

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands and usage tips.}