feat(filesystem): add DirectFile with O_DIRECT/F_NOCACHE support

[bneb]

Summary

Direct I/O support (#34) via O_DIRECT (Linux) / F_NOCACHE (macOS), with alignment enforced at the type level.

Design

A #[repr(align(4096))] struct Block([u8; BLOCK_SIZE]) carries the alignment invariant. It propagates through BlockStore::read/write → PageCache frames → FileBlockStore superblock → DirectFile. The type system rejects unaligned or wrong-sized buffers at compile time — no unsafe, no runtime size checks, no bounce buffer.

DirectFile provides two constructors: open() (attempts O_DIRECT/F_NOCACHE, falls back silently) and open_buffered(). Both take &Block/&mut Block for I/O.

HeapOptions { frames, direct_io } and EngineBuilder consolidate the constructor surface.

When to use

Benchmark sweep (16K rows, ~300 pages, pool sizes 32–512 frames) places the inversion point at 384 frames (3 MiB). When pool_frames ≥ dataset_pages, direct I/O matches buffered throughput without double-buffering in the OS page cache.

Scope

Does not propagate Block to LSM SSTable reads (the LSM uses Cache, not PageCache). Does not change the default I/O path.

Verification

Clippy clean, 291 tests pass, benchmarks compile and run.

[nevzheng]

Hi bneb,

Thanks for the PR — I can see the effort and thought put into it. I have a few concerns, and I'm happy to iterate on them with you. I'd like to start with the following.

Copy-through strategy doesn't seem right to me. We're avoiding changes to the upper layers by creating a local region of alignment, when we should be propagating the alignment assumption through the stack. As maintainer, I'm open to a larger blast radius if it cleans up the code.

Changes I'd like to see:

• Make 4 KiB (4096) alignment a property of the buffer type, not a per-I/O fixup. Introduce an aligned Block:

  #[repr(align(4096))]
  struct Block([u8; BLOCK_SIZE]);   // aligned + fixed-size by construction, zero unsafe

• Propagate Block through the stack — the page-cache frames, the superblock buffer, and BlockStore::read/write — so an unaligned or wrong-sized buffer can't be passed in the first place.

This simplifies the code by deleting the special-casing the current approach needs:

• AlignedBuf and its unsafe — the bounce buffer in direct.rs — gone; aligned frames are DMA'd directly.
• The per-I/O if direct { bounce } else { … } branch — gone.
• The check_len / BadBlockSize runtime size checks in file.rs — gone; the type fixes the size.

The alignment assumption then lives in one type, instead of being implied across the I/O path.

Thanks!
Nevin

PS: Added an implementation note / hint. A Options Struct + single object would be helpful. #34 (comment) #34 (comment)

[nevzheng]

Putting this here to be direct — I wrote the fuller version in #34, but here's what I'd like, plainly:

1. Blocks in the block cache should be aligned. A page-aligned Block type (#[repr(align(4096))]), so direct I/O works without the copy-through bounce or the unsafe allocator.

2. A single File abstraction (with FileOptions) — one type that owns the open policy (direct vs buffered) instead of open / open_direct twins. Lives in crates/filesystem/src/file.rs.

3. Propagate it to the places we open files today:

   • crates/filesystem/src/block/file.rs (FileBlockStore) — page cache + workspace
   • crates/journal/src/lib.rs (Journal)
   • crates/lsm/src/storage/sstable.rs (SSTables)

happy to talk through scope, split it, or pair.

[bneb]

Thanks for the review. Rewritten to propagate alignment as you described.

• Block now flows through BlockStore, PageCache frames, and DirectFile.
• AlignedBuf, the per-I/O branch, and check_len/BadBlockSize are all removed.

I did not propagate Block to LSM SSTable reads. This is because presently the LSM uses Cache<BlockId, …>, not PageCache. Its I/O path is separate. Follow-up if you want it.

Re: FileOptions on DirectFile: I kept the two-constructor API since it is just two variants. Consolidated Heap and StorageEngine with options/builder structs instead. Happy to add DirectFileOptions if you prefer consistency throughout.

[nevzheng]

Hi @bneb I decided to close this in favor of #36.
Honestly I had a hard time explaining what I was after, and in the interest of efficiency I ended up building it myself. #36 landed the shape I had in mind in fewer lines.

And while this PR added benchmarks, I decided that I'm not ready to accept and maintain them, so I'd rather not pull them in right now.

Thanks for the work here. I'm sorry for any confusion.