SemL0

SemL0 is a Rust prototype of a query-semantic LSM storage engine for dynamic property graphs. It attaches semantic summaries to CSR segments, uses graph access signatures to prune reads, feeds read observations back into L0 compaction, and preserves semantic partitions while cascading data to higher levels.

This repository is the public kernel release. It contains the database kernel, its single command-line binary, and regression tests. It is not an end-to-end paper artifact: paper drafts, paper-specific experiment runners, benchmark traces, generated stores, evaluation results, and third-party LDBC drivers are intentionally not included.

中文说明 · Startup guide · 中文启动说明 · Architecture · 中文架构

Architecture at a glance

flowchart LR
    A["Graph API / SNB adapter"] --> B["MemGraph"]
    B -->|"flush"| C["L0 CSR segments"]
    C --> D["Semantic metadata"]
    A --> E["Graph access signature"]
    E --> F["Semantic pruning"]
    D --> F
    F --> G["CSR read + MVCC visibility"]
    G --> H["Read feedback"]
    H --> I["Maintenance scheduler"]
    I --> J["Semantic L0 → L1 compaction"]
    J --> K["LevelMergePolicy"]
    K --> L["L2+ semantic segments"]
    C --> M["Manifest + schema catalog"]
    J --> M
    L --> M
    M --> N["Recovery / reopen"]

Loading

The detailed document covers module ownership, write and read paths, compaction policy, schema and tombstone safety, and recovery invariants.

What is implemented

• LSM graph engine with MemGraph, immutable CSR segments, manifest replay, and MVCC-visible reads.
• Semantic segment metadata for source label, edge type, degree class, destination label, property presence, schema epoch, and tombstone safety.
• Exact-proof segment pruning with reason-specific metrics.
• Read-feedback-driven L0 selection and optional automatic maintenance.
• Semantic L0-to-L1 compaction and fanout-driven L1+ cascading.
• Compaction reports containing logical update bytes and pruning-surface measurements before and after each merge.
• Schema evolution and schema-safe reopen.
• Dynamic graph view plus an LDBC SNB-compatible HTTP adapter.
• Blocking I/O by default, with optional Linux direct-I/O and io_uring features.

Query-interface scope

Graph access signatures are conservative segment-admission descriptors, not row-level property filters. Under its segment-level admission rule, with_required_property may reject a segment when complete metadata proves that the property is absent. with_absent_or_default_property remains conservative because a property-presence summary cannot prove absence for each visible edge. Therefore, get_neighbors_by_signature must not be interpreted as an exact per-edge property predicate or as the exact MVCC filtering path.

Exact row-level presence and absence are exposed separately through the prototype boundaries get_neighbors_with_present_property_prototype and get_neighbors_with_absent_property_prototype. Both paths retain matching and non-matching candidate versions until MVCC selects the latest visible version of each logical edge. A result is returned only when that version is an insert and satisfies the requested predicate; in particular, a newer non-matching insert or tombstone still blocks an older matching insert.

The per-segment property-presence bitmap is a property-decode optimization, not a visibility shortcut. When the bitmap proves that a segment lacks a property, the exact paths can avoid decoding that segment's property region. They still read the corresponding edge records and include them in the MVCC merge so that blockers are preserved. Unrepresentable property identifiers and incomplete summaries fall back conservatively.

The optional two-hop mode in storage-bench is a driver-composed workload. It performs one one-hop read, selects a bounded frontier, and issues another one-hop read for each selected destination. It is neither a native two-hop storage operator nor the paper's formal experiment runner.

Build and test

Rust 1.92.0 is the pinned and tested toolchain. Linux is the primary runtime for all I/O backends; the default blocking backend is portable.

git clone https://github.com/WhiteGive-base/SemL0.git
cd SemL0

cargo test --all-targets
cargo build --release --bin lsmgraph

Optional Linux backends:

cargo build --release --bin lsmgraph --features direct-io
cargo build --release --bin lsmgraph --features uring

Quick start with LDBC SNB data

No dataset is bundled. Point --input at an LDBC SNB root containing dynamic/ and static/ directories. The smallest import path only requires dynamic/person_knows_person_0_0.csv.

export SNB_DATA=/path/to/social_network
export SEML0_STORE=./store/sf1

# Import the person-knows relation and compact L0.
cargo run --release --bin lsmgraph -- import \
  --input "$SNB_DATA" \
  --data-dir "$SEML0_STORE" \
  --relation person_knows \
  --compact

# Inspect persisted levels and metrics.
cargo run --release --bin lsmgraph -- stats \
  --data-dir "$SEML0_STORE"

# Check imported edges against the source CSV.
cargo run --release --bin lsmgraph -- validate-knows \
  --input "$SNB_DATA" \
  --data-dir "$SEML0_STORE" \
  --max-vertices 1024

For all topology relations use --relation all-topology; for topology plus properties use --relation snb-full.

The SNB HTTP adapter requires the vertex, edge-property, and adjacency sidecars created by snb-full. A person_knows or all-topology store is not enough. Import the full relation set into a separate store, then start the adapter:

export SEML0_FULL_STORE=./store/sf1-full

cargo run --release --bin lsmgraph -- import \
  --input "$SNB_DATA" \
  --data-dir "$SEML0_FULL_STORE" \
  --relation snb-full \
  --compact

cargo run --release --bin lsmgraph -- snb-server \
  --data-dir "$SEML0_FULL_STORE" \
  --host 127.0.0.1 \
  --port 9090

From another shell:

curl http://127.0.0.1:9090/
curl http://127.0.0.1:9090/metrics

The first request returns {"status":"ok"}. External LDBC validation files are not bundled; pass them explicitly to snb-validate, snb-validate-batch, or snb-validate-mixed.

Useful commands

# Show every command.
cargo run --release --bin lsmgraph -- --help

# Inspect neighbors for an encoded vertex ID.
cargo run --release --bin lsmgraph -- neighbors \
  --data-dir "$SEML0_STORE" \
  --src 72057594037927937

# Compact the highest-scoring L0 partition.
cargo run --release --bin lsmgraph -- compact \
  --data-dir "$SEML0_STORE" \
  --auto-pick

# Select the Linux io_uring backend (binary must include the uring feature).
cargo run --release --features uring --bin lsmgraph -- \
  --io-backend uring stats --data-dir "$SEML0_STORE"

Repository boundary

src/                         database kernel and SNB adapter
tests/                       kernel correctness and regression tests
README.md / README-cn.md     project entry points
LSMGRAPH-STARTUP-GUIDE*.md   bilingual startup and validation guides
LSMGRAPH-ARCHITECTURE*.md    bilingual architecture and invariants

Generated stores, datasets, experiment runners, evaluation artifacts, paper materials, and third-party drivers should remain outside the repository.

License

SemL0 is licensed under the Apache License 2.0.