Skip to content

ci: optimize CI workflow with smoke test and nextest - #151

Open
qishipengqsp wants to merge 4 commits into
TuGraph-family:masterfrom
qishipengqsp:feat/ci
Open

ci: optimize CI workflow with smoke test and nextest#151
qishipengqsp wants to merge 4 commits into
TuGraph-family:masterfrom
qishipengqsp:feat/ci

Conversation

@qishipengqsp

Copy link
Copy Markdown
Collaborator

Summary

  • Add a smoke test job (build + unit tests) that runs after lint checks for quick early feedback
  • Merge build and test into a single build_and_test job to avoid duplicate compilation across platforms
  • Replace cargo test with cargo nextest for faster, more reliable test execution
  • Combine the separate wasm check and wasm_test jobs into a single wasm job
  • Add doc tests step to the build_and_test job

Test plan

  • Verify CI pipeline passes on this branch
  • Confirm smoke test runs before full build_and_test matrix
  • Verify nextest output format and test results are correct
  • Confirm WASM build and test run in combined job

🤖 Generated with Claude Code

qishipengqsp and others added 4 commits April 16, 2026 22:00
- Add comprehensive user guide with GQL syntax reference
- Add detailed architecture documentation covering storage engine,
  query engine, transaction management, and vector index
- Update README with documentation links and quick start examples

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Fix RUSTSEC-2026-0097 - Rand is unsound with a custom logger using
`rand::rng()`. The vulnerability affects versions >= 0.7, < 0.9.3.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- Add smoke test job for quick early feedback (build + unit tests)
- Merge build and test into single job to avoid duplicate compilation
- Replace cargo test with cargo nextest for faster test execution
- Combine wasm check and wasm test into single job
- Add doc tests step to build_and_test job

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Copilot AI review requested due to automatic review settings May 14, 2026 12:03

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

The PR is described as a CI workflow optimization, but it actually combines two largely unrelated change sets: a CI restructuring and a large documentation addition. On the CI side, the lint → build → test pipeline is reshaped into lint → smoke → (build_and_test matrix, build_no_std, wasm, docs), with cargo test swapped to cargo nextest run, doc tests added, and the previously separate wasm/wasm_test jobs merged. On the docs side, the PR rewrites README.md and adds two large Chinese-language guides (docs/user-guide.md, docs/architecture.md) covering installation, GQL syntax, architecture diagrams, and code snippets describing internal types.

Changes:

  • Restructure .github/workflows/ci.yml: introduce a smoke job, merge build+test into build_and_test, merge WASM check+test, and re-point downstream jobs to depend on smoke.
  • Rewrite README.md with project links, examples, and an architecture diagram; add docs/user-guide.md (GQL/Shell user guide) and docs/architecture.md (system architecture and extension guide).
  • Adopt cargo-nextest@0.9.88 for the main test step and add an explicit cargo test --doc step.

Reviewed changes

Copilot reviewed 4 out of 4 changed files in this pull request and generated 8 comments.

File Description
.github/workflows/ci.yml Adds smoke gate, merges build/test and wasm jobs, switches main tests to nextest, adds doc-test step.
README.md Replaces TBA placeholders with quick-start, examples, architecture diagram, and feature list; adds links to new docs.
docs/user-guide.md New ~660-line Chinese user guide covering install, Shell, GQL DDL/DML/DQL, types, functions, vector search, procedures, tuning.
docs/architecture.md New ~1253-line Chinese architecture doc with layered diagrams and embedded Rust snippets for storage, planner, executor, MVCC, and DiskANN.
Comments suppressed due to low confidence (2)

docs/architecture.md:912

  • Many code snippets and trait definitions in this file (e.g., VectorIndex trait with ann_search/search/soft_delete/save/load signatures, MemTransaction fields, CheckpointConfig, Operation/DeltaOp enums, Binder and PlanNode/PhysicalNode variants) are presented as accurate Rust definitions but are not auto-generated from the source. Hand-maintained code listings in architecture docs quickly drift out of sync with the implementation. Consider either pruning these to high-level diagrams/descriptions, or adding a note that snippets are illustrative and may lag the source, and pointing readers at the actual source files for the authoritative definitions.
```rust
// 内存图结构
pub struct MemoryGraph {
    // 顶点存储:ID -> 版本化顶点
    pub(super) vertices: DashMap<VertexId, VersionedVertex>,

    // 边存储:ID -> 版本化边
    pub(super) edges: DashMap<EdgeId, VersionedEdge>,

    // 邻接表:顶点ID -> 邻接容器
    pub(super) adjacency_list: DashMap<VertexId, AdjacencyContainer>,

    // 向量索引
    pub(super) vector_indices: DashMap<VectorIndexKey, Arc<RwLock<Box<dyn VectorIndex>>>>,
}

// 邻接表容器
pub(super) struct AdjacencyContainer {
    pub(super) incoming: Arc<SkipSet<Neighbor>>,  // 入边
    pub(super) outgoing: Arc<SkipSet<Neighbor>>,  // 出边
}

// 版本化数据结构 (MVCC)
pub(super) struct VersionChain<D: Clone> {
    pub(super) current: RwLock<CurrentVersion<D>>,
    pub(super) undo_ptr: RwLock<UndoPtr>,  // 撤销链指针
}

关键设计

  1. 并发控制: 使用 DashMap 实现高效的并发 HashMap,减少锁竞争
  2. 邻接表: 使用无锁跳表 SkipSet 存储邻接关系,支持高效遍历
  3. 版本链: MVCC 版本链支持快照读取

AP 存储 (OLAP)

AP 存储面向分析查询,位于 storage/src/ap/

核心数据结构

pub struct OlapStorage {
    // ID 映射
    pub logic_id_counter: AtomicU64,
    pub dense_id_map: DashMap<VertexId, VertexId>,  // 稀疏ID -> 密集ID

    // 列式存储
    pub vertices: RwLock<Vec<OlapVertex>>,
    pub edges: RwLock<Vec<EdgeBlock>>,
    pub property_columns: RwLock<Vec<PropertyColumn>>,

    // 压缩存储
    pub is_edge_compressed: AtomicBool,
    pub compressed_edges: RwLock<Vec<CompressedEdgeBlock>>,
    pub is_property_compressed: AtomicBool,
    pub compressed_properties: RwLock<Vec<CompressedPropertyColumn>>,
}

// 边块 (CSR 格式)
pub const BLOCK_CAPACITY: usize = 256;
pub struct EdgeBlock {
    pub src_id: VertexId,
    pub edges: Vec<Edge>,
}

压缩策略

// Delta 编码压缩
pub struct CompressedEdgeBlock {
    pub delta_bit_width: u8,                          // 增量位宽
    pub first_dst_id: VertexId,                       // 起始目标ID
    pub compressed_dst_ids: BitVec<u64, Lsb0>,       // 压缩的目标ID
    pub label_ids: [Option<LabelId>; BLOCK_CAPACITY],
}

持久化层

数据库文件格式

+------------------+--------------------------+-----------------------+
|  Header (256B)   |  Checkpoint Region (Var) |    WAL Region (Var)   |
+------------------+--------------------------+-----------------------+
0                 256            header.wal_offset                   EOF

文件头结构

pub struct DbFileHeader {
    pub magic: [u8; 8],           // "MINIGU\0\0"
    pub version: u32,             // 文件格式版本
    pub header_size: u32,         // 头大小 (256字节)
    pub flags: DbFileFlags,       // 特性标志位
    pub checkpoint_offset: u64,   // 检查点区域偏移
    pub checkpoint_length: u64,   // 检查点区域长度
    pub wal_offset: u64,          // WAL区域偏移
    pub wal_length: u64,          // WAL区域长度
    pub last_lsn: u64,            // 最后的日志序列号
    pub last_commit_ts: u64,      // 最后提交时间戳
    pub header_crc: u32,          // CRC32校验和
}

WAL (Write-Ahead Log)

pub struct RedoEntry {
    pub lsn: u64,                  // 日志序列号
    pub txn_id: Timestamp,         // 事务ID
    pub iso_level: IsolationLevel, // 隔离级别
    pub op: Operation,             // 操作类型
}

pub enum Operation {
    BeginTransaction(Timestamp),
    CommitTransaction(Timestamp),
    AbortTransaction,
    Delta(DeltaOp),  // 数据变更
}

pub enum DeltaOp {
    DelVertex(VertexId),
    DelEdge(EdgeId),
    CreateVertex(Vertex),
    CreateEdge(Edge),
    SetVertexProps(VertexId, SetPropsOp),
    SetEdgeProps(EdgeId, SetPropsOp),
    AddLabel(LabelId),
    RemoveLabel(LabelId),
}

检查点机制

pub struct GraphCheckpoint {
    pub meta: CheckpointMetadata,
    pub vertices: HashMap<VertexId, SerializedVertex>,
    pub edges: HashMap<EdgeId, SerializedEdge>,
    pub adjacency_list: HashMap<VertexId, SerializedAdjacency>,
}

// 自动检查点触发
pub struct CheckpointConfig {
    pub wal_threshold: usize,  // WAL条目阈值,默认1000
}

查询引擎

查询引擎位于 minigu/gql/,采用经典的 Parser → Planner → Executor 架构。

查询处理流程

┌─────────────┐    ┌─────────────┐    ┌─────────────┐    ┌─────────────┐
│   GQL Text  │───►│   Lexer     │───►│   Parser    │───►│    AST      │
└─────────────┘    └─────────────┘    └─────────────┘    └─────────────┘
                                                              │
                                                              ▼
┌─────────────┐    ┌─────────────┐    ┌─────────────┐    ┌─────────────┐
│   Result    │◄───│  Executor   │◄───│  Optimizer  │◄───│   Binder    │
└─────────────┘    └─────────────┘    └─────────────┘    └─────────────┘
                         │                                     │
                         ▼                                     ▼
                   ┌─────────────┐                      ┌─────────────┐
                   │ Physical    │                      │ Logical     │
                   │ Plan        │                      │ Plan        │
                   └─────────────┘                      └─────────────┘

解析器 (Parser)

位于 gql/parser/,使用 Logos + Winnow 实现。

词法分析 (Lexer)

// 使用 Logos 定义 Token
#[derive(Logos, Clone, Copy, Debug, PartialEq, Eq, Hash)]
#[logos(skip r"[ \t\r\n\f]+")]
#[logos(skip r"--[^\n]*")]
#[logos(skip r"//[^\n]*")]
pub enum TokenKind {
    // 关键字
    #[token("MATCH", ignore_case)]
    Match,
    #[token("RETURN", ignore_case)]
    Return,
    #[token("WHERE", ignore_case)]
    Where,
    // ...

    // 标识符和字面量
    #[regex(r"[a-zA-Z_][a-zA-Z0-9_]*", ignore_case)]
    RegularIdentifier,
    #[regex(r"'[^']*'")]
    CharacterStringLiteral,
    #[regex(r"[0-9]+")]
    UnsignedInteger,
    // ...
}

语法分析 (Parser)

// 使用 Winnow 解析器组合子
pub fn parse_gql(gql: &str) -> Result<Spanned<Program>, Error> {
    let tokens = Lexer::new(gql).collect::<Result<Vec<_>, _>>()?;
    let input = LocatedSlice::new(&tokens);
    program.parse(input).map_err(Into::into)
}

// 解析 MATCH 语句示例
fn match_statement(input: &mut Input) -> PResult<Spanned<MatchStatement>> {
    let start = input.start();
    let _ = TokenKind::Match.parse_next(input)?;
    let pattern = graph_pattern.parse_next(input)?;
    let where_clause = opt(where_clause).parse_next(input)?;
    let end = input.end();
    Ok(Spanned::new(
        MatchStatement { pattern, where_clause },
        start..end,
    ))
}

AST 结构

// 程序入口
pub struct Program {
    pub activity: OptSpanned<ProgramActivity>,
    pub session_close: Option<Spanned<SessionCloseCommand>>,
}

// 图模式
pub struct GraphPattern {
    pub match_mode: Option<Spanned<MatchMode>>,
    pub element_bindings: Spanned<ElementBindings>,
    pub where_clause: Option<Spanned<WhereClause>>,
}

// 节点模式
pub struct NodePattern {
    pub variable: Option<Spanned<Ident>>,
    pub label_expression: Option<Spanned<LabelExpression>>,
    pub predicate: Option<Spanned<ElementPatternPredicate>>,
}

// 边模式
pub struct EdgePattern {
    pub direction: EdgeDirection,
    pub variable: Option<Spanned<Ident>>,
    pub label_expression: Option<Spanned<LabelExpression>>,
    pub predicate: Option<Spanned<ElementPatternPredicate>>,
    pub quantifier: Option<Spanned<GraphPatternQuantifier>>,
}

查询规划器 (Planner)

位于 gql/planner/,负责将 AST 转换为执行计划。

绑定器 (Binder)

pub struct Binder<'a> {
    catalog: &'a dyn CatalogProvider,
    current_schema: Option<SchemaRef>,
    home_schema: Option<SchemaRef>,
    current_graph: Option<NamedGraphRef>,
    home_graph: Option<NamedGraphRef>,
    active_data_schema: Option<DataSchema>,
}

impl Binder<'_> {
    pub fn bind(&self, procedure: &Procedure) -> PlanResult<BoundStatement> {
        // 1. 名称解析
        // 2. 类型检查
        // 3. Schema 推导
        // 4. 标签 ID 解析
    }
}

逻辑计划节点

pub enum PlanNode {
    // 扫描
    LogicalMatch(LogicalMatch),
    LogicalVertexPropertyFetch(LogicalVertexPropertyFetch),
    LogicalEdgePropertyFetch(LogicalEdgePropertyFetch),

    // 变换
    LogicalFilter(LogicalFilter),
    LogicalProject(LogicalProject),
    LogicalSort(LogicalSort),
    LogicalLimit(LogicalLimit),
    LogicalOffset(LogicalOffset),

    // 连接
    LogicalHashJoin(LogicalHashJoin),

    // 向量搜索
    LogicalVectorIndexScan(LogicalVectorIndexScan),

    // DDL
    LogicalCreateVectorIndex(LogicalCreateVectorIndex),
    LogicalDropVectorIndex(LogicalDropVectorIndex),

    // 其他
    LogicalOneRow(LogicalOneRow),
    LogicalExplain(LogicalExplain),
    LogicalCall(LogicalCall),
}

物理计划节点

pub enum PhysicalNode {
    // 扫描
    PhysicalNodeScan(PhysicalNodeScan),
    PhysicalExpand(PhysicalExpand),
    PhysicalVertexPropertyFetch(PhysicalVertexPropertyFetch),

    // 变换
    PhysicalFilter(PhysicalFilter),
    PhysicalProject(PhysicalProject),
    PhysicalSort(PhysicalSort),
    PhysicalLimit(PhysicalLimit),
    PhysicalOffset(PhysicalOffset),

    // 连接
    PhysicalHashJoin(PhysicalHashJoin),

    // 向量搜索
    PhysicalVectorIndexScan(PhysicalVectorIndexScan),

    // 聚合
    PhysicalAggregate(PhysicalAggregate),
}

优化规则

// 向量索引重写规则
pub struct VectorIndexScanRewrite;

impl OptimizerRule for VectorIndexScanRewrite {
    fn apply(&self, plan: &PlanNode) -> PlanResult<Option<PlanNode>> {
        // 检测模式: Sort(VECTOR_DISTANCE) + LIMIT APPROXIMATE
        // 重写为: HashJoin(VectorIndexScan, PropertyFetch)
        if let PlanNode::LogicalSort(sort) = plan {
            if let PlanNode::LogicalLimit(limit) = sort.child.as_ref() {
                if limit.is_approximate {
                    // 执行重写
                    return self.rewrite_to_vector_index_scan(plan);
                }
            }
        }
        Ok(None)
    }
}

查询执行器 (Executor)

位于 gql/execution/,采用 Volcano 模型实现。

执行器接口

pub trait Executor: Debug + Send {
    fn next_chunk(&mut self) -> Option<ExecutionResult<DataChunk>>;
}

// 类型别名
pub type BoxedExecutor = Box<dyn Executor>;

执行器构建

pub struct ExecutorBuilder {
    session: SessionContext,
}

impl ExecutorBuilder {
    pub fn build(self, plan: &PlanNode) -> BoxedExecutor {
        match plan {
            PlanNode::PhysicalNodeScan(scan) => self.build_node_scan(scan),
            PlanNode::PhysicalExpand(expand) => self.build_expand(expand),
            PlanNode::PhysicalFilter(filter) => self.build_filter(filter),
            PlanNode::PhysicalProject(project) => self.build_project(project),
            PlanNode::PhysicalSort(sort) => self.build_sort(sort),
            PlanNode::PhysicalLimit(limit) => self.build_limit(limit),
            PlanNode::PhysicalHashJoin(join) => self.build_hash_join(join),
            PlanNode::PhysicalVectorIndexScan(scan) => self.build_vector_scan(scan),
            PlanNode::PhysicalAggregate(agg) => self.build_aggregate(agg),
            // ...
        }
    }
}

核心执行器

// 过滤执行器
pub struct FilterExecutor {
    child: BoxedExecutor,
    predicate: Box<dyn Evaluator>,
}

impl Executor for FilterExecutor {
    fn next_chunk(&mut self) -> Option<ExecutionResult<DataChunk>> {
        while let Some(result) = self.child.next_chunk() {
            let chunk = result?;
            let mask = self.predicate.evaluate(&chunk)?.as_bool_mask();
            if mask.any() {
                return Some(Ok(chunk.filter(&mask)));
            }
        }
        None
    }
}

// 投影执行器
pub struct ProjectExecutor {
    child: BoxedExecutor,
    expressions: Vec<Box<dyn Evaluator>>,
}

impl Executor for ProjectExecutor {
    fn next_chunk(&mut self) -> Option<ExecutionResult<DataChunk>> {
        self.child.next_chunk().map(|result| {
            let chunk = result?;
            let columns: Vec<ArrayRef> = self.expressions
                .iter()
                .map(|expr| expr.evaluate(&chunk)?.into_array())
                .collect();
            Ok(DataChunk::new(columns))
        })
    }
}

// 扩展执行器 (图遍历)
pub struct ExpandExecutor<S: ExpandSource> {
    child: BoxedExecutor,
    input_column_index: usize,
    edge_labels: Option<Vec<Vec<LabelId>>>,
    target_vertex_labels: Option<Vec<Vec<LabelId>>>,
    source: Arc<S>,
}

impl<S: ExpandSource> Executor for ExpandExecutor<S> {
    fn next_chunk(&mut self) -> Option<ExecutionResult<DataChunk>> {
        // 从子执行器获取顶点ID
        // 通过邻接表扩展边
        // 返回扩展结果
    }
}

表达式求值

pub trait Evaluator: Debug + Send + Sync {
    fn evaluate(&self, chunk: &DataChunk) -> ExecutionResult<DatumRef>;
}

// 常量求值器
pub struct Constant {
    value: Datum,
}

// 列引用求值器
pub struct ColumnRef {
    index: usize,
}

// 二元运算求值器
pub struct Binary {
    left: Box<dyn Evaluator>,
    op: BinaryOp,
    right: Box<dyn Evaluator>,
}

// 向量距离求值器
pub struct VectorDistanceEvaluator {
    left: Box<dyn Evaluator>,
    right: Box<dyn Evaluator>,
    metric: VectorMetric,
}

事务管理

事务管理位于 minigu/transaction/storage/src/tp/

MVCC 架构

┌─────────────────────────────────────────────────────────────────┐
│                      Transaction Manager                         │
│  ┌───────────────────────────────────────────────────────────┐  │
│  │  active_txns: SkipMap<Timestamp, Arc<Transaction>>        │  │
│  │  committed_txns: SkipMap<Timestamp, Arc<Transaction>>     │  │
│  │  commit_lock: Mutex<()>                                    │  │
│  │  latest_commit_ts: AtomicU64                               │  │
│  │  watermark: AtomicU64                                      │  │
│  └───────────────────────────────────────────────────────────┘  │
└─────────────────────────────────────────────────────────────────┘
                                │
                                ▼
┌─────────────────────────────────────────────────────────────────┐
│                        Transaction                               │
│  ┌───────────────────────────────────────────────────────────┐  │
│  │  start_ts: Timestamp          // 开始时间戳                │  │
│  │  commit_ts: OnceLock<Timestamp> // 提交时间戳              │  │
│  │  isolation_level: IsolationLevel                          │  │
│  │  vertex_reads: DashSet<VertexId>  // 读集合                │  │
│  │  edge_reads: DashSet<EdgeId>      // 边读集合              │  │
│  │  undo_buffer: Vec<UndoEntry>      // 撤销日志              │  │
│  │  redo_buffer: Vec<RedoEntry>      // 重做日志              │  │
│  └───────────────────────────────────────────────────────────┘  │
└─────────────────────────────────────────────────────────────────┘

事务结构

pub struct MemTransaction {
    graph: Arc<MemoryGraph>,
    isolation_level: IsolationLevel,
    start_ts: Timestamp,
    commit_ts: OnceLock<Timestamp>,
    txn_id: Timestamp,

    // 读集合 (用于可串行化验证)
    vertex_reads: DashSet<VertexId>,
    edge_reads: DashSet<EdgeId>,

    // 日志缓冲
    undo_buffer: RwLock<Vec<Arc<UndoEntry>>>,
    redo_buffer: RwLock<Vec<RedoEntry>>,

    is_handled: Arc<AtomicBool>,
}

隔离级别

pub enum IsolationLevel {
    Snapshot,       // 快照隔离
    Serializable,   // 可串行化
}

提交协议

pub fn commit_at(&self, commit_ts: Option<Timestamp>, skip_wal: bool) -> StorageResult<Timestamp> {
    // 1. 获取提交时间戳
    let commit_ts = global_timestamp_generator().next()?;

    // 2. 获取全局提交锁
    let _guard = self.graph.txn_manager.commit_lock.lock().unwrap();

    // 3. 可串行化验证
    if let IsolationLevel::Serializable = self.isolation_level {
        self.validate_read_sets()?;
    }

    // 4. 设置提交时间戳
    self.commit_ts.set(commit_ts)?;

    // 5. 处理撤销缓冲区
    for undo_entry in undo_entries.iter() {
        // 更新版本链
    }

    // 6. 写入 WAL 并刷盘
    for entry in redo_entries {
        self.graph.persistence.append_wal(&entry)?;
    }
    self.graph.persistence.flush_wal()?;

    // 7. 更新最新提交时间戳
    self.graph.txn_manager.finish_transaction(self)?;

    // 8. 检查自动检查点
    self.graph.check_auto_checkpoint()?;

    Ok(commit_ts)
}

垃圾回收

fn garbage_collect(&self, graph: &MemoryGraph) -> Result<(), StorageError> {
    let min_read_ts = self.low_watermark().raw();

    // 1. 收集过期事务
    for entry in self.committed_txns.iter() {
        if entry.key().raw() > min_read_ts { break; }
        expired_txns.push(entry.value().clone());
    }

    // 2. 清理版本链中的过期版本
    self.cleanup_version_chains(graph, &expired_undo_entries)?;

    // 3. 移除过期事务记录
    for txn in expired_txns {
        self.committed_txns.remove(&txn.commit_ts()?);
    }
}

向量索引

向量索引基于 DiskANN 算法实现,位于 storage/diskann-rs/

架构设计

┌─────────────────────────────────────────────────────────────────┐
│                     Vector Index Interface                       │
│  ┌───────────────────────────────────────────────────────────┐  │
│  │  build(vectors) -> ()                                      │  │
│  │  ann_search(query, k, l, filter) -> Vec<(id, distance)>   │  │
│  │  insert(vectors) -> ()                                     │  │
│  │  soft_delete(ids) -> ()                                    │  │
│  │  save(path) / load(path)                                   │  │
│  └───────────────────────────────────────────────────────────┘  │
└─────────────────────────────────────────────────────────────────┘
                                │
                                ▼
┌─────────────────────────────────────────────────────────────────┐
│                      InMemANNAdapter                             │
│  ┌───────────────────────────────────────────────────────────┐  │
│  │  inner: Box<dyn ANNInmemIndex<f32>>  // DiskANN 核心       │  │
│  │  dimension: usize                                          │  │
│  │  node_to_vector: DashMap<u64, u32>   // 节点->向量映射     │  │
│  │  vector_to_node: ShardedVectorMap    // 向量->节点映射     │  │
│  │  next_vector_id: AtomicU32                                  │  │
│  └───────────────────────────────────────────────────────────┘  │
└─────────────────────────────────────────────────────────────────┘

核心接口

pub trait VectorIndex: Send + Sync {
    /// 构建索引
    fn build(&mut self, vectors: &[(u64, &[f32])]) -> StorageResult<()>;

    /// 近似最近邻搜索
    fn ann_search(
        &self,
        query: &[f32],
        k: usize,
        l_value: u32,
        filter_mask: Option<&dyn DiskANNFilterMask>,
        should_pre: bool,
    ) -> StorageResult<Vec<(u64, f32)>>;

    /// 带过滤的搜索
    fn search(
        &self,
        query: &[f32],
        k: usize,
        l_value: u32,
        filter_mask: Option<&FilterMask>,
        should_pre: bool,
    ) -> StorageResult<Vec<(u64, f32)>>;

    /// 插入向量
    fn insert(&mut self, vectors: &[(u64, &[f32])]) -> StorageResult<()>;

    /// 软删除
    fn soft_delete(&mut self, node_ids: &[u64]) -> StorageResult<()>;

    /// 持久化
    fn save(&mut self, path: &str) -> StorageResult<()>;
    fn load(&mut self, path: &str) -> StorageResult<()>;
}

分片映射优化

// 分片向量映射,减少锁竞争
pub struct ShardedVectorMap {
    shards: Vec<RwLock<Vec<Option<u64>>>>,  // 16个分片
    shard_bits: u32,
}

docs/user-guide.md:617

  • docs/user-guide.md documents many GQL features (e.g., CREATE NODE TYPE/CREATE EDGE TYPE with property blocks, SET GRAPH, INSERT (a)-[e:FRIEND]-(b) undirected edges, MATCH (p:Person|Animal) / (p:Person&Employee) / (p:!Bot) label expressions, STARTS WITH, IS LABELED, LIMIT APPROXIMATE, CALL show_graph/import_graph/export_graph/create_test_graph, SET TRANSACTION ISOLATION LEVEL ..., START TRANSACTION/COMMIT/ROLLBACK, EXPLAIN, VECTOR_DISTANCE, VECTOR(128) type, etc.) as if they are all supported. Before merging, please verify each documented construct is actually implemented and accepted by the parser/planner — otherwise users will hit confusing errors following the guide. If some features are aspirational, mark them clearly as planned/experimental.
### 基本操作示例

```sql
-- 创建 Schema
CREATE SCHEMA my_schema;

-- 创建图
CREATE GRAPH my_graph;

-- 设置当前图
SET GRAPH my_graph;

-- 创建顶点标签
CREATE NODE TYPE Person {
  name STRING,
  age INT,
  email STRING
};

-- 创建边标签
CREATE EDGE TYPE KNOWS {
  since INT
};

-- 插入顶点
INSERT (a:Person {name: 'Alice', age: 30}),
       (b:Person {name: 'Bob', age: 25}),
       (c:Person {name: 'Charlie', age: 35});

-- 插入边
INSERT (a:Person {name: 'Alice'})-[e:KNOWS {since: 2020}]->(b:Person {name: 'Bob'});

-- 查询顶点
MATCH (p:Person)
RETURN p.name, p.age
ORDER BY p.age DESC
LIMIT 10;

-- 模式匹配查询
MATCH (a:Person)-[e:KNOWS]->(b:Person)
RETURN a.name, b.name, e.since;

-- 过滤查询
MATCH (p:Person)
WHERE p.age > 25
RETURN p.name, p.age;

GQL 查询语言

数据定义语言 (DDL)

创建 Schema

CREATE SCHEMA schema_name;

删除 Schema

DROP SCHEMA schema_name;

创建图

-- 创建空图
CREATE GRAPH graph_name;

-- 创建指定类型的图
CREATE GRAPH graph_name OF TYPE graph_type_name;

删除图

DROP GRAPH graph_name;

���建顶点类型

CREATE NODE TYPE LabelName {
  property1 Type1,
  property2 Type2,
  ...
};

创建边类型

CREATE EDGE TYPE EdgeLabel {
  property1 Type1,
  property2 Type2,
  ...
} FROM SourceLabel TO TargetLabel;

创建向量索引

CREATE VECTOR INDEX index_name
FOR (n:Label)
ON n.property_name
DIMENSION 128
METRIC L2;

删除向量索引

DROP VECTOR INDEX index_name;

数据操作语言 (DML)

插入顶点

-- 插入单个顶点
INSERT (a:Person {name: 'Alice', age: 30});

-- 插入多个顶点
INSERT (a:Person {name: 'Alice'}),
       (b:Person {name: 'Bob'}),
       (c:Person {name: 'Charlie'});

插入边

-- 插入有向边
INSERT (a:Person {name: 'Alice'})-[e:KNOWS {since: 2020}]->(b:Person {name: 'Bob'});

-- 插入无向边
INSERT (a:Person)-[e:FRIEND {since: 2021}]-(b:Person);

更新属性

MATCH (p:Person {name: 'Alice'})
SET p.age = 31;

删除元素

-- 删除顶点
MATCH (p:Person {name: 'Alice'})
DELETE p;

-- 删除边
MATCH (a:Person)-[e:KNOWS]->(b:Person)
WHERE a.name = 'Alice' AND b.name = 'Bob'
DELETE e;

数据查询语言 (DQL)

MATCH 语句

MATCH 是图模式匹配的核心语句:

-- 简单顶点匹配
MATCH (p:Person)
RETURN p;

-- 边模式匹配
MATCH (a:Person)-[e:KNOWS]->(b:Person)
RETURN a, e, b;

-- 多跳路径匹配
MATCH (a:Person)-[e1:KNOWS]->(b:Person)-[e2:KNOWS]->(c:Person)
RETURN a.name, c.name;

-- 可变长度路径
MATCH (a:Person)-[e:KNOWS*1..3]->(b:Person)
RETURN a.name, b.name;

边方向

-- 出边 (指向右侧)
MATCH (a)-[e]->(b)

-- 入边 (指向左侧)
MATCH (a)<-[e]-(b)

-- 无向边 (任意方向)
MATCH (a)-[e]-(b)

-- 双向边
MATCH (a)<-[e]->(b)

标签表达式

-- 单标签
MATCH (p:Person)

-- 多标签 (或)
MATCH (p:Person|Animal)

-- 标签交集 (与)
MATCH (p:Person&Employee)

-- 标签取反
MATCH (p:!Bot)

WHERE 过滤

-- 比较过滤
MATCH (p:Person)
WHERE p.age > 25
RETURN p;

-- 逻辑组合
MATCH (p:Person)
WHERE p.age > 25 AND p.name STARTS WITH 'A'
RETURN p;

-- 存在性检查
MATCH (p:Person)
WHERE p.email IS NOT NULL
RETURN p;

-- 标签检查
MATCH (p)
WHERE p IS LABELED Person
RETURN p;

RETURN 投影

-- 返回属性
MATCH (p:Person)
RETURN p.name, p.age;

-- 使用别名
MATCH (p:Person)
RETURN p.name AS name, p.age AS age;

-- 表达式计算
MATCH (p:Person)
RETURN p.name, p.age * 2 AS double_age;

-- 聚合函数
MATCH (p:Person)
RETURN COUNT(p) AS person_count;

-- 去重
MATCH (p:Person)
RETURN DISTINCT p.age;

ORDER BY 排序

-- 升序排序
MATCH (p:Person)
RETURN p.name, p.age
ORDER BY p.age ASC;

-- 降序排序
MATCH (p:Person)
RETURN p.name, p.age
ORDER BY p.age DESC;

-- 多字段排序
MATCH (p:Person)
RETURN p.name, p.age
ORDER BY p.age DESC, p.name ASC;

LIMIT 和 OFFSET 分页

-- 限制结果数量
MATCH (p:Person)
RETURN p
LIMIT 10;

-- 分页查询
MATCH (p:Person)
RETURN p
ORDER BY p.name
LIMIT 10 OFFSET 20;

GROUP BY 分组

MATCH (p:Person)
RETURN p.age, COUNT(p) AS count
GROUP BY p.age;

数据类型

基本类型

类型 说明 示例
BOOL 布尔值 true, false
INT 64位整数 42, -100
FLOAT 64位浮点数 3.14, -0.5
STRING 字符串 'hello', "world"
BYTES 字节串 x'48656c6c6f'

时间类型

类型 说明 示例
DATE 日期 DATE '2024-01-15'
TIME 时间 TIME '14:30:00'
DATETIME 日期时间 DATETIME '2024-01-15T14:30:00'
DURATION 时间间隔 DURATION 'P1Y2M3D'

复合类型

类型 说明 示例
LIST<T> 列表 [1, 2, 3]
RECORD 记录 {name: 'Alice', age: 30}
VECTOR 向量 VECTOR [1.0, 2.0, 3.0]

图元素类型

类型 说明
NODE 顶点引用
EDGE 边引用
PATH 路径

内置函数

聚合函数

函数 说明
COUNT(x) 计数
SUM(x) 求和
AVG(x) 平均值
MIN(x) 最小值
MAX(x) 最大值
COLLECT(x) 收集为列表

字符串函数

函数 说明
UPPER(s) 转大写
LOWER(s) 转小写
TRIM(s) 去除首尾空白
SUBSTRING(s, start, len) 子字符串
CONCAT(s1, s2, ...) 字符串连接
LENGTH(s) 字符串长度

数值函数

函数 说明
ABS(x) 绝对值
FLOOR(x) 向下取整
CEIL(x) 向上取整
ROUND(x) 四舍五入
SQRT(x) 平方根
POWER(x, y) 幂运算

图函数

函数 说明
ELEMENT_ID(e) 获取元素 ID
LABELS(n) 获取顶点标签列表
PROPERTIES(e) 获取元素属性
START_NODE(e) 获取边的起始顶点
END_NODE(e) 获取边的目标顶点

向量搜索

miniGU 内置向量索引支持,可以进行高效的向量相似性搜索。

创建向量属性

-- 创建带向量属性的顶点类型
CREATE NODE TYPE Article {
  title STRING,
  content STRING,
  embedding VECTOR(128)
};

-- 插入带向量的顶点
INSERT (a:Article {
  title: 'Introduction to Graphs',
  content: '...',
  embedding: VECTOR [0.1, 0.2, 0.3, ...]
});

创建向量索引

CREATE VECTOR INDEX article_embedding_idx
FOR (a:Article)
ON a.embedding
DIMENSION 128
METRIC L2;

支持的距离度量:

  • L2: 欧几里得距离
  • COSINE: 余弦相似度
  • INNER_PRODUCT: 内积

向量相似性搜索

-- 计算向量距离
MATCH (a:Article)
RETURN a.title, VECTOR_DISTANCE([0.1, 0.2, ...], a.embedding, L2) AS distance
ORDER BY distance
LIMIT 10;

-- 使用索引进行近似搜索
MATCH (a:Article)
RETURN a.title, VECTOR_DISTANCE([0.1, 0.2, ...], a.embedding, L2) AS distance
ORDER BY distance
LIMIT APPROXIMATE 10;

LIMIT APPROXIMATE 提示查询优化器使用向量索引进行近似最近邻搜索,可以显著提升查询性能。


存储过程

miniGU 提供内置存储过程用于数据库管理。

查看图信息

CALL show_graph()
YIELD name, vertex_count, edge_count
RETURN name, vertex_count, edge_count;

导入图数据

CALL import_graph('/path/to/data.json')
YIELD status
RETURN status;

导出图数据

CALL export_graph('/path/to/output.json')
YIELD status
RETURN status;

创建测试图

CALL create_test_graph()
YIELD status
RETURN status;

配置与调优

数据库文件

miniGU 使用单文件存储格式,默认数据文件为 .minigu 扩展名。

文件结构:

+----------------+--------------------------+-----------------------+
|  Header (256B) |  Checkpoint Region (Var) |    WAL Region (Var)   |
+----------------+--------------------------+-----------------------+

事务隔离级别

-- 设置隔离级别
SET TRANSACTION ISOLATION LEVEL SNAPSHOT;
SET TRANSACTION ISOLATION LEVEL SERIALIZABLE;

-- 开始事务
START TRANSACTION;

-- 提交事务
COMMIT;

-- 回滚事务
ROLLBACK;

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Comment thread .github/workflows/ci.yml
Comment on lines +79 to +95
# Quick smoke test to catch obvious issues early
smoke:
needs: [ typos, toml, fmt, clippy, machete, deny ]
name: Smoke Test
runs-on: ubuntu-latest
timeout-minutes: 10
steps:
- uses: actions/checkout@v4
- uses: actions-rust-lang/setup-rust-toolchain@v1
- uses: Swatinem/rust-cache@v2
- name: Quick build check
run: cargo build --features ${{ env.DEFAULT_FEATURES }} -p minigu-cli
- name: Quick unit tests
run: cargo test --lib --features ${{ env.DEFAULT_FEATURES }} -- --test-threads=4

# Combined build and test job to avoid duplicate compilation
build_and_test:
Comment thread docs/user-guide.md
DROP GRAPH graph_name;
```

#### ���建顶点类型
Comment thread .github/workflows/ci.yml
Comment on lines +79 to +96
# Quick smoke test to catch obvious issues early
smoke:
needs: [ typos, toml, fmt, clippy, machete, deny ]
name: Smoke Test
runs-on: ubuntu-latest
timeout-minutes: 10
steps:
- uses: actions/checkout@v4
- uses: actions-rust-lang/setup-rust-toolchain@v1
- uses: Swatinem/rust-cache@v2
- name: Quick build check
run: cargo build --features ${{ env.DEFAULT_FEATURES }} -p minigu-cli
- name: Quick unit tests
run: cargo test --lib --features ${{ env.DEFAULT_FEATURES }} -- --test-threads=4

# Combined build and test job to avoid duplicate compilation
build_and_test:
needs: [ smoke ]
Comment thread .github/workflows/ci.yml
Comment on lines +88 to +92
- uses: Swatinem/rust-cache@v2
- name: Quick build check
run: cargo build --features ${{ env.DEFAULT_FEATURES }} -p minigu-cli
- name: Quick unit tests
run: cargo test --lib --features ${{ env.DEFAULT_FEATURES }} -- --test-threads=4
Comment thread .github/workflows/ci.yml

# Combined build and test job to avoid duplicate compilation
build_and_test:
needs: [ smoke ]
Comment thread docs/architecture.md
Comment on lines +132 to +134
├── minigu-cli/ # 命令行工具
├── minigu-test/ # 测试框架
Comment thread docs/user-guide.md
Comment on lines +35 to +36
- Rust 1.75+ (推荐使用最新稳定版)
- Cargo 包管理器
Comment thread README.md
详细文档TBA
- [用户指南](docs/user-guide.md) - 安装、快速开始、GQL 语法参考
- [架构设计](docs/architecture.md) - 系统架构、核心模块、扩展指南
- [解析器开发指南](docs/parser/development.md) - GQL 解析器开发文档
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants