Skip to content

feat: implement global index registry for O(1) index discovery #198

@majiayu000

Description

@majiayu000

Summary

Currently, leann list scans directories to discover indexes, which can be slow for large directory trees. A global registry system would make index discovery O(1) by tracking all index paths centrally.

Proposed Solution

Implement a global registry at ~/.leann/indexes.json that stores the paths of all LEANN indexes:

{
  "indexes": [
    {
      "name": "my-docs",
      "path": "/Users/foo/projects/my-project/.leann/indexes/my-docs",
      "type": "cli",
      "created_at": "2024-12-25T10:00:00Z"
    },
    {
      "name": "app-index",
      "path": "/Users/foo/apps/data/app-index.leann",
      "type": "app",
      "created_at": "2024-12-25T11:00:00Z"
    }
  ]
}

Implementation Steps

  1. Register on build: When leann build creates an index, automatically register it in the global registry
  2. Register on API usage: When apps create indexes via the Python API, register them as well
  3. Update leann list: Read from registry instead of scanning directories
  4. Cleanup stale entries: Validate paths exist and remove stale entries periodically
  5. Migration: First run should scan existing indexes and populate the registry

Benefits

  • leann list becomes O(1) instead of O(n) directory scan
  • Works correctly even when running from $HOME or other large directories
  • Provides a single source of truth for all LEANN indexes

Related

Metadata

Metadata

Assignees

Labels

No labels
No labels

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions