Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
440 changes: 430 additions & 10 deletions ai-chat.js

Large diffs are not rendered by default.

38 changes: 19 additions & 19 deletions doc/vdl-neptune-search.md
Original file line number Diff line number Diff line change
Expand Up @@ -16,14 +16,14 @@ Zero-dependency, client-side hybrid search engine for the **[vd3 docs](https://v
▼ ▼
┌───────────────────────────┐ ┌─────────────────────────────────────┐
│ LAYER 1: Fuse.js │ │ LAYER 2: Transformers.js │
│ (triggers on keystroke) │ │ (triggers on Enter / submit)
│ │ │
│ • Lazy-loads on first │ │ • Lazy-loads vectors.json + model
│ keystroke │ │ on first submit (~23MB once)
│ • Fuzzy match across: │ │ • Embeds query with MiniLM-L6-v2
│ title (2.5), headings │ │ • Cosine similarity vs all docs
│ (2.0), keywords (2.5), │ │ • Returns top 10 (score > 0.30)
│ bodyText (1.0), │ │ • Computation: <10ms for 500 docs
│ (debounced keystroke) │ │ (debounced keystroke when ready;
│ │ │ Enter submits immediately)
│ • Lazy-loads on first │ │ • Preloads on UI mount (~23MB once)
│ keystroke │ │ • Embeds query with MiniLM-L6-v2
│ • Fuzzy match across: │ │ • Cosine similarity vs all docs
│ title (2.5), headings │ │ • Returns top 10 (score > 0.30)
│ (2.0), keywords (2.5), │ │ • Computation: <10ms for 500 docs
│ bodyText (1.0), │ │
│ classes (1.5), │ │ │
│ chunks.text (0.8) │ │ │
│ • Threshold: 0.45 │ │ │
Expand Down Expand Up @@ -94,7 +94,7 @@ const search = new NeptuneSearch();
const fuzzy = await search.search('button', { mode: 'fuzzy' });
console.log(fuzzy.merged); // [{ doc, score, source: 'fuzzy' }]

// Hybrid search (fuzzy + semantic, Enter-key behavior)
// Hybrid search (fuzzy + semantic)
const hybrid = await search.search('how do I make a button', { mode: 'hybrid' });
console.log(hybrid.merged); // [{ doc, score, source: 'semantic'|'fuzzy' }]
```
Expand Down Expand Up @@ -123,14 +123,14 @@ ui.mount();

| Layer | Trigger | Engine | Data |
|-------|---------|--------|------|
| **Fuzzy** | Every keystroke (debounced 150ms) | Fuse.js v7 (CDN) | `data/search-index.json` |
| **Semantic** | Enter key / form submit (model preloaded on UI mount) | Transformers.js v3 + `Xenova/all-MiniLM-L6-v2` (CDN, ~23MB) | `data/vectors.json` |
| **Fuzzy** | Debounced keystroke while semantic model is still loading | Fuse.js v7 (CDN) | `data/search-index.json` |
| **Semantic / hybrid** | Debounced keystroke once model is ready (Enter submits immediately) | Transformers.js v3 + `Xenova/all-MiniLM-L6-v2` (CDN, ~23MB) | `data/vectors.json` |
| **Merge** | After both complete | Custom ranker | Score-sorted interleave across semantic + fuzzy, deduped, capped |

### Eager Semantic Preload + Lazy Fuzzy

- **Fuzzy (Fuse.js)** loads on first use (`initFuzzy()` / first keystroke). The Vue UI also warms fuzzy on mount so category chips are ready.
- **Transformers.js + MiniLM** (`Xenova/all-MiniLM-L6-v2`, ~23MB quantized) starts loading as soon as the search UI mounts (`NeptuneSearchUI` / `VdlNeptuneSearchUI`), in the background. Fuzzy typing stays available while it downloads.
- **Transformers.js + MiniLM** (`Xenova/all-MiniLM-L6-v2`, ~23MB quantized) starts loading as soon as the search UI mounts (`NeptuneSearchUI` / `VdlNeptuneSearchUI`), in the background. Fuzzy typing stays available while it downloads; when the model becomes ready, the current query is re-run as hybrid automatically.
- Progress is shown via a non-blocking progress strip (`onSemanticProgress`), not by freezing the input.
- The model download happens once per browser session; subsequent visits/uses hit the browser cache. `initSemantic()` is promise-cached per `NeptuneSearch` instance (and the Vue UI reuses a module singleton across HMR / `v-if` remounts).
- Headless callers can still call `await search.initSemantic()` explicitly; UI mount already does this fire-and-forget.
Expand All @@ -141,7 +141,7 @@ ui.mount();
|--------------|----------|
| CDN blocked / network error (Fuse.js) | `search()` throws; caller handles |
| CDN blocked / network error (Transformers.js) | `_semanticFailed` flag set; `search()` falls back to fuzzy-only with `console.warn` |
| Model download fails mid-stream | Same as above; subsequent Enter presses use fuzzy-only |
| Model download fails mid-stream | Same as above; subsequent searches use fuzzy-only |
| Query < 2 characters | Returns empty results immediately |
| Malformed index/vector payloads | Deterministically rejected during initialization |
| Unsafe route/base URL or icon payload | Sanitized/fallback handling in render path |
Expand Down Expand Up @@ -255,7 +255,7 @@ await search.initSemantic();

##### `semanticSearch(query: string): Promise<SemanticResult[]>`

Performs a semantic search query. Embeds the query using MiniLM-L6-v2 and computes cosine similarity against all document vectors. Returns top 10 results above `semanticThreshold`. Awaits `initSemantic()` internally (no need to pre-call). UI components call `initSemantic()` on mount so the model is usually warm before the first Enter; use `onSemanticProgress()` to show download progress.
Performs a semantic search query. Embeds the query using MiniLM-L6-v2 and computes cosine similarity against all document vectors. Returns top 10 results above `semanticThreshold`. Awaits `initSemantic()` internally (no need to pre-call). UI components call `initSemantic()` on mount so the model is usually warm before the first auto-hybrid search; use `onSemanticProgress()` to show download progress.

```javascript
const results = await search.semanticSearch('how do I style cards');
Expand Down Expand Up @@ -369,16 +369,17 @@ new NeptuneSearchUI(options: NeptuneSearchUIOptions)
| `search` | `NeptuneSearch` | *(required)* | Headless search instance |
| `onResultClick` | `(result: MergedResult) => void` | `() => {}` | Callback fired when a result is clicked or selected via Enter |
| `placeholder` | `string` | `'Search docs…'` | Input placeholder text |
| `debounceMs` | `number` | `150` | Debounce delay for fuzzy search on keystroke |
| `showSemanticHint` | `boolean` | `true` | Show "Enter for AI search" hint in the input |
| `debounceMs` | `number` | `350` | Debounce delay for auto hybrid/fuzzy search on keystroke |
| `showSemanticHint` | `boolean` | `true` | Show "AI · fuzzy" capability hint in the input |
| `autofocus` | `boolean` | `true` | Focus the input on mount when safe (skips modals / other fields) |
| `baseUrl` | `string` | `'https://vanduo-oss.github.io/vd3-docs'` | Base URL for result card "Open docs" path links |
| `emptyMessage` | `string` | `'No docs found. Try another query or pick a category filter.'` | Message shown when search returns no results |

#### Methods

##### `mount(): void`

Builds the DOM, binds event listeners, and injects default styles. Starts a background `initSemantic()` preload (non-blocking). Idempotent — calling `mount()` on an already-mounted instance is a no-op.
Builds the DOM, binds event listeners, and injects default styles. Starts a background `initSemantic()` preload (non-blocking) and autofocuses the input when safe. Idempotent — calling `mount()` on an already-mounted instance is a no-op.

```javascript
ui.mount();
Expand All @@ -399,8 +400,7 @@ ui.destroy();
| `ArrowDown` | Dropdown open with results | Move selection down |
| `ArrowUp` | Dropdown open with results | Move selection up (or deselect) |
| `Enter` | Result selected | Trigger `onResultClick` for selected result |
| `Enter` | No selection, results visible | Run hybrid (semantic) search |
| `Enter` | No results, input focused | Run hybrid (semantic) search |
| `Enter` | No selection | Cancel debounce and run hybrid search immediately |
| `Escape` | Dropdown open | Close dropdown and blur input |
| `Cmd+K` / `Ctrl+K` | Anywhere on page | Focus search input |

Expand Down
131 changes: 89 additions & 42 deletions neptune-search.js
Original file line number Diff line number Diff line change
Expand Up @@ -403,8 +403,9 @@ export class NeptuneSearchUI {
this.search = options.search;
this.onResultClick = options.onResultClick ?? (() => {});
this.placeholder = options.placeholder ?? 'Search docs…';
this.debounceMs = options.debounceMs ?? 150;
this.debounceMs = options.debounceMs ?? 350;
this.showSemanticHint = options.showSemanticHint ?? true;
this.autofocus = options.autofocus ?? true;
this.baseUrl = options.baseUrl ?? DEFAULT_DOCS_BASE_URL;
this.emptyMessage = options.emptyMessage ?? 'No docs found. Try another query or pick a category filter.';

Expand All @@ -416,6 +417,10 @@ export class NeptuneSearchUI {
this._keyboardHandler = null;
this._clickOutsideHandler = null;
this._unsubscribeSemantic = null;
/** Last accepted query string (for enrich-on-ready). */
this._lastQuery = '';
/** Ensures model-ready enrichment runs at most once per warm-up. */
this._didEnrichOnReady = false;
/** Bumps when input/close/fuzzy runs so stale hybrid completions skip UI updates */
this._semanticSeq = 0;
}
Expand All @@ -427,19 +432,23 @@ export class NeptuneSearchUI {
this._buildDOM();
this._bindEvents();
this._mounted = true;
// Warm Transformers.js + MiniLM in the background so Enter feels instant.
// Warm Transformers.js + MiniLM in the background so hybrid auto-search is ready sooner.
// Fuzzy search remains available while this runs (non-blocking progress bar).
this._preloadSemantic();
this._tryAutofocus();
}

destroy() {
if (!this._mounted) return;
clearTimeout(this._debounceTimer);
this._unbindEvents();
this.container.innerHTML = '';
this._mounted = false;
this._elements = {};
this._results = [];
this._selectedIndex = -1;
this._lastQuery = '';
this._didEnrichOnReady = false;
}

/**
Expand Down Expand Up @@ -476,7 +485,7 @@ export class NeptuneSearchUI {
aria-activedescendant=""
/>
<span class="vdl-neptune-hint" aria-hidden="true">
${this.showSemanticHint ? '<kbd>Enter</kbd> for AI search' : ''}
${this.showSemanticHint ? 'AI · fuzzy' : ''}
</span>
</div>
<div class="vdl-neptune-dropdown" id="vdl-neptune-results" role="listbox" hidden>
Expand Down Expand Up @@ -558,33 +567,36 @@ export class NeptuneSearchUI {
}
}

/**
* Focus search when safe: skip open modals and other text fields, but allow
* taking focus from buttons after mount.
*/
_tryAutofocus() {
if (!this.autofocus || !this._elements.input) return;
const ae = document.activeElement;
if (ae?.closest?.('dialog, [role="dialog"], [aria-modal="true"]')) return;
if (ae && ae !== this._elements.input) {
const tag = ae.tagName;
if (
tag === 'INPUT' ||
tag === 'TEXTAREA' ||
tag === 'SELECT' ||
ae.isContentEditable
) {
return;
}
}
this._elements.input.focus({ preventScroll: true });
}

// ── Event Handlers ───────────────────────────────────────────────────

_onInput(e) {
const query = normalizeSearchQuery(e.target.value, {
maxLength: this.search?.queryMaxLength ?? 240,
});

clearTimeout(this._debounceTimer);
this._semanticSeq++;
this._elements.loader.hidden = true;

const queryCheck = validateSearchQuery(query, {
minLength: this.search?.queryMinLength ?? 2,
maxLength: this.search?.queryMaxLength ?? 240,
});
if (!queryCheck.allowed) {
this._clearResults();
return;
}

this._debounceTimer = setTimeout(async () => {
await this._runFuzzy(query);
}, this.debounceMs);
this._scheduleSearch(e.target.value);
}

_onKeyDown(e) {
const { dropdown, results, input } = this._elements;
const { dropdown, input } = this._elements;

if (!dropdown.hidden && this._results.length > 0) {
if (e.key === 'ArrowDown') {
Expand All @@ -604,14 +616,13 @@ export class NeptuneSearchUI {
if (this._selectedIndex >= 0) {
this._selectResult(this._results[this._selectedIndex]);
} else {
// Enter without selection → semantic search
this._runSemantic(input.value.trim());
this._scheduleSearch(input.value, { immediate: true });
}
return;
}
} else if (e.key === 'Enter') {
e.preventDefault();
this._runSemantic(input.value.trim());
this._scheduleSearch(input.value, { immediate: true });
return;
}
}
Expand All @@ -625,12 +636,20 @@ export class NeptuneSearchUI {
if (data.progress?.loaded && data.progress?.total) {
const pct = Math.round((data.progress.loaded / data.progress.total) * 100);
progressBar.style.width = `${pct}%`;
} else {
} else {
progressBar.style.width = '0%';
}
}
} else if (data.stage === 'ready') {
progress.hidden = true;
progressBar.style.width = '0%';
// Enrich current query once when the model first becomes ready.
if (!this._didEnrichOnReady && this._lastQuery) {
this._didEnrichOnReady = true;
clearTimeout(this._debounceTimer);
this._debounceTimer = setTimeout(() => {
this._runSearch(this._lastQuery);
}, 0);
}
} else if (data.stage === 'error') {
progress.hidden = true;
progressBar.style.width = '0%';
Expand All @@ -640,19 +659,37 @@ export class NeptuneSearchUI {

// ── Search Execution ─────────────────────────────────────────────────

async _runFuzzy(query) {
_scheduleSearch(rawQuery, { immediate = false } = {}) {
if (!this.search) return;
const query = normalizeSearchQuery(rawQuery, {
maxLength: this.search.queryMaxLength ?? 240,
});

clearTimeout(this._debounceTimer);
this._semanticSeq++;
const seq = this._semanticSeq;
const result = await this.search.search(query, { mode: 'fuzzy' });
if (seq !== this._semanticSeq) return;
this._results = result.merged;
this._selectedIndex = -1;
this._elements.loader.hidden = true;
this._renderResults();

const queryCheck = validateSearchQuery(query, {
minLength: this.search.queryMinLength ?? 2,
maxLength: this.search.queryMaxLength ?? 240,
});
if (!queryCheck.allowed) {
this._lastQuery = '';
this._clearResults();
return;
}

this._lastQuery = query;
if (immediate) {
this._runSearch(query, { forceHybrid: true });
return;
}
this._debounceTimer = setTimeout(() => {
this._runSearch(query);
}, this.debounceMs);
}

async _runSemantic(query) {
async _runSearch(query, { forceHybrid = false } = {}) {
if (!this.search) return;
const normalizedQuery = normalizeSearchQuery(query, {
maxLength: this.search.queryMaxLength ?? 240,
Expand All @@ -663,33 +700,42 @@ export class NeptuneSearchUI {
});
if (!queryCheck.allowed) return;

clearTimeout(this._debounceTimer);
const seq = this._semanticSeq;
const useHybrid = forceHybrid || this.search.isSemanticReady();
const seq = ++this._semanticSeq;
this._lastQuery = normalizedQuery;
this._results = [];
this._selectedIndex = -1;
this._elements.empty.hidden = true;
this._elements.results.innerHTML = '';
this._elements.loader.hidden = false;
const loaderText = this._elements.loader.querySelector('.vdl-neptune-loader-text');
if (loaderText) {
loaderText.textContent = useHybrid ? 'Searching with AI…' : 'Searching…';
}
this._elements.dropdown.hidden = false;
this._elements.input.setAttribute('aria-expanded', 'true');

try {
const result = await this.search.search(normalizedQuery, { mode: 'hybrid' });
const result = await this.search.search(normalizedQuery, {
mode: useHybrid ? 'hybrid' : 'fuzzy',
});
if (seq !== this._semanticSeq) return;
this._results = result.merged;
this._selectedIndex = -1;
this._elements.loader.hidden = true;
this._renderResults();
} catch (err) {
console.warn('[NeptuneUI] Semantic search error:', err);
console.warn('[NeptuneUI] Search error:', err);
if (seq !== this._semanticSeq) return;
this._elements.loader.hidden = true;
const result = await this.search.search(normalizedQuery, { mode: 'fuzzy' });
if (seq !== this._semanticSeq) return;
this._results = result.merged;
this._renderResults();
} finally {
this._elements.loader.hidden = true;
if (seq === this._semanticSeq) {
this._elements.loader.hidden = true;
}
}
}

Expand Down Expand Up @@ -795,6 +841,7 @@ export class NeptuneSearchUI {

_clearResults() {
this._semanticSeq++;
this._lastQuery = '';
this._results = [];
this._selectedIndex = -1;
this._elements.results.innerHTML = '';
Expand Down
24 changes: 23 additions & 1 deletion openspec/specs/vdl-neptune-search/spec.md
Original file line number Diff line number Diff line change
@@ -1,7 +1,8 @@
# vdl-neptune-search Specification

## Purpose
Hybrid in-browser docs search that warms the semantic path early so Enter-triggered AI search feels responsive.
Hybrid in-browser docs search that warms the semantic path early and runs hybrid search automatically as the user types (no Enter required for AI results).

## Requirements
### Requirement: Semantic preload on mount
When the Neptune search UI mounts, the system MUST start preloading the semantic search path (embedding model / transformers stack) in the background without blocking the fuzzy search path.
Expand All @@ -14,3 +15,24 @@ When the Neptune search UI mounts, the system MUST start preloading the semantic
- **WHEN** semantic preload is still in progress
- **THEN** the user can still use fuzzy / instant search results

### Requirement: Debounced auto hybrid search
The Neptune search UI MUST run search automatically after the user pauses typing, without requiring Enter. While the semantic model is not ready, results MAY be fuzzy-only; once ready, searches MUST use hybrid mode. Enter MUST still submit hybrid search immediately (canceling any pending debounce).

#### Scenario: Typing pauses triggers search
- **WHEN** the user types a valid query and pauses for the debounce interval
- **THEN** search runs without requiring Enter

#### Scenario: Enter submits immediately
- **WHEN** the user presses Enter with no result selected
- **THEN** any pending debounce is canceled and hybrid search runs immediately

#### Scenario: Enrich when model becomes ready
- **WHEN** semantic preload completes and a valid query is still in the input
- **THEN** results are refreshed with hybrid search

### Requirement: Autofocus search on mount
When the Neptune search UI mounts, the system SHOULD focus the search input so the user can type immediately, unless focus is already in a modal or another text field.

#### Scenario: Safe autofocus
- **WHEN** the UI mounts and no modal or other text field holds focus
- **THEN** the search input receives focus
Loading
Loading