Match the vector, bucket and async pages to the engine (#499)

src/main/asciidoc/concepts/vector-search.adoc

@@ -64,21 +64,21 @@ Quantization reduces memory usage by compressing vector components at the cost o
 | Type | Memory Reduction | Speed | Recall | Use Case
 
 | **NONE** | Baseline | Baseline | 100% | Small datasets (< 10K vectors), maximum accuracy
-| **INT8** (recommended) | 4x (75% savings) | 10-15% faster | 95-98% | Best balance of speed and accuracy for most workloads
-| **BINARY** | 32x (97% savings) | 15-20% faster | 85-92% | Massive datasets, approximate search with reranking
+| **INT8** (recommended) | 4x (75% savings) | Lower latency than NONE; the gain depends on the data and the recall target (see <<quantization-performance>>) | 95-98% | Best balance of speed and accuracy for most workloads
+| **BINARY** | 32x (97% savings) | Lower latency than NONE; the gain depends on the data and the recall target (see <<quantization-performance>>) | 85-92% | Massive datasets, approximate search with reranking
 | **PRODUCT** | 16-64x for the PQ codes in memory (the stored vectors are still read to re-rank) | Search walks on the PQ codes, then re-ranks on the stored vectors | Close to NONE with the default re-rank | Very large datasets (100K+), enables zero-disk-I/O graph construction. INT8 is the better default unless the PQ codes are what you need
 |===
 
-TIP: **Use INT8 quantization for most use cases.** It provides 4x memory savings with minimal accuracy loss and significantly faster search. Only use NONE for very small datasets where maximum precision matters.
+TIP: **Use INT8 quantization for most use cases.** It provides 4x memory savings with minimal accuracy loss and usually lower search latency (see <<quantization-performance>>). Only use NONE for very small datasets where maximum precision matters.
 
 [[quantization-performance]]
-===== Why INT8 is faster
+===== Why INT8 can be faster
 
-Quantization doesn't just save memory -- it fundamentally changes how vectors are read during search.
+Besides saving memory, quantization changes how vectors are read during search.
 
 Without quantization (`NONE`), each node visited during graph traversal requires a full document lookup: read the record from disk, deserialize it, and extract the vector property. With INT8, vectors are stored in compact contiguous index pages and read directly -- no document deserialization needed.
 
-In benchmarks with 500K 384-dimensional vectors (matching the `all-MiniLM-L6-v2` embedding model), INT8 reduces search latency by **2.5x** compared to NONE:
+One benchmark with 500K 384-dimensional vectors (matching the `all-MiniLM-L6-v2` embedding model) measured the latencies below. Treat them as one setup, not a general figure. The gap between INT8 and NONE depends on the data, on whether the NONE vectors are served from memory or read from documents, and on the recall target, so measure it on your own data:
 
 [cols="1,1,1,1"]
 |===
@@ -88,7 +88,7 @@ In benchmarks with 500K 384-dimensional vectors (matching the `all-MiniLM-L6-v2`
 | **INT8** | **1.59 ms** | **1.94 ms** | Index pages (sequential I/O)
 |===
 
-The difference becomes even more significant under memory pressure. With NONE quantization, vector data is 4x larger, evicting more data from memory caches and forcing real disk I/O. INT8 keeps the working set small enough to stay in memory even with constrained resources.
+The difference can be larger under memory pressure. With NONE quantization, vector data is 4x larger, evicting more data from memory caches and forcing real disk I/O. INT8 keeps the working set small enough to stay in memory even with constrained resources.
 
 Quantization is transparent -- queries work identically regardless of quantization setting. The index automatically quantizes on insert and dequantizes on retrieval.
 
@@ -110,11 +110,11 @@ NOTE: A rebuild replaces the graph and its PQ codes a moment apart. Before v26.1
 .Memory and performance example: 100K vectors with 384 dimensions
 [cols="1,1,1"]
 |===
-| Quantization | Vector memory | Search speed
+| Quantization | Vector memory | Search latency
 
 | NONE | 156 MB | Baseline
-| INT8 | 39 MB | ~2.5x faster
-| BINARY | 5 MB | ~3x faster (lower recall)
+| INT8 | 39 MB | Lower than NONE; depends on the data and the recall target
+| BINARY | 5 MB | Lower than NONE, with lower recall; depends on the data and the rerank
 |===
 
 [[vector-encoding]]
@@ -205,7 +205,7 @@ To migrate an existing populated type to external storage in one step, use `ALTE
 | `similarity` | `COSINE` | Distance metric: COSINE, DOT_PRODUCT, or EUCLIDEAN
 | `encoding` | `FLOAT32` | Wire / document-storage encoding: FLOAT32 (`ARRAY_OF_FLOATS` property) or INT8 (`BINARY` property, byte per dim, 4x smaller payload). See <<vector-encoding,Vector Encoding>>.
 | `quantization` | `NONE` | Index-internal compression: NONE, INT8, BINARY, or PRODUCT. INT8 is recommended for most use cases.
-| `efSearch` | adaptive | Search beam width at query time. Controls recall vs speed trade-off. See <<adaptive-efsearch>>.
+| `efSearch` | sized from the graph | Search beam width at query time. Controls recall vs speed trade-off. See <<adaptive-efsearch>>.
 | `rerankOversample` | 10 | `PRODUCT` only: candidates per requested neighbor that a search re-ranks on the stored vectors, which makes the returned distances exact. `0` returns the PQ scores alone, without reading a stored vector. A query overrides it with the `rerank` option of `vector.neighbors`. See <<product-quantization-search>>. Since 26.11.1.
 | `pqTrainingLimit` | 128000 | `PRODUCT` only: vectors (a random sample) the PQ codebook is trained on, at most 128000.
 | `fusedPQ` | false | `PRODUCT` only: store the PQ codes of each node's neighbors in the graph file (jvector FusedPQ). Makes the graph file several times larger; useful only when the PQ codes cannot stay in memory. Since 26.11.1.
@@ -222,8 +222,8 @@ To migrate an existing populated type to external storage in one step, use `ALTE
 A rebuild of the graph is always a full one -- there is no incremental update, so any rebuild reindexes every
 vector currently in the index, not just the ones written since the last build. `close()` normally rebuilds and
 persists the graph before shutting down, so a later reopen finds a ready-to-search graph rather than rebuilding on
-first open. For an index of 1,000 vectors or more (the same threshold `efSearch`'s adaptive strategy and
-mutation-triggered rebuilds use), a session that closes with unpersisted writes pending instead defers that
+first open. For an index of 1,000 vectors or more (the same threshold mutation-triggered
+rebuilds use), a session that closes with unpersisted writes pending instead defers that
 rebuild rather than blocking `close()` on it: the vectors themselves are already durably persisted independently
 of the graph, so nothing is lost, but the graph's on-disk topology is left stale or absent. The deferred rebuild
 then runs the next time that index is actually searched, after any reopen. A session that closes and never
@@ -441,18 +441,16 @@ so `CHECK DATABASE FIX` (or `REBUILD INDEX <name>`) rebuilds it and lifts the re
 ====
 
 [[adaptive-efsearch]]
-==== efSearch and Adaptive Search
+==== efSearch and the Default Beam
 
-The `efSearch` parameter controls how many candidate nodes the search explores in the vector graph. Higher values find more accurate results but take longer.
+The `efSearch` parameter is the beam width of the graph search: how many candidate nodes the search keeps while it walks the vector graph. Higher values find more accurate results but take longer. The beam is never narrower than `k`.
 
-When `efSearch` is not explicitly set (either on the index or per-query), ArcadeDB uses an **adaptive two-pass strategy**:
+When `efSearch` is not set (neither on the index nor per query), ArcadeDB sizes the beam from the graph and runs the search once:
 
-1. **First pass** -- Uses a moderate beam width (`2 × k`), which is sufficient for most queries on well-clustered data.
-2. **Second pass** -- If the first pass returns insufficient results, the search automatically widens the beam to `10 × k`.
+1. For a graph under 10,000 nodes the beam is `max(k, 100)`.
+2. For a larger graph the beam is `max(2 × k, 100)`. Before 26.11.1 it was `max(2 × k, 20)`, so a 10-neighbor query on a large graph walked a beam of 20 and lost recall.
 
-For small indexes (< 10K vectors), the full default efSearch is always used since the cost is negligible.
-
-This adaptive behavior gives you fast queries on easy lookups while still maintaining recall on harder queries -- without requiring any tuning.
+There is no second pass that widens the beam when a search comes up short: a wider beam would visit the same nodes. A search that still returns fewer than `k` results is topped up by a brute-force scan instead. The default beam does not adapt to hard queries. If recall matters, set `efSearch` explicitly (see below): a larger value trades latency for recall.
 
 ===== Setting efSearch
 
@@ -483,9 +481,9 @@ CREATE INDEX ON Doc (embedding) LSM_VECTOR METADATA {
 }
 ----
 
-*Adaptive* (default) -- when neither per-query nor per-index efSearch is specified, the adaptive strategy described above is used.
+*Default beam* -- when neither per-query nor per-index efSearch is specified, the beam is sized from the graph as described above.
 
-TIP: For most workloads, the adaptive default works well. Only set efSearch explicitly if you need consistently high recall regardless of query difficulty, or if you have strict latency requirements.
+TIP: The default beam suits most workloads. If recall matters, set efSearch explicitly and raise it until the recall you measure on your own queries is enough. Set it lower only if you have strict latency requirements. On a version before 26.11.1, the default beam of a graph of 10,000 nodes or more is narrow, so set efSearch there when recall matters.
 
 ==== Filtered Search
 

src/main/asciidoc/how-to/data-modeling/vector-embeddings.adoc

@@ -38,7 +38,7 @@ CREATE INDEX ON Document (embedding) LSM_VECTOR METADATA {
 }
 ----
 
-INT8 quantization is recommended for all production workloads. It provides 2.5x faster search and 4x lower memory usage with negligible accuracy loss (see <<quantization-performance>>). Only omit quantization for very small datasets (< 10K vectors) where maximum precision matters.
+INT8 quantization is recommended for all production workloads. It provides 4x lower memory usage with negligible accuracy loss, and it reads vectors from index pages instead of documents, which usually lowers search latency; how much depends on the data and the recall target (see <<quantization-performance>>). Only omit quantization for very small datasets (< 10K vectors) where maximum precision matters.
 
 Production-ready index with additional tuning:
 
@@ -66,7 +66,7 @@ CREATE INDEX ON Document (embedding) LSM_VECTOR METADATA {
 
 ===== Quantization Trade-offs
 
-Use INT8 quantization for most use cases. It provides 4x memory savings with minimal accuracy loss and significantly faster ingestion and search:
+Use INT8 quantization for most use cases. It provides 4x memory savings with minimal accuracy loss and usually lower search latency (see <<quantization-performance>>):
 
 * **< 10K vectors**: `NONE` is fine, but `INT8` works well too
 * **10K - 1M vectors**: Use `INT8` (4x memory savings, < 2% accuracy loss) -- **recommended**
@@ -304,7 +304,7 @@ CREATE VERTEX Document SET content = 'Second document', embedding = [0.3, 0.4, .
 COMMIT
 ----
 
-TIP: For large bulk loads, increase `mutationsBeforeRebuild` to delay index rebuilds until after the load completes, then trigger a rebuild.
+TIP: For a large load through SQL or the record API, increase `mutationsBeforeRebuild` and `inactivityRebuildTimeoutMs` so the graph is not rebuilt part-way (a pause longer than `inactivityRebuildTimeoutMs`, 15 seconds by default, can trigger a rebuild whatever `mutationsBeforeRebuild` is), then run <<sql-rebuild-index,`REBUILD INDEX`>> once the load is finished, for example `+REBUILD INDEX `Document[embedding]`+`. `GraphBatch` loads need neither.
 
 NOTE: Since v26.10.1 a bulk load through `GraphBatch` - which is what `GraphImporter`, the `arcadedb-integration`
 importers and the HTTP batch endpoint use - needs none of that tuning. ArcadeDB holds the automatic graph rebuild off

src/main/asciidoc/how-to/operations/performance-tuning.adoc

@@ -126,7 +126,14 @@ difference between the two forms being several times apart in latency.
 
 ArcadeDB distributes records across buckets for parallel access. By default a new type gets one bucket (the `arcadedb.typeDefaultBuckets` setting is `1`); set a higher value, or pass the number of buckets when creating the type, to get more.
 
-For write-heavy types, add buckets to the type. `ALTER TYPE` adds and removes buckets by name (`+` adds, `-` removes), and a bucket that does not exist is created:
+For write-heavy types, choose the number of buckets when you create the type, with `BUCKETS`. For asynchronous loads, use a count equal to, or a multiple of, the async parallel level (`arcadedb.asyncWorkerThreads`, cores minus 1 by default):
+
+[source,sql]
+----
+CREATE DOCUMENT TYPE HighWriteType BUCKETS 8
+----
+
+You can also add a bucket to an existing type later, but only records created afterwards go into it, and existing records are not moved. `ALTER TYPE` adds and removes buckets by name (`+` adds, `-` removes), and a bucket that does not exist is created:
 
 [source,sql]
 ----

src/main/asciidoc/reference/extended-functions/vector.adoc

@@ -221,7 +221,7 @@ vector.neighbors(<index-spec>, <query-vector>, <k>, { <option>: <value>, ... })
 * `index-spec` - Index specification as `'TypeName[propertyName]'`
 * `query-vector` - Query vector or key to look up
 * `k` - Number of neighbors to return. Must be a non-negative integer: `0` returns no rows (like `LIMIT 0`), a negative or fractional value is rejected with an error. The same rule applies to the `k` of `vector.sparseNeighbors()`, `vector.discover()`, `vector.recommend()` and `vector.rerank()`. Since 26.11.1 (a negative `k` returned no rows before).
-* `efSearch` - (optional, positional) Search beam width. Controls the trade-off between recall and speed. Higher values improve recall but increase latency. When omitted, the index uses <<adaptive-efsearch,adaptive efSearch>>.
+* `efSearch` - (optional, positional) Search beam width. Controls the trade-off between recall and speed. Higher values improve recall but increase latency. When omitted, the index uses its <<adaptive-efsearch,default beam>>, sized from the graph.
 
 *Options map* (optional, alternative to the positional `efSearch` argument). An option of type Integer accepts a whole number only (`2` or `2.0`): a fractional value such as `2.5` is rejected with an error instead of being truncated, since 26.11.1.
 
@@ -247,7 +247,7 @@ once to return an exact answer: the same cost the `filter` subquery pays on ever
 
 [source,sql]
 ----
--- Default (adaptive efSearch)
+-- Default beam
 SELECT vector.neighbors('Document[embedding]', [0.1, 0.2, 0.3], 5)
 
 -- Explicit efSearch for higher recall (positional form, backward compatible)

src/main/asciidoc/reference/java-api/java-ref-database-async.adoc

@@ -346,7 +346,7 @@ this, so a later conflict on the same worker never re-applies those writes and n
 [[asynccreate]]
 ====== createRecord(record, newRecordCallback [,errorCallback])
 
-Create a record in async way. Once the record is created in the database, the callback will be executed.
+Create a record in async way. Once the record is applied to the worker's current batch, the callback will be executed. This is not yet durable: if a later operation in the same batch fails, the whole batch rolls back, and the record with it, even though the callback already fired. For a signal that fires only once the batch has committed, register <<asynconok,`onOk()`>> on the executor or call `waitCompletion()`.
 This method returns immediately.
 The result can be managed in the NewRecordCallback callback and errors in ErrorCallback callback.
 
@@ -376,7 +376,7 @@ database.async().createRecord(vertex,
 [[asyncupdate]]
 ====== updateRecord(record, updateRecordCallback [,errorCallback])
 
-Updates a record in async way. Once the record is updated in the database, the callback will be executed.
+Updates a record in async way. Once the update is applied to the worker's current batch, the callback will be executed. This is not yet durable: if a later operation in the same batch fails, the whole batch rolls back, and the update with it, even though the callback already fired. For a signal that fires only once the batch has committed, register <<asynconok,`onOk()`>> on the executor or call `waitCompletion()`.
 This method returns immediately.
 The result can be managed in the UpdatedRecordCallback callback and errors in ErrorCallback callback.
 
@@ -405,7 +405,7 @@ database.async().updateRecord(vertex,
 [[asyncdelete]]
 ====== deleteRecord(record, deleteRecordCallback [,errorCallback])
 
-Deletes a record in async way. Once the record is deleted from the database, the callback will be executed.
+Deletes a record in async way. Once the delete is applied to the worker's current batch, the callback will be executed. This is not yet durable: if a later operation in the same batch fails, the whole batch rolls back, and the delete with it, even though the callback already fired. For a signal that fires only once the batch has committed, register <<asynconok,`onOk()`>> on the executor or call `waitCompletion()`.
 This method returns immediately.
 The result can be managed in the DeletedRecordCallback callback and errors in ErrorCallback callback.
 
@@ -428,3 +428,31 @@ Example on inserting a vertex asynchronously.
 database.async().deleteRecord(vertex,
                               v -> { System.out.println("Record " + v.toJSON() + " updated") });
 ```
+
+'''
+
+[[asynconok]]
+====== onOk( okCallback )
+
+Registers a global callback that fires every time a worker's batch transaction has committed: at each `setCommitEvery()` boundary, when a durability setting (<<asyncdurability,`setTransactionUseWAL()` and `setTransactionSync()`>>) changes mid-batch, and when `waitCompletion()` commits the pending batches. This is the durability signal for `createRecord()`, `updateRecord()` and `deleteRecord()`: their own callbacks fire as soon as the write is applied to the worker's current batch, which a later failure in the same batch can still roll back.
+
+Syntax:
+
+```java
+void onOk(final OkCallback callback)
+```
+
+Example:
+
+```java
+final AtomicLong commits = new AtomicLong();
+database.async().onOk(() -> commits.incrementAndGet());
+
+database.async().createRecord(vertex,
+                              v -> { System.out.println("Record " + v.toJSON() + " applied, not yet durable"); });
+database.async().waitCompletion();
+```
+
+The callback is not tied to one record: it tells you a batch has committed, not which records were in it. Each worker commits on its own, so the callback runs on the committing worker's thread, possibly from several threads at once. It must be thread-safe and cheap, because a slow callback slows that worker down. A worker also calls it on shutdown even when it had nothing left to commit, so a call at shutdown is not proof that a particular write became durable at that moment.
+
+NOTE: "Committed" here means durable in the write-ahead log and safe to replay on the next open. With the default `setTransactionSync(NO)` there is no `fsync` at that point, so the write can still be lost to a power loss or an operating system crash (not to a JVM crash or restart). Set `setTransactionSync(YES_NOMETADATA)` or `YES_FULL` if you need that guarantee; the timing of this callback does not change with it.