From 3ca6346d7bc464bc926e430744261f697df7824f Mon Sep 17 00:00:00 2001 From: robfrank Date: Wed, 28 Oct 2026 18:00:35 +0100 Subject: [PATCH 1/2] docs(#509, #501, #494, #498): Prometheus paths, native image CPU baseline, bounded hybrid search - #509: every Prometheus-compatible endpoint is under /api/v1/ts/{database}/prom/... (timeseries concepts page, time-series tutorial, reference chapter, upgrade note) - #501: native image CPU requirements (x86-64-v3 / armv8.1-a), the startup error, how to check a host, and the workarounds (JVM image, -Dnative.march=compatibility) - #494: reference entries for vector.fuse (options incl. limit) and vector.rerank; a section on bounding the full-text leg and the fusion; bounded examples - #498: already covered by the Query Languages table (8f6e1398) - _typos.toml: whitelist the identifier ANDed, which the hook rewrote to "ANDead" Co-Authored-By: Claude Opus 5.5 --- _typos.toml | 3 + src/main/asciidoc/concepts/timeseries.adoc | 30 ++++----- src/main/asciidoc/concepts/vector-search.adoc | 30 ++++++++- .../how-to/data-modeling/full-text-index.adoc | 4 +- .../how-to/operations/install-docker.adoc | 1 + .../how-to/operations/native-image.adoc | 43 ++++++++++++ .../asciidoc/how-to/operations/upgrade.adoc | 2 +- .../reference/extended-functions/vector.adoc | 65 +++++++++++++++++++ .../reference/time-series/chapter.adoc | 2 +- .../tutorials/time-series-tutorial.adoc | 4 +- 10 files changed, 162 insertions(+), 22 deletions(-) diff --git a/_typos.toml b/_typos.toml index 3830af4ce..ac77cc049 100644 --- a/_typos.toml +++ b/_typos.toml @@ -8,3 +8,6 @@ sesnor = "sesnor" Ser = "Ser" nin = "nin" EXAT = "EXAT" + +[default.extend-identifiers] +ANDed = "ANDed" diff --git a/src/main/asciidoc/concepts/timeseries.adoc b/src/main/asciidoc/concepts/timeseries.adoc index c0ba8ba4c..2083b184d 100644 --- a/src/main/asciidoc/concepts/timeseries.adoc +++ b/src/main/asciidoc/concepts/timeseries.adoc @@ -462,7 +462,7 @@ Configure Prometheus to write to ArcadeDB: ---- # prometheus.yml remote_write: - - url: "http://localhost:2480/ts/mydb/prom/write" + - url: "http://localhost:2480/api/v1/ts/mydb/prom/write" basic_auth: username: root password: password @@ -1085,7 +1085,7 @@ The PromQL endpoints follow the Prometheus HTTP API format and return standard ` [source] ---- -GET /ts/{database}/prom/api/v1/query?query=avg(cpu_usage{host="srv1"})&time=1700000000 +GET /api/v1/ts/{database}/prom/api/v1/query?query=avg(cpu_usage{host="srv1"})&time=1700000000 ---- The `time` parameter is Unix seconds (float). Defaults to current time if omitted. @@ -1094,7 +1094,7 @@ The `time` parameter is Unix seconds (float). Defaults to current time if omitte [source] ---- -GET /ts/{database}/prom/api/v1/query_range?query=rate(http_requests_total[5m])&start=1700000000&end=1700003600&step=60 +GET /api/v1/ts/{database}/prom/api/v1/query_range?query=rate(http_requests_total[5m])&start=1700000000&end=1700003600&step=60 ---- All timestamps in Unix seconds. The `step` parameter accepts a duration string (`60s`, `1m`) or seconds as a float. @@ -1130,10 +1130,10 @@ series returned, not to the number of samples scanned to compute it. [source] ---- -GET /ts/{database}/prom/api/v1/labels -GET /ts/{database}/prom/api/v1/label/{name}/values -GET /ts/{database}/prom/api/v1/label/{name}/values?start=1700000000&end=1700003600 -GET /ts/{database}/prom/api/v1/series?match[]=cpu_usage{host=~"srv.*"} +GET /api/v1/ts/{database}/prom/api/v1/labels +GET /api/v1/ts/{database}/prom/api/v1/label/{name}/values +GET /api/v1/ts/{database}/prom/api/v1/label/{name}/values?start=1700000000&end=1700003600 +GET /api/v1/ts/{database}/prom/api/v1/series?match[]=cpu_usage{host=~"srv.*"} ---- NOTE: These endpoints follow the Prometheus data model, where a label with an empty value counts as absent. A tag @@ -1237,7 +1237,7 @@ Configure Prometheus to read from ArcadeDB for long-term storage: ---- # prometheus.yml remote_read: - - url: "http://localhost:2480/ts/mydb/prom/read" + - url: "http://localhost:2480/api/v1/ts/mydb/prom/read" basic_auth: username: root password: password @@ -1445,31 +1445,31 @@ The explorer provides four tabs: | Returns the most recent data point, with optional tag filter. Repeat `tag` to narrow across several tag columns; every occurrence must match. An occurrence that is malformed, or names no `TAG` column, is refused with 400. A `DATE`/`DATETIME` tag value is read in the database's time zone. | `GET` -| `/ts/{database}/prom/api/v1/query?query=&time=` +| `/api/v1/ts/{database}/prom/api/v1/query?query=&time=` | PromQL instant query. | `GET` -| `/ts/{database}/prom/api/v1/query_range?query=&start=&end=&step=` +| `/api/v1/ts/{database}/prom/api/v1/query_range?query=&start=&end=&step=` | PromQL range query. | `GET` -| `/ts/{database}/prom/api/v1/labels` +| `/api/v1/ts/{database}/prom/api/v1/labels` | List all label names (metric names and tag columns). | `GET` -| `/ts/{database}/prom/api/v1/label/{name}/values` +| `/api/v1/ts/{database}/prom/api/v1/label/{name}/values` | List distinct values for a label. | `GET` -| `/ts/{database}/prom/api/v1/series?match[]=` +| `/api/v1/ts/{database}/prom/api/v1/series?match[]=` | Find series matching PromQL selector(s). | `POST` -| `/ts/{database}/prom/write` +| `/api/v1/ts/{database}/prom/write` | Prometheus remote_write 1.0 endpoint (snappy-compressed Protobuf). Answers 415 to a remote write 2.0 `Content-Type`. | `POST` -| `/ts/{database}/prom/read` +| `/api/v1/ts/{database}/prom/read` | Prometheus remote_read endpoint (snappy-compressed Protobuf). | `GET` diff --git a/src/main/asciidoc/concepts/vector-search.adoc b/src/main/asciidoc/concepts/vector-search.adoc index a5c778dc0..384a07f3d 100644 --- a/src/main/asciidoc/concepts/vector-search.adoc +++ b/src/main/asciidoc/concepts/vector-search.adoc @@ -845,11 +845,39 @@ SELECT expand(`vector.fuse`( SELECT expand(`vector.fuse`( `vector.neighbors`('Doc[dense]', :denseVec, 100), `vector.sparseNeighbors`('Doc[tokens,weights]', :qIdx, :qVal, 100), - (SELECT @rid, $score FROM Doc WHERE SEARCH_INDEX('Doc[content]', :keywords) = true), + (SELECT @rid, $score FROM Doc WHERE SEARCH_INDEX('Doc[content]', :keywords) = true LIMIT 100), { fusion: 'RRF', k: 60 } )) LIMIT 10 ---- +[[hybrid-search-bounded]] +===== Bounding the cost of a full-text leg + +`vector.fuse` fuses every row its sources produce and, without a `limit` option, loads every fused candidate. +A dense or sparse leg is bounded by its `k`, but a full-text leg without a `LIMIT` yields one row per match: on a common keyword over a large corpus that is thousands of rows, and the query gets slower with the match count instead of with `k`. +The same holds when the fused rows feed `vector.rerank`. + +Bound the full-text leg with a `LIMIT` and the fusion with `limit`: + +[source,sql] +---- +SELECT expand(`vector.rerank`( + `vector.fuse`( + `vector.neighbors`('Doc[dense]', :denseVec, :fetch), + (SELECT @rid, $score FROM Doc WHERE SEARCH_INDEX('Doc[content]', :keywords) = true LIMIT :fetch), + { fusion: 'RRF', limit: :candidates } + ), + :denseVec, 'dense', :k +)) +---- + +`:fetch` is the number of rows each leg contributes, `:candidates` the number of fused rows handed to `vector.rerank`, and `:k` the number of results (for example 100, 50 and 10). +A `SEARCH_INDEX` subquery without an `ORDER BY` returns its matches by descending score, so its `LIMIT` keeps the best-scoring ones and the bounded form loses no relevance over the unbounded one. + +Since 26.11.1, `SEARCH_INDEX(...)` can also be passed directly as a source, as in `vector.fuse(vector.neighbors(...), SEARCH_INDEX('Doc[content]', :keywords), { fusion: 'LINEAR' })`. +It carries the score of each match, so `DBSF` and `LINEAR` accept it, but it cannot take a `LIMIT`: every match is fused, so prefer the subquery when the match count can be large. +In the same release `vector.rerank` stopped loading again the records that `vector.fuse` and `vector.neighbors` rows already carry, and keeps only the best `k` while scanning. + ===== Outer composition The fused output is itself a result set, so it composes with outer `WHERE`, `ORDER BY`, `LIMIT`, joins, and projections: diff --git a/src/main/asciidoc/how-to/data-modeling/full-text-index.adoc b/src/main/asciidoc/how-to/data-modeling/full-text-index.adoc index 6eb4ed0e6..bd48995a4 100644 --- a/src/main/asciidoc/how-to/data-modeling/full-text-index.adoc +++ b/src/main/asciidoc/how-to/data-modeling/full-text-index.adoc @@ -759,7 +759,7 @@ FROM ( SELECT expand(`vector.fuse`( `vector.neighbors`('Case[dense]', :denseVec, 100), `vector.sparseNeighbors`('Case[tokens,weights]', :qIdx, :qVal, 100), - (SELECT @rid, $score FROM Case WHERE SEARCH_INDEX('Case[content]', :keywords) = true), + (SELECT @rid, $score FROM Case WHERE SEARCH_INDEX('Case[content]', :keywords) = true LIMIT 100), { fusion: 'RRF', k: 60 } )) ) @@ -777,7 +777,7 @@ FROM ( FROM ( SELECT expand(`vector.fuse`( `vector.neighbors`('Case[dense]', :denseVec, 100), - (SELECT @rid, $score FROM Case WHERE SEARCH_INDEX('Case[content]', :keywords) = true), + (SELECT @rid, $score FROM Case WHERE SEARCH_INDEX('Case[content]', :keywords) = true LIMIT 100), { fusion: 'RRF' } )) LIMIT 10 ) diff --git a/src/main/asciidoc/how-to/operations/install-docker.adoc b/src/main/asciidoc/how-to/operations/install-docker.adoc index 61b861986..5d551703f 100644 --- a/src/main/asciidoc/how-to/operations/install-docker.adoc +++ b/src/main/asciidoc/how-to/operations/install-docker.adoc @@ -30,5 +30,6 @@ docker pull ghcr.io/arcadedata/arcadedb:{revnumber}-java17 An experimental GraalVM native image of the server is also published, under the `-native` tags (`-native` for a release, `latest-native` for the nightly build of `main`). It starts faster and needs less memory, but leaves out Gremlin and has other limitations: see <>. +It also needs a recent CPU (`x86-64-v3` on amd64, `armv8.1-a` on arm64) and exits at startup on an older one: see <>. To run the container, configure it and persist its data, see <>. diff --git a/src/main/asciidoc/how-to/operations/native-image.adoc b/src/main/asciidoc/how-to/operations/native-image.adoc index f38e84b7a..ecd9d7b7c 100644 --- a/src/main/asciidoc/how-to/operations/native-image.adoc +++ b/src/main/asciidoc/how-to/operations/native-image.adoc @@ -37,6 +37,35 @@ The two architectures use different base images: the `linux/amd64` binary is ful The archive contains a single executable with the same name as the archive, minus the extension (`.exe` on Windows). Studio and the configuration templates are embedded in it. +[[native-image-cpu-requirements]] +===== CPU Requirements + +A native binary is compiled for one CPU baseline and starts only on a CPU that has every feature of it. +The published images and release archives use the GraalVM defaults: + +[%header,cols="2,1,3,3"] +|=== +| Architecture | Baseline | Required CPU features | Excludes, for example +| amd64 (Linux, Windows) | `x86-64-v3` | AVX, AVX2, BMI1, BMI2, FMA, F16C, LZCNT, MOVBE | Intel before Haswell (2013), AMD before Excavator, Atom, Celeron and Pentium parts without AVX, and VMs whose CPU model hides AVX2 (the QEMU default `qemu64`, the Proxmox defaults `x86-64-v2-AES` and `kvm64`) +| arm64 (Linux, macOS Apple Silicon) | `armv8.1-a` | ASIMD, CRC32, LSE atomics | ARMv8.0 cores such as the Cortex-A53 and A72 (Raspberry Pi 3 and 4) +|=== + +On an older CPU the binary exits as soon as it starts, with: + +---- +The current machine does not support all of the following CPU features that are required by the image: [...]. +Please rebuild the executable with an appropriate setting of the -march option. +---- + +To check a Linux amd64 host, run `/lib64/ld-linux-x86-64.so.2 --help`: on glibc 2.33 and later it lists `x86-64-v3 (supported, searched)` when the CPU qualifies. +Otherwise look for the `avx2`, `bmi2` and `fma` flags in `/proc/cpuinfo`. +In a VM, expose the host CPU (`-cpu host` in QEMU, CPU type `host` in Proxmox) or a model that supports `x86-64-v3`. + +If the CPU does not qualify, either: + +* run the JVM image (`arcadedata/arcadedb:`) or the JVM distribution, whose JIT compiles for the CPU it runs on, or +* build a native binary for a lower baseline with `-Dnative.march=compatibility`, which runs on any x86-64 or ARMv8 CPU (see <>). + ===== Running the Native Server Run the executable directly, passing settings as `-D` arguments exactly as you would pass them to the JVM: @@ -112,6 +141,7 @@ Do not expose the native image's JavaScript surface to untrusted users in a shar The CA bundle copied into the Docker images serves only code paths that read the operating system's bundle explicitly; it does not change the trust anchors baked into the binary. * *Size.* The executable is large, around 907 MiB on macOS ARM64, because every classpath resource, including the Studio assets, is embedded in it. +[[native-image-build]] ===== Building the Native Image Yourself Most users should download a published binary or image. @@ -125,6 +155,19 @@ Two scripts in the ArcadeDB source tree build one. ./native/scripts/build-native-docker.sh # Linux Docker image for the current architecture ---- +By default the build targets the baseline of the published binaries (see <>). +To build for an older CPU, lower it: `compatibility` runs on any x86-64 or ARMv8 CPU, and `x86-64-v2` (amd64 only) needs SSE4.2 and POPCNT but no AVX. +A lower baseline gives up the AVX2, FMA and BMI2 (or LSE) code of the default build. + +[source,bash] +---- +./native/scripts/build-native.sh -- -Dnative.march=compatibility +./native/scripts/build-native-docker.sh --march compatibility +mvn -Pnative -pl native -am -DskipTests -Dnative.march=compatibility package +---- + +Do not distribute a binary built with `-march=native`: it requires every feature of the CPU that built it. + Native Image cannot cross-compile: each build produces a binary for the platform it runs on, and `build-native-docker.sh` builds the Linux binary inside a container so it also works on macOS. The build is memory hungry. `build-native-docker.sh` requires a Docker VM with about 16 GiB of memory, and for a local `build-native.sh` you can set the builder heap with `NATIVE_IMAGE_OPTIONS="-J-Xmx24g"` if the default runs out of memory. See `docs/native-image.md` in the ArcadeDB repository for the full build reference. diff --git a/src/main/asciidoc/how-to/operations/upgrade.adoc b/src/main/asciidoc/how-to/operations/upgrade.adoc index 11b265970..754bbcda9 100644 --- a/src/main/asciidoc/how-to/operations/upgrade.adoc +++ b/src/main/asciidoc/how-to/operations/upgrade.adoc @@ -579,7 +579,7 @@ The PromQL engine now answers HTTP `400` `bad_data` on `/prom/api/v1/query` and ====== 26.10.2 - Time-series filters, Prometheus endpoints and `ts.movingAvg` change results * The PromQL label matchers `=`, `!=`, `=~` and `!~` read a typed tag the same way on `/query`, remote read and `/series`, so `=` and `!=` select complementary sets, and `=~"007"` selects a `LONG` tag holding 7 (issues #9833, #9880). -* The JSON tag filter of `POST /ts/query`, `GET /ts/latest`, the Grafana query and the gRPC `TimeSeriesQuery` and `TimeSeriesLatest` calls reads a `DATE` or `DATETIME` value in the database time zone instead of the JVM one (issues #9879, #9929). Embedded API: `TimeSeriesGateway.andTag()` and `buildTagFilter()` now take the `Database` as an argument. +* The JSON tag filter of `POST /api/v1/ts/{database}/query`, `GET /api/v1/ts/{database}/latest`, the Grafana query and the gRPC `TimeSeriesQuery` and `TimeSeriesLatest` calls reads a `DATE` or `DATETIME` value in the database time zone instead of the JVM one (issues #9879, #9929). Embedded API: `TimeSeriesGateway.andTag()` and `buildTagFilter()` now take the `Database` as an argument. * Prometheus remote write and read answer `415` for a `proto=` content type they do not implement, so a remote-write 2.0 sender falls back to 1.0 instead of losing the batch, and `400` instead of `204` for a non-empty body with no remote-write 1.0 field (issues #9624, #9755). * `ts.movingAvg()` keeps one element per row: a null sample keeps its slot instead of being dropped, which shifted every later average into an earlier row (issue #9958). diff --git a/src/main/asciidoc/reference/extended-functions/vector.adoc b/src/main/asciidoc/reference/extended-functions/vector.adoc index 5e9390c88..d094870dc 100644 --- a/src/main/asciidoc/reference/extended-functions/vector.adoc +++ b/src/main/asciidoc/reference/extended-functions/vector.adoc @@ -513,6 +513,71 @@ SELECT vector.dequantizebinary(vector.quantizebinary([0.1, 0.5, 0.9])) ===== Scoring and Fusion +[[vector-fuse-ext]] +====== vector.fuse() + +Fuses the ranked rows of two or more sources into one ranking, server-side. See <> for the strategies and worked examples. + +*Syntax:* `vector.fuse(, , ... [, {