Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 3 additions & 0 deletions _typos.toml
Original file line number Diff line number Diff line change
Expand Up @@ -8,3 +8,6 @@ sesnor = "sesnor"
Ser = "Ser"
nin = "nin"
EXAT = "EXAT"

[default.extend-identifiers]
ANDed = "ANDed"
30 changes: 15 additions & 15 deletions src/main/asciidoc/concepts/timeseries.adoc
Original file line number Diff line number Diff line change
Expand Up @@ -462,7 +462,7 @@ Configure Prometheus to write to ArcadeDB:
----
# prometheus.yml
remote_write:
- url: "http://localhost:2480/ts/mydb/prom/write"
- url: "http://localhost:2480/api/v1/ts/mydb/prom/write"
basic_auth:
username: root
password: password
Expand Down Expand Up @@ -1085,7 +1085,7 @@ The PromQL endpoints follow the Prometheus HTTP API format and return standard `

[source]
----
GET /ts/{database}/prom/api/v1/query?query=avg(cpu_usage{host="srv1"})&time=1700000000
GET /api/v1/ts/{database}/prom/api/v1/query?query=avg(cpu_usage{host="srv1"})&time=1700000000
----

The `time` parameter is Unix seconds (float). Defaults to current time if omitted.
Expand All @@ -1094,7 +1094,7 @@ The `time` parameter is Unix seconds (float). Defaults to current time if omitte

[source]
----
GET /ts/{database}/prom/api/v1/query_range?query=rate(http_requests_total[5m])&start=1700000000&end=1700003600&step=60
GET /api/v1/ts/{database}/prom/api/v1/query_range?query=rate(http_requests_total[5m])&start=1700000000&end=1700003600&step=60
----

All timestamps in Unix seconds. The `step` parameter accepts a duration string (`60s`, `1m`) or seconds as a float.
Expand Down Expand Up @@ -1130,10 +1130,10 @@ series returned, not to the number of samples scanned to compute it.

[source]
----
GET /ts/{database}/prom/api/v1/labels
GET /ts/{database}/prom/api/v1/label/{name}/values
GET /ts/{database}/prom/api/v1/label/{name}/values?start=1700000000&end=1700003600
GET /ts/{database}/prom/api/v1/series?match[]=cpu_usage{host=~"srv.*"}
GET /api/v1/ts/{database}/prom/api/v1/labels
GET /api/v1/ts/{database}/prom/api/v1/label/{name}/values
GET /api/v1/ts/{database}/prom/api/v1/label/{name}/values?start=1700000000&end=1700003600
GET /api/v1/ts/{database}/prom/api/v1/series?match[]=cpu_usage{host=~"srv.*"}
----

NOTE: These endpoints follow the Prometheus data model, where a label with an empty value counts as absent. A tag
Expand Down Expand Up @@ -1237,7 +1237,7 @@ Configure Prometheus to read from ArcadeDB for long-term storage:
----
# prometheus.yml
remote_read:
- url: "http://localhost:2480/ts/mydb/prom/read"
- url: "http://localhost:2480/api/v1/ts/mydb/prom/read"
basic_auth:
username: root
password: password
Expand Down Expand Up @@ -1445,31 +1445,31 @@ The explorer provides four tabs:
| Returns the most recent data point, with optional tag filter. Repeat `tag` to narrow across several tag columns; every occurrence must match. An occurrence that is malformed, or names no `TAG` column, is refused with 400. A `DATE`/`DATETIME` tag value is read in the database's time zone.

| `GET`
| `/ts/{database}/prom/api/v1/query?query=<promql>&time=<unix_seconds>`
| `/api/v1/ts/{database}/prom/api/v1/query?query=<promql>&time=<unix_seconds>`
| PromQL instant query.

| `GET`
| `/ts/{database}/prom/api/v1/query_range?query=<promql>&start=<s>&end=<s>&step=<s>`
| `/api/v1/ts/{database}/prom/api/v1/query_range?query=<promql>&start=<s>&end=<s>&step=<s>`
| PromQL range query.

| `GET`
| `/ts/{database}/prom/api/v1/labels`
| `/api/v1/ts/{database}/prom/api/v1/labels`
| List all label names (metric names and tag columns).

| `GET`
| `/ts/{database}/prom/api/v1/label/{name}/values`
| `/api/v1/ts/{database}/prom/api/v1/label/{name}/values`
| List distinct values for a label.

| `GET`
| `/ts/{database}/prom/api/v1/series?match[]=<selector>`
| `/api/v1/ts/{database}/prom/api/v1/series?match[]=<selector>`
| Find series matching PromQL selector(s).

| `POST`
| `/ts/{database}/prom/write`
| `/api/v1/ts/{database}/prom/write`
| Prometheus remote_write 1.0 endpoint (snappy-compressed Protobuf). Answers 415 to a remote write 2.0 `Content-Type`.

| `POST`
| `/ts/{database}/prom/read`
| `/api/v1/ts/{database}/prom/read`
| Prometheus remote_read endpoint (snappy-compressed Protobuf).

| `GET`
Expand Down
30 changes: 29 additions & 1 deletion src/main/asciidoc/concepts/vector-search.adoc
Original file line number Diff line number Diff line change
Expand Up @@ -845,11 +845,39 @@ SELECT expand(`vector.fuse`(
SELECT expand(`vector.fuse`(
`vector.neighbors`('Doc[dense]', :denseVec, 100),
`vector.sparseNeighbors`('Doc[tokens,weights]', :qIdx, :qVal, 100),
(SELECT @rid, $score FROM Doc WHERE SEARCH_INDEX('Doc[content]', :keywords) = true),
(SELECT @rid, $score FROM Doc WHERE SEARCH_INDEX('Doc[content]', :keywords) = true LIMIT 100),
{ fusion: 'RRF', k: 60 }
)) LIMIT 10
----

[[hybrid-search-bounded]]
===== Bounding the cost of a full-text leg

`vector.fuse` fuses every row its sources produce and, without a `limit` option, loads every fused candidate.
A dense or sparse leg is bounded by its `k`. A full-text leg without a `LIMIT` is not: it yields one row per match, a common keyword in a large corpus can match thousands of rows, and the query then slows down with the match count instead of with `k`.
The same holds when the fused rows feed `vector.rerank`.

Bound the full-text leg with a `LIMIT` and the fusion with `limit`:

[source,sql]
----
SELECT expand(`vector.rerank`(
`vector.fuse`(
`vector.neighbors`('Doc[dense]', :denseVec, :fetch),
(SELECT @rid, $score FROM Doc WHERE SEARCH_INDEX('Doc[content]', :keywords) = true LIMIT :fetch),
{ fusion: 'RRF', limit: :candidates }
),
:denseVec, 'dense', :k
))
----

`:fetch` is the number of rows each leg contributes, `:candidates` the number of fused rows handed to `vector.rerank`, and `:k` the number of results (for example 100, 50 and 10).
A `SEARCH_INDEX` subquery without an `ORDER BY` returns its matches by descending score, so its `LIMIT` keeps the best-scoring ones. Bounding it costs no relevance.

Since 26.11.1, `SEARCH_INDEX(...)` can also be passed directly as a source, as in `vector.fuse(vector.neighbors(...), SEARCH_INDEX('Doc[content]', :keywords), { fusion: 'LINEAR' })`.
It carries the score of each match, so `DBSF` and `LINEAR` accept it. It cannot take a `LIMIT`, though, and every match gets fused: prefer the subquery when the match count can be large.
In the same release `vector.rerank` stopped reloading records that `vector.fuse` and `vector.neighbors` rows already carry, and now keeps only the best `k` while scanning.

===== Outer composition

The fused output is itself a result set, so it composes with outer `WHERE`, `ORDER BY`, `LIMIT`, joins, and projections:
Expand Down
4 changes: 2 additions & 2 deletions src/main/asciidoc/how-to/data-modeling/full-text-index.adoc
Original file line number Diff line number Diff line change
Expand Up @@ -759,7 +759,7 @@ FROM (
SELECT expand(`vector.fuse`(
`vector.neighbors`('Case[dense]', :denseVec, 100),
`vector.sparseNeighbors`('Case[tokens,weights]', :qIdx, :qVal, 100),
(SELECT @rid, $score FROM Case WHERE SEARCH_INDEX('Case[content]', :keywords) = true),
(SELECT @rid, $score FROM Case WHERE SEARCH_INDEX('Case[content]', :keywords) = true LIMIT 100),
{ fusion: 'RRF', k: 60 }
))
)
Expand All @@ -777,7 +777,7 @@ FROM (
FROM (
SELECT expand(`vector.fuse`(
`vector.neighbors`('Case[dense]', :denseVec, 100),
(SELECT @rid, $score FROM Case WHERE SEARCH_INDEX('Case[content]', :keywords) = true),
(SELECT @rid, $score FROM Case WHERE SEARCH_INDEX('Case[content]', :keywords) = true LIMIT 100),
{ fusion: 'RRF' }
)) LIMIT 10
)
Expand Down
1 change: 1 addition & 0 deletions src/main/asciidoc/how-to/operations/install-docker.adoc
Original file line number Diff line number Diff line change
Expand Up @@ -30,5 +30,6 @@ docker pull ghcr.io/arcadedata/arcadedb:{revnumber}-java17

An experimental GraalVM native image of the server is also published, under the `-native` tags (`<version>-native` for a release, `latest-native` for the nightly build of `main`).
It starts faster and needs less memory, but leaves out Gremlin and has other limitations: see <<native-image>>.
It also needs a recent CPU (`x86-64-v3` on amd64, `armv8.1-a` on arm64) and exits at startup on an older one: see <<native-image-cpu-requirements>>.

To run the container, configure it and persist its data, see <<docker>>.
43 changes: 43 additions & 0 deletions src/main/asciidoc/how-to/operations/native-image.adoc
Original file line number Diff line number Diff line change
Expand Up @@ -37,6 +37,35 @@ The two architectures use different base images: the `linux/amd64` binary is ful
The archive contains a single executable with the same name as the archive, minus the extension (`.exe` on Windows).
Studio and the configuration templates are embedded in it.

[[native-image-cpu-requirements]]
===== CPU Requirements

A native binary is compiled for one CPU baseline and starts only on a CPU that has every feature of it.
The published images and release archives use the GraalVM defaults:

[%header,cols="2,1,3,3"]
|===
| Architecture | Baseline | Required CPU features | Excludes, for example
| amd64 (Linux, Windows) | `x86-64-v3` | AVX, AVX2, BMI1, BMI2, FMA, F16C, LZCNT, MOVBE | Intel before Haswell (2013), AMD before Excavator, Atom, Celeron and Pentium parts without AVX, and VMs whose CPU model hides AVX2 (the QEMU default `qemu64`, the Proxmox defaults `x86-64-v2-AES` and `kvm64`)
| arm64 (Linux, macOS Apple Silicon) | `armv8.1-a` | ASIMD, CRC32, LSE atomics | ARMv8.0 cores such as the Cortex-A53 and A72 (Raspberry Pi 3 and 4)
|===

On an older CPU the binary exits as soon as it starts, with:

----
The current machine does not support all of the following CPU features that are required by the image: [...].
Please rebuild the executable with an appropriate setting of the -march option.
----

To check a Linux amd64 host, run `/lib64/ld-linux-x86-64.so.2 --help`: on glibc 2.33 and later it lists `x86-64-v3 (supported, searched)` when the CPU qualifies.
Otherwise look for the `avx2`, `bmi2` and `fma` flags in `/proc/cpuinfo`.
In a VM, expose the host CPU (`-cpu host` in QEMU, CPU type `host` in Proxmox) or a model that supports `x86-64-v3`.

If the CPU does not qualify, either:

* run the JVM image (`arcadedata/arcadedb:<version>`) or the JVM distribution, whose JIT compiles for the CPU it runs on, or
* build a native binary for a lower baseline with `-Dnative.march=compatibility`, which runs on any x86-64 or ARMv8 CPU (see <<native-image-build>>).

===== Running the Native Server

Run the executable directly, passing settings as `-D` arguments exactly as you would pass them to the JVM:
Expand Down Expand Up @@ -112,6 +141,7 @@ Do not expose the native image's JavaScript surface to untrusted users in a shar
The CA bundle copied into the Docker images serves only code paths that read the operating system's bundle explicitly; it does not change the trust anchors baked into the binary.
* *Size.* The executable is large, around 907 MiB on macOS ARM64, because every classpath resource, including the Studio assets, is embedded in it.

[[native-image-build]]
===== Building the Native Image Yourself

Most users should download a published binary or image.
Expand All @@ -125,6 +155,19 @@ Two scripts in the ArcadeDB source tree build one.
./native/scripts/build-native-docker.sh # Linux Docker image for the current architecture
----

By default the build targets the baseline of the published binaries (see <<native-image-cpu-requirements>>).
To build for an older CPU, lower it: `compatibility` runs on any x86-64 or ARMv8 CPU, and `x86-64-v2` (amd64 only) needs SSE4.2 and POPCNT but no AVX.
A lower baseline gives up the AVX2, FMA and BMI2 (or LSE) code of the default build.

[source,bash]
----
./native/scripts/build-native.sh -- -Dnative.march=compatibility
./native/scripts/build-native-docker.sh --march compatibility
mvn -Pnative -pl native -am -DskipTests -Dnative.march=compatibility package
----

Do not distribute a binary built with `-march=native`: it requires every feature of the CPU that built it.

Native Image cannot cross-compile: each build produces a binary for the platform it runs on, and `build-native-docker.sh` builds the Linux binary inside a container so it also works on macOS.
The build is memory hungry. `build-native-docker.sh` requires a Docker VM with about 16 GiB of memory, and for a local `build-native.sh` you can set the builder heap with `NATIVE_IMAGE_OPTIONS="-J-Xmx24g"` if the default runs out of memory.
See `docs/native-image.md` in the ArcadeDB repository for the full build reference.
2 changes: 1 addition & 1 deletion src/main/asciidoc/how-to/operations/upgrade.adoc
Original file line number Diff line number Diff line change
Expand Up @@ -579,7 +579,7 @@ The PromQL engine now answers HTTP `400` `bad_data` on `/prom/api/v1/query` and
====== 26.10.2 - Time-series filters, Prometheus endpoints and `ts.movingAvg` change results

* The PromQL label matchers `=`, `!=`, `=~` and `!~` read a typed tag the same way on `/query`, remote read and `/series`, so `=` and `!=` select complementary sets, and `=~"007"` selects a `LONG` tag holding 7 (issues #9833, #9880).
* The JSON tag filter of `POST /ts/query`, `GET /ts/latest`, the Grafana query and the gRPC `TimeSeriesQuery` and `TimeSeriesLatest` calls reads a `DATE` or `DATETIME` value in the database time zone instead of the JVM one (issues #9879, #9929). Embedded API: `TimeSeriesGateway.andTag()` and `buildTagFilter()` now take the `Database` as an argument.
* The JSON tag filter of `POST /api/v1/ts/{database}/query`, `GET /api/v1/ts/{database}/latest`, the Grafana query and the gRPC `TimeSeriesQuery` and `TimeSeriesLatest` calls reads a `DATE` or `DATETIME` value in the database time zone instead of the JVM one (issues #9879, #9929). Embedded API: `TimeSeriesGateway.andTag()` and `buildTagFilter()` now take the `Database` as an argument.
* Prometheus remote write and read answer `415` for a `proto=` content type they do not implement, so a remote-write 2.0 sender falls back to 1.0 instead of losing the batch, and `400` instead of `204` for a non-empty body with no remote-write 1.0 field (issues #9624, #9755).
* `ts.movingAvg()` keeps one element per row: a null sample keeps its slot instead of being dropped, which shifted every later average into an earlier row (issue #9958).

Expand Down
65 changes: 65 additions & 0 deletions src/main/asciidoc/reference/extended-functions/vector.adoc
Original file line number Diff line number Diff line change
Expand Up @@ -513,6 +513,71 @@ SELECT vector.dequantizebinary(vector.quantizebinary([0.1, 0.5, 0.9]))

===== Scoring and Fusion

[[vector-fuse-ext]]
====== vector.fuse()

Fuses the ranked rows of two or more sources into one ranking, server-side. See <<hybrid-search,Hybrid Search with `vector.fuse`>> for the strategies and worked examples.

*Syntax:* `vector.fuse(<source1>, <source2>, ... [, { <option>: <value>, ... }])`

*Parameters:*

* `source` - At least two. Each is a list of rows carrying a `@rid`: the output of `vector.neighbors` or `vector.sparseNeighbors`, a sub-SELECT such as `(SELECT @rid, $score FROM Doc WHERE SEARCH_INDEX('Doc[content]', :q) = true LIMIT 100)`, or `SEARCH_INDEX('Doc[content]', :q)` passed directly.

*Options map:*

[%header,cols="1,1,3"]
|===
| Key | Type | Description
| `fusion` | String | `RRF` (default), `DBSF` or `LINEAR`.
| `k` | Integer | RRF constant. Defaults to `60`.
| `weights` | List of numbers | One weight per source. Defaults to `1.0` for each.
| `groupBy` | String | Keeps at most `groupSize` rows per value of this property.
| `groupSize` | Integer | Rows per group when `groupBy` is set. Defaults to `1`.
| `limit` | Integer | Maximum number of rows returned. Defaults to no limit: every fused candidate is returned, and loaded.
|===

*Returns:* List - One row per fused record, with its properties and the fused `score`, best first.

Every row of every source is fused, and without `limit` every fused candidate is then loaded, so the cost grows with the number of rows the sources produce.
A full-text leg without a `LIMIT` produces one row per match, so the query gets slower as the full-text match count grows, whatever `k` the other legs use.
Bound both, as shown in <<hybrid-search-bounded,Bounding the cost of a full-text leg>>.

Since 26.11.1, `SEARCH_INDEX(...)` passed directly as a source carries the score of each match, so `fusion: 'DBSF'` and `'LINEAR'` accept a full-text leg written that way (before, they refused it for lack of a score).

'''

[[vector-rerank-ext]]
====== vector.rerank()

Re-scores the candidates of a first, cheaper search on full-precision vectors and returns the best `k`: the "prefetch and rerank" pattern.
The first stage can use a lossy index (`BINARY`, `PRODUCT`, sparse, full-text) with a high over-fetch, since the rerank corrects its ranking.

*Syntax:* `vector.rerank(<source>, <query-vector>, <embedding-property>, <k>)`

*Parameters:*

* `source` - The candidate rows: the output of `vector.neighbors`, `vector.fuse`, or a sub-SELECT returning records.
* `query-vector` - The full-precision query vector.
* `embedding-property` - The property holding each candidate's full-precision vector. A candidate without it, or whose record no longer exists, is skipped. A vector whose dimension differs from the query's is an error.
* `k` - Number of rows to return.

*Returns:* List - The `k` candidates with the highest cosine similarity to the query vector, as a `score`, best first.

[source,sql]
----
SELECT expand(`vector.rerank`(
`vector.neighbors`('Doc[binary_embedding]', :qBinary, 1000),
:qFull,
'embedding',
100))
----

Since 26.11.1, a candidate row that already carries its record (the rows of `vector.fuse`, `vector.neighbors` and a plain `SELECT` do) is scored on that record without being reloaded, and only the best `k` candidates are kept while scanning.
The cost still includes reading the embedding of every candidate, so bound the source (`k` of `vector.neighbors`, `limit` of `vector.fuse`) to the number of candidates worth re-scoring.

'''

[[vector-hybridscore-ext]]
====== vector.hybridscore()

Expand Down
2 changes: 1 addition & 1 deletion src/main/asciidoc/reference/time-series/chapter.adoc
Original file line number Diff line number Diff line change
Expand Up @@ -21,7 +21,7 @@ ArcadeDB's native time series engine is documented in the <<timeseries,Time Seri
| Ingest (Line Protocol) | `POST /api/v1/ts/{db}/write` with `SensorReading,sensor_id=s1 temperature=22.5 1708430400000000000`
| Ingest (SQL) | `INSERT INTO SensorReading SET ts = '2026-02-20T10:00:00Z', sensor_id = 's1', temperature = 22.5`
| Query (SQL) | `SELECT ts.timeBucket('1h', ts) AS hour, avg(temperature) FROM SensorReading GROUP BY hour`
| Query (PromQL) | `GET /ts/{db}/prom/api/v1/query?query=avg(SensorReading{sensor_id="s1"})`
| Query (PromQL) | `GET /api/v1/ts/{db}/prom/api/v1/query?query=avg(SensorReading{sensor_id="s1"})`
| Retention | `CREATE TIMESERIES TYPE ... RETENTION 90 DAYS`
| Downsampling | `ALTER TIMESERIES TYPE ... ADD DOWNSAMPLING POLICY AFTER 7 DAYS GRANULARITY 1 MINUTES`
| Compact now | `COMPACT TIMESERIES TYPE SensorReading` (seals the mutable tail without waiting for the scheduler)
Expand Down
4 changes: 2 additions & 2 deletions src/main/asciidoc/tutorials/time-series-tutorial.adoc
Original file line number Diff line number Diff line change
Expand Up @@ -136,11 +136,11 @@ ArcadeDB includes a native PromQL evaluator. Query via HTTP:
[source,bash]
----
# Instant query
curl "http://localhost:2480/ts/mydb/prom/api/v1/query?query=avg(SensorReading{sensor_id='sensor-A'})" \
curl "http://localhost:2480/api/v1/ts/mydb/prom/api/v1/query?query=avg(SensorReading{sensor_id='sensor-A'})" \
-u root:arcadedb

# Range query with rate calculation
curl "http://localhost:2480/ts/mydb/prom/api/v1/query_range?\
curl "http://localhost:2480/api/v1/ts/mydb/prom/api/v1/query_range?\
query=rate(SensorReading{sensor_id='sensor-A'}[5m])&\
start=1708430400&end=1708434000&step=60" \
-u root:arcadedb
Expand Down
Loading