From 9268498ff69761cf46291934465d3eb9fff6939b Mon Sep 17 00:00:00 2001
From: Nick Woolmer <29717167+nwoolmer@users.noreply.github.com>
Date: Thu, 9 Oct 2025 18:13:04 +0100
Subject: [PATCH] wip
---
documentation/reference/api/java-embedded.md | 150 +++++++++++++------
1 file changed, 108 insertions(+), 42 deletions(-)
diff --git a/documentation/reference/api/java-embedded.md b/documentation/reference/api/java-embedded.md
index aea96c190..b6aba3ef3 100644
--- a/documentation/reference/api/java-embedded.md
+++ b/documentation/reference/api/java-embedded.md
@@ -31,6 +31,7 @@ import TabItem from "@theme/TabItem"
questdb
${release.name}
+
`}
)
@@ -55,59 +56,124 @@ import TabItem from "@theme/TabItem"
-## Writing data
-This section provides example codes to write data to WAL and non-WAL tables. See
-[Write Ahead Log](/docs/concept/write-ahead-log) for details about the
-differences between WAL and non-WAL tables.
+## Overview
-The following writers are available for data ingestion:
+QuestDB is composed of three major packages:
-- `WalWriter` for WAL tables
-- `TableWriter` for non-WAL tables
-- `TableWriterAPI` for both WAL and non-WAL tables as it is an interface for
- `WalWriter` and `Table Writer`
+- `cairo`, a custom storage engine which handles reading and writing data to disk, underpinning all database operations.
+The underlying native format is [colummnar](/concept/storage-model), and optimised for time-series workloads.
+- `griffin`, a custom query engine, which provides a rich SQL layer to interact with the data.
+- `cutlass`, a custom network I/O stack, which provides TCP, HTTP, PG Wire, and other endpoints to write and read data.
+
+QuestDB is written for performance, and therefore diverges from more traditional Java projects. Instead, the database uses off-heap memory,
+object pooling, vectorised execution, and other key techniques to ensure that each of these packages is as performant as possible. Some key features
+are written in native code directly, using C/C++/Rust - for example, the JIT compiler for WHERE filters, and Parquet encoding.
+
+When working with QuestDB as an embedded storage engine, it is important to understand this and ensure that any shims or APIs you
+add in front of the database a similarly performant, or else you will not be able to get the most out of it.
+
+
+### High-level API
+
+
+When using QuestDB as an embedded storage engine, the first class you will interact with is [`CairoEngine`](https://github.com/questdb/questdb/blob/master/core/src/main/java/io/questdb/cairo/CairoEngine.java) class.
+
+`CairoEngine` represents a specific database instance, with a root directory. Only a single engine can own a data directory at any one time.
+
+As the central context, `CairoEngine` becomes the root for your storage and querying operations.
+
+`cairo` is the custom storage engine that handles reading and writing data to disk. `cairo` is combined with `griffin` , our SQL query engine,
+to add rich SQL support on top. `cutlass` is an additional package that handles external API and network I/O.
+
+In order
+
+### Creating tables
+
+There are several ways you can create a table in QuestDB. However, the most straightforward method is to use
+the SQL interface to execute a DDL.
+
+Queries are executed using a `SqlExecutionContext`. This is a context for compiling and running a particular query.
+
+Therefore, to create a table, we will instruct `CairoEngine` a `CREATE TABLE` statement within a particular `SqlExecutionContext`.
+
+```java title="Creating a table"
+// Create a configuration for the engine.
+final CairoConfiguration configuration = new DefaultCairoConfiguration("/path/to/dbroot");
+
+// Initialise the engine
+try (CairoEngine engine = new CairoEngine(configuration)) {
+ // Create a new query execution context
+ try (SqlExecutionContext executionContext =
+ new SqlExecutionContextImpl(engine, 1) // single-threaded query execution
+ .with(AllowAllSecurityContext.INSTANCE, null)) // query can access any table)
+ {
+ // Execute the create table statement
+ engine.execute(
+ "CREATE TABLE 'trades' (" +
+ "symbol SYMBOL CAPACITY 256 CACHE, " +
+ "side SYMBOL CAPACITY 256 CACHE, " +
+ "price DOUBLE, " +
+ "amount DOUBLE, " +
+ "timestamp TIMESTAMP" +
+ ") timestamp(timestamp) PARTITION BY DAY WAL",
+ executionContext
+ );
+ } catch (SqlException e) {
+ throw new RuntimeException(e);
+ }
+}
+```
+
+
+### Writing data
+
+QuestDB supports two main types of tables - WAL ([write-ahead-log](/docs/concept/write-ahead-log)) and non-WAL. Most users should use WAL tables for their time-series data, as WAL brings many benefits, notably concurrent writes from multiple APIs.
+
+Table writing is governed by the [`TableWriterAPI`](https://github.com/questdb/questdb/blob/master/core/src/main/java/io/questdb/cairo/TableWriterAPI.java) interface.
+
+This defines common functionality across both of the `WalWriter` (WAL tables) and `TableWriter` (non-WAL tables) classes.
+
+This API cannot be used to create a table - the table must already exist.
+
+In the prior example, we created a WAL table, so we will now write to it using `WalWriter`.
### Writing data using `WalWriter`
-The `WalWriter` facilitates table writes to WAL tables. To successfully create
-an instance of `WalWriter`, the table must already exist.
+`WalWriter` first writes the data into a write-head-log file. Then, the `ApplyWal2TableJob` will later commit this data
+to the table, making it visible for read.
+
+In a standalone QuestDB deployment, a thread-pool automatically executes `ApplyWal2TableJob`, keeping your tables up to date.
+
+For this example, we will create and execution a job to move the data from the WAL to the table.
+
+
```java title="Example WalWriter"
-final CairoConfiguration configuration = new DefaultCairoConfiguration("data_dir");
-try (CairoEngine engine = new CairoEngine(configuration)) {
- final SqlExecutionContext ctx = new SqlExecutionContextImpl(engine, 1)
- .with(AllowAllSecurityContext.INSTANCE, null);
- engine.ddl("CREATE TABLE testTable (" +
- "a int, b byte, c short, d long, e float, g double, h date, " +
- "i symbol, j string, k boolean, l geohash(8c), ts timestamp" +
- ") TIMESTAMP(ts) PARTITION BY DAY WAL", ctx);
+//
- // write data into WAL
- final TableToken tableToken = engine.getTableTokenIfExists("testTable");
- try (WalWriter writer = engine.getWalWriter(tableToken)) {
- for (int i = 0; i < 3; i++) {
- TableWriter.Row row = writer.newRow(Os.currentTimeMicros());
- row.putInt(0, 123);
- row.putByte(1, (byte) 1111);
- row.putShort(2, (short) 222);
- row.putLong(3, 333);
- row.putFloat(4, 4.44f);
- row.putDouble(5, 5.55);
- row.putDate(6, System.currentTimeMillis());
- row.putSym(7, "xyz");
- row.putStr(8, "abc");
- row.putBool(9, true);
- row.putGeoHash(10, GeoHashes.fromString("u33dr01d", 0, 8));
- row.append();
- }
- writer.commit();
- }
+// first, acquire a handle for the table
+final TableToken tableToken = engine.getTableTokenIfExists("trades");
- // apply WAL to the table
- try (ApplyWal2TableJob walApplyJob = new ApplyWal2TableJob(engine, 1, 1)) {
- while (walApplyJob.run(0)) ;
+// get a WalWriter from the CairoEngine's object pool
+try (WalWriter writer = engine.getWalWriter(tableToken)) {
+ // append rows to the table
+ for (int i = 0; i < 3; i++) {
+ TableWriter.Row row = writer.newRow(Os.currentTimeMicros());
+ row.putSym(0, "ETH-USD");
+ row.putSym(1, "sell");
+ row.putDouble(2, 2615.54);
+ row.putDouble(3, 0.00044);
+ row.append();
}
+ // commit the data to the write-ahead-log
+ writer.commit();
+}
+
+// apply WAL to the table
+try (ApplyWal2TableJob walApplyJob =
+ new ApplyWal2TableJob(engine, 1)) {
+ while (walApplyJob.run(0)) ;
}
```