Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
35 changes: 35 additions & 0 deletions .github/PULL_REQUEST_TEMPLATE.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,35 @@
## Problem

<!-- What is wrong / missing, and for whom. Link the issue if one exists. -->

## Changes

<!-- What this PR does. For cross-repo or stacked work, list companion PRs
and merge-order guidance ("safe in either order because…"). -->

## Tests

<!-- Evidence, not assertion: paste the numbers.
e.g. `cargo test`: 19 passed / 0 failed; `npm test` smoke green.
`npm test` loads the built .node — build before testing. -->

## Compatibility

<!-- Does this touch the record log, blob layout, snapshot encoding or wire
format? If so: do existing stores still open, and does a store written
by this branch still open on current main? "No format change" is a fine
answer. -->

## Not verified

<!-- Store size exercised, whether recovery / torn-tail paths were hit, which
targets you built. "Nothing — full suite plus a real store" is a fine
answer; silence is not. -->

---

- [ ] `CHANGELOG.md` updated under `## Unreleased` — or this change is
internal-only / test-only / docs-only (apply the `no-changelog` label).

<!-- AI-assisted contributions are welcome and normal here — see
CONTRIBUTING.md for the attribution convention (footer + Co-Authored-By). -->
32 changes: 32 additions & 0 deletions .github/workflows/changelog.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,32 @@
name: Changelog

on:
pull_request:
types: [opened, synchronize, reopened, labeled, unlabeled]

permissions:
contents: read

jobs:
changelog-entry:
name: Changelog entry present
runs-on: ubuntu-latest
if: "!contains(github.event.pull_request.labels.*.name, 'no-changelog')"

steps:
- uses: actions/checkout@d23441a48e516b6c34aea4fa41551a30e30af803 # v6.1.0
with:
fetch-depth: 0

- name: Require CHANGELOG.md update when src/ changes
run: |
base="${{ github.event.pull_request.base.sha }}"
head="${{ github.event.pull_request.head.sha }}"
changed=$(git diff --name-only "$base...$head")
echo "Changed files:"
echo "$changed"
if echo "$changed" | grep -q '^src/' && ! echo "$changed" | grep -qx 'CHANGELOG.md'; then
echo "::error::This PR touches src/ but not CHANGELOG.md. Add an entry under 'Unreleased' (see CONTRIBUTING.md), or apply the 'no-changelog' label if the change is internal-only."
exit 1
fi
echo "OK"
12 changes: 8 additions & 4 deletions .github/workflows/ci.yml
Original file line number Diff line number Diff line change
Expand Up @@ -15,18 +15,22 @@ jobs:
runs-on: ubuntu-latest

steps:
- uses: actions/checkout@v4
- uses: actions/checkout@11d5960a326750d5838078e36cf38b85af677262 # v4.4.0
with:
persist-credentials: false

- name: Setup Node.js
uses: actions/setup-node@v4
uses: actions/setup-node@49933ea5288caeca8642d1e84afbd3f7d6820020 # v4.4.0
with:
node-version: 20

- name: Install Rust toolchain
uses: dtolnay/rust-toolchain@stable
uses: dtolnay/rust-toolchain@4360b52568e2003a75bf9bc1d59f33a8e3fc893c # stable
with:
toolchain: stable

- name: Cache Rust build
uses: Swatinem/rust-cache@v2
uses: Swatinem/rust-cache@6323deb102c322ba6fcbdcafc7e3dddab59af2b6 # v2.9.2

# npm ci installs the lock verbatim: npm >= 11 'npm install' silently
# re-resolves platform packages missing from the lock, so only npm ci
Expand Down
76 changes: 68 additions & 8 deletions .github/workflows/publish.yml
Original file line number Diff line number Diff line change
Expand Up @@ -31,16 +31,22 @@ jobs:
name: Build - ${{ matrix.target }}

steps:
- uses: actions/checkout@v4
- uses: actions/checkout@11d5960a326750d5838078e36cf38b85af677262 # v4.4.0
with:
persist-credentials: false

- name: Setup Node.js
uses: actions/setup-node@v4
uses: actions/setup-node@49933ea5288caeca8642d1e84afbd3f7d6820020 # v4.4.0
with:
node-version: 20

- name: Install Rust toolchain
uses: dtolnay/rust-toolchain@stable
uses: dtolnay/rust-toolchain@4360b52568e2003a75bf9bc1d59f33a8e3fc893c # stable
with:
# Named explicitly: `stable` was a BRANCH ref, so the channel used to
# come from the ref itself. Pinned to a SHA, it would otherwise come
# from whichever branch's action.yml that SHA carries.
toolchain: stable
targets: ${{ matrix.target }}

- name: Install cross-compilation tools (aarch64-linux)
Expand All @@ -60,15 +66,15 @@ jobs:
CARGO_TARGET_AARCH64_UNKNOWN_LINUX_GNU_LINKER: aarch64-linux-gnu-gcc

- name: Upload native module
uses: actions/upload-artifact@v4
uses: actions/upload-artifact@ea165f8d65b6e75b540449e92b4886f43607fa02 # v4.6.2
with:
name: bindings-${{ matrix.target }}
path: '*.node'
if-no-files-found: error

- name: Upload JS bindings (once)
if: matrix.target == 'x86_64-unknown-linux-gnu'
uses: actions/upload-artifact@v4
uses: actions/upload-artifact@ea165f8d65b6e75b540449e92b4886f43607fa02 # v4.6.2
with:
name: js-bindings
path: |
Expand All @@ -80,6 +86,11 @@ jobs:
name: Publish to npm
runs-on: ubuntu-latest
needs: build
# Gated on the ref, not the event, so a manual re-dispatch against an
# existing tag still publishes — but a dispatch against a branch cannot,
# which would otherwise publish whatever version package.json happened to
# carry, bypassing the changelog guard below.
if: startsWith(github.ref, 'refs/tags/v')

# OIDC trusted publishing: GitHub mints a short-lived id-token, npm CLI
# exchanges it for a publish credential. No long-lived NPM_TOKEN secret.
Expand All @@ -90,10 +101,21 @@ jobs:
id-token: write

steps:
- uses: actions/checkout@v4
- uses: actions/checkout@11d5960a326750d5838078e36cf38b85af677262 # v4.4.0
with:
persist-credentials: false

- name: Require changelog section for this release
run: |
ver="${GITHUB_REF_NAME#v}"
esc=$(printf '%s' "$ver" | sed 's/[.]/\\./g')
if ! grep -Eq "^## ${esc}([^0-9]|$)" CHANGELOG.md; then
echo "::error::CHANGELOG.md has no '## ${ver}' section for tag ${GITHUB_REF_NAME}. Retitle the Unreleased section before tagging (see CONTRIBUTING.md)."
exit 1
fi

- name: Setup Node.js
uses: actions/setup-node@v4
uses: actions/setup-node@49933ea5288caeca8642d1e84afbd3f7d6820020 # v4.4.0
with:
# node 24 ships npm >= 11.5.1, required for OIDC publishing.
node-version: 24
Expand All @@ -103,7 +125,7 @@ jobs:
run: npm ci

- name: Download all artifacts
uses: actions/download-artifact@v4
uses: actions/download-artifact@d3f86a106a0bac45b974a628896c90dbdf5c8093 # v4.3.0
with:
path: artifacts

Expand All @@ -130,3 +152,41 @@ jobs:

- name: Publish
run: npm publish --access public --provenance

github-release:
name: GitHub release notes
runs-on: ubuntu-latest
# Deliberately independent of the npm publish job — and of the native
# build matrix it needs: some consumers run github-clone checkouts, and
# release notes must exist even when the cross-build or publish fails.
if: startsWith(github.ref, 'refs/tags/v')

permissions:
contents: write

steps:
- uses: actions/checkout@11d5960a326750d5838078e36cf38b85af677262 # v4.4.0
with:
persist-credentials: false

- name: Mirror changelog section into release notes
env:
GH_TOKEN: ${{ github.token }}
run: |
ver="${GITHUB_REF_NAME#v}"
# Section header is '## X.Y.Z — YYYY-MM-DD'; match the version
# field exactly (string compare, no regex escaping needed).
awk -v ver="$ver" '
/^## / { if (in_section) exit; if ($2 == ver) { in_section = 1; next } }
in_section { print }
' CHANGELOG.md | sed '/./,$!d' > notes.md
if ! [ -s notes.md ]; then
echo "::error::No CHANGELOG.md section found for ${ver}."
exit 1
fi
if gh release view "$GITHUB_REF_NAME" >/dev/null 2>&1; then
gh release edit "$GITHUB_REF_NAME" --notes-file notes.md
else
gh release create "$GITHUB_REF_NAME" --title "$GITHUB_REF_NAME" \
--notes-file notes.md --verify-tag
fi
121 changes: 121 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,121 @@
# Changelog

Notable changes to `@animalabs/chronicle`, loosely following
[Keep a Changelog](https://keepachangelog.com/). Entries land with the change
that causes them — see [CONTRIBUTING.md](CONTRIBUTING.md#changelog).

Releases up to and including 0.2.7 predate this file; for their contents see
`git log` and the
[releases page](https://github.com/anima-research/chronicle/releases).

## Unreleased

## 0.3.0 — 2026-08-01

### Breaking (on-disk format)

- **`state_update` records are now MessagePack-encoded, and a store that has
been written by 0.3.0 cannot be read by earlier chronicle.** New records are
written with `to_vec_named` and `serde_bytes` on the value fields, so
`StateOperation` payloads travel as raw binary instead of arrays of JSON
integers.
- **Who needs to act:** anyone who may need to roll a store back to a
pre-0.3.0 chronicle. The format change is forward-only — take a copy
before upgrading if rollback has to stay open.
- **Migration:** none. Nothing rewrites existing records; the new encoding
applies to records written from here on.
- **Unchanged:** existing stores open and read normally. `decode` gates on
the record's encoding tag and historical JSON payloads (written as
`RecordInput::raw`) decode through the fallback arm, pinned by a
handwritten-legacy-payload test. Mixed JSON-history/MessagePack-tail
stores were verified against a copy of a live 6 GB store. The JSON wire
format itself is unchanged.
- JS consumers that previously reached into record payloads with
`JSON.parse` should move to the new `getStateUpdateJson(id)`, which is
encoding-agnostic.

### Added

- **`update_state_strategy`** (napi: `updateStateStrategy(registration)`) — the
explicit upsert leg for snapshot cadence. Registrations persist in
`state.bin`, so consumers that re-register on boot could never change cadence
on an existing store: `register_state` errors with `StateExists` and the
first-registration values won forever. Restricted to the same strategy kind,
because changing the kind under a live chain would change reconstruction
semantics for records already on disk; cadence fields steer future snapshot
scheduling only. `initialValue` is ignored, and `register_state` deliberately
stays non-upserting.
- **`getStateUpdateJson(id)`** — encoding-agnostic `state_update` reader for JS
consumers.
- **`materialize_operations`** re-exported at `lib.rs` beside
`apply_operation`.
- **Scaling gates as CI tests** — `tests/disk_scaling.rs` (on-disk bytes vs op
count), `tests/reconstruction_scaling.rs` (cold-read latency at worst-phase
sizes), and `tests/materialize_equivalence.rs` (differential
fold-vs-single-pass equivalence, including a 2000-case fuzz). Plus manual
probes: `tests/perf_probe.rs` for fixed-size per-op numbers and
`examples/store_probe.rs` for measuring against a store copy.
- The criterion bench suite compiles again — it had imported the crate under
its pre-rename name (`record_store`) since the initial release — and now
covers point reads, JSON appends and tree ops.

### Changed

- **Full-snapshot spacing is size-aware** (#11). Fulls fired on a fixed
`delta × full` op interval and embed the whole state, so growing AppendLog
and Tree states paid O(N²/2K) disk and O(N/K) append latency. The interval
now grows with the state — a full fires once the ops covered since the last
full reach the item count at that full — with the configured interval as a
floor, so small states keep their configured cadence exactly. Total snapshot
bytes telescope to ≤ 2× final state size. JSON appends measured 77 µs →
1,709 µs per op across 1k → 50k accumulated entries under the old cadence;
they are now flat at ~20 µs regardless of accumulated size.
- **Reconstruction is single-pass.** Reopen, cache miss, branch switch and time
travel folded `apply_operation` over the op chain, re-parsing and
re-serializing the entire state per op — O(tail × N). `materialize_operations`
decodes the base once, applies the tail in place, and encodes once —
O(state + ops), with semantics preserved exactly. Cold reads went from
22.4 → 10.7 ms and 259.7 → 44.2 ms at 3.9k/15.9k appends (11.59× → 4.13×
per 4× N: linear).
- **Edits no longer force early full snapshots.** With tails cheap to traverse,
the `has_non_append` forced-full is retired: a non-Append op still blocks
delta snapshots, but the raw tail now rides until the size-aware full fires.
The old behavior cost O(N²/D) disk on any growing log taking regular edits —
the summary-merge pattern measured 439 MB of store for 16k logical ops,
now 18.2 MB (24×).
- **Point lookups of the last item are O(1).** Every state write pops the LRU
caches, and a write-through consumer's lookup of the item it just appended
re-materialized the whole state inside every append. When the head record is
an Append and the caller asks for the last index, the item is now served from
the head record alone. Against a copy of a live store (19,767 messages, 33 MB
state, 6 GB log), append+lookup went 421 ms → 94 µs.
- **Tree point reads are cached in decoded form.** `tree_get`/`tree_list`
deserialized the entire path→entry map per call — 16.7 µs → 2.28 ms across
100 → 10k entries. The state manager now caches the decoded `TreeState` per
`head_offset` (same key scheme and staleness discipline as the per-item
cache); `tree_get` is flat at ~150 ns at all measured sizes. The trade is
that the decoded-tree LRU is bounded by entries, not bytes.
- Tree op counts are tracked as an upper bound (overwrites count as inserts,
corrected at each full snapshot) — sufficient for spacing, and it only ever
spaces fulls further apart.

### Fixed

- **Snapshot-strategy states no longer walk the entire chain on cold reads.**
`reconstruct_from_disk` / `get_state_at` / `find_chain_info_at` broke the
backward walk only on `Snapshot` records; `Set` fell through to the catch-all
and the walk continued to sequence 0. Snapshot-strategy states are written
via `Set` and never get a periodic `Snapshot` record, so cold reads
deserialized every full-state `Set` back to the beginning and discarded all
but the last — chains up to 3,928 deep across an 11.7 GB log, measured at
6–8 minutes to boot. `Set` replaces the whole state exactly like `Snapshot`,
so the newest one is a valid terminal; these reads are now O(1).
- `CompactionStats::ops_since_last_full_snapshot` counted ops plus delta count
instead of ops (deltas × `delta_snapshot_every` + raw ops), undercounting
roughly 100×.
- Interval-floor arithmetic in both strategy arms uses `saturating_mul`; an
overflowing config previously wrapped.
- Legacy indexes deserialize the new head fields to 0 and keep the configured
cadence until the first full snapshot stamps a baseline. `StateChainHead`'s
positional wire format is now documented and guarded against both the v2
(pre-#11) and v1 (pre-`item_count`) layouts.
Loading
Loading