Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
46 changes: 23 additions & 23 deletions .env.example
Original file line number Diff line number Diff line change
Expand Up @@ -81,41 +81,41 @@ OPENBUCKET_REGION=us-east-1
# WEBHOOK_POLL_MS=15000

# --- async replication to an external S3 target (optional, STORY-0900) ---
# Set OB_REPLICATION_ENABLED=true to asynchronously mirror every object PUT/DELETE
# Set OPENBUCKET_REPLICATION_ENABLED=true to asynchronously mirror every object PUT/DELETE
# to an external S3-compatible bucket (AWS S3 / Cloudflare R2 / Backblaze B2 / MinIO)
# via a durable transactional outbox + background drain worker (per-key ordering,
# last-writer-wins coalescing, exponential backoff, dead-letter cap). When enabled,
# OB_REPLICATION_BUCKET and both credentials are REQUIRED (a partial config refuses
# OPENBUCKET_REPLICATION_BUCKET and both credentials are REQUIRED (a partial config refuses
# to boot). The worker sends object PLAINTEXT, so an http:// endpoint warns at boot.
# OB_REPLICATION_ENABLED=false
# OB_REPLICATION_ENDPOINT=https://<accountid>.r2.cloudflarestorage.com # omit for real AWS S3
# OB_REPLICATION_REGION=us-east-1
# OB_REPLICATION_BUCKET=my-remote-mirror
# OB_REPLICATION_ACCESS_KEY_ID=
# OB_REPLICATION_SECRET_ACCESS_KEY=
# OB_REPLICATION_FORCE_PATH_STYLE=true # true for MinIO/S3-compat; false for AWS
# OPENBUCKET_REPLICATION_ENABLED=false
# OPENBUCKET_REPLICATION_ENDPOINT=https://<accountid>.r2.cloudflarestorage.com # omit for real AWS S3
# OPENBUCKET_REPLICATION_REGION=us-east-1
# OPENBUCKET_REPLICATION_BUCKET=my-remote-mirror
# OPENBUCKET_REPLICATION_ACCESS_KEY_ID=
# OPENBUCKET_REPLICATION_SECRET_ACCESS_KEY=
# OPENBUCKET_REPLICATION_FORCE_PATH_STYLE=true # true for MinIO/S3-compat; false for AWS
# Drain tuning (defaults shown).
# OB_REPLICATION_MAX_ATTEMPTS=12
# OB_REPLICATION_DRAIN_INTERVAL_MS=5000
# OB_REPLICATION_BATCH_KEYS=50
# OB_REPLICATION_LARGE_OBJECT_THRESHOLD_BYTES=67108864
# OPENBUCKET_REPLICATION_MAX_ATTEMPTS=12
# OPENBUCKET_REPLICATION_DRAIN_INTERVAL_MS=5000
# OPENBUCKET_REPLICATION_BATCH_KEYS=50
# OPENBUCKET_REPLICATION_LARGE_OBJECT_THRESHOLD_BYTES=67108864

# --- scheduled backups & retention (optional, STORY-1203) ---
# Set OB_SCHEDULED_BACKUP_ENABLED=true to write a .zip snapshot (identical to the
# Set OPENBUCKET_SCHEDULED_BACKUP_ENABLED=true to write a .zip snapshot (identical to the
# admin download) on a schedule, with retention. Exactly ONE of INTERVAL_MINUTES /
# CRON must be set (both/neither refuses to boot; a malformed cron fails fast).
# Snapshots hold decrypted plaintext object bytes — the backup dir (default
# <DATA_DIR>/backups, files 0o600 under a 0o700 dir) inherits the data volume's
# trust boundary. Retention is union (keep-last OR max-age); disk growth is bounded.
# OB_SCHEDULED_BACKUP_ENABLED=false
# OB_SCHEDULED_BACKUP_SCOPE=instance # instance = one snapshot; buckets = one per bucket
# OB_SCHEDULED_BACKUP_INTERVAL_MINUTES=1440 # every 24h (mutually exclusive with CRON)
# OB_SCHEDULED_BACKUP_CRON=0 3 * * * # 5-field cron (mutually exclusive with INTERVAL)
# OB_SCHEDULED_BACKUP_DIR=/data/backups # default <DATA_DIR>/backups
# OB_SCHEDULED_BACKUP_KEEP_LAST=7 # keep the newest N (a hard floor)
# OB_SCHEDULED_BACKUP_MAX_AGE_DAYS=30 # also keep anything younger than this
# OB_SCHEDULED_BACKUP_CHECK_INTERVAL_MS=60000 # wake tick — how often "is a snapshot due?"
# OB_SCHEDULED_BACKUP_PUSH_TO_REPLICATION=false # also push the .zip to the replication target
# OPENBUCKET_SCHEDULED_BACKUP_ENABLED=false
# OPENBUCKET_SCHEDULED_BACKUP_SCOPE=instance # instance = one snapshot; buckets = one per bucket
# OPENBUCKET_SCHEDULED_BACKUP_INTERVAL_MINUTES=1440 # every 24h (mutually exclusive with CRON)
# OPENBUCKET_SCHEDULED_BACKUP_CRON=0 3 * * * # 5-field cron (mutually exclusive with INTERVAL)
# OPENBUCKET_SCHEDULED_BACKUP_DIR=/data/backups # default <DATA_DIR>/backups
# OPENBUCKET_SCHEDULED_BACKUP_KEEP_LAST=7 # keep the newest N (a hard floor)
# OPENBUCKET_SCHEDULED_BACKUP_MAX_AGE_DAYS=30 # also keep anything younger than this
# OPENBUCKET_SCHEDULED_BACKUP_CHECK_INTERVAL_MS=60000 # wake tick — how often "is a snapshot due?"
# OPENBUCKET_SCHEDULED_BACKUP_PUSH_TO_REPLICATION=false # also push the .zip to the replication target

# --- shutdown (optional) ---
# SHUTDOWN_DRAIN_MS=30000
19 changes: 18 additions & 1 deletion CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -9,6 +9,23 @@ versions may include breaking changes.

## [Unreleased]

### Changed

- **BREAKING (pre-1.0): standalone env-var prefix unified onto `OPENBUCKET_`.**
The replication, scheduled-backup, and integrity-scrub feature blocks used an
`OB_` prefix while every other feature block (tiering, endpoint, region, SSE)
used `OPENBUCKET_`. They are now consistent. Rename in your `.env` / compose /
deploy config:
- `OB_REPLICATION_*` → `OPENBUCKET_REPLICATION_*`
- `OB_SCHEDULED_BACKUP_*` → `OPENBUCKET_SCHEDULED_BACKUP_*`
- `OB_INTEGRITY_*` → `OPENBUCKET_INTEGRITY_*`

This is a hard rename with **no backward-compatible aliases** — the old names
are no longer read. The bare core vars (`DATA_DIR`, `JWT_SECRET`,
`ROOT_ACCESS_KEY_ID`, `PORT`, …) and the already-bare feature knobs
(`WEBHOOK_*`, `METRICS_*`, `RESTORE_*`, …) are unchanged. The programmatic
`OpenBucketModule.forRoot` API is unaffected.

## [0.1.0-alpha.18] — 2026-07-08

### Added
Expand Down Expand Up @@ -168,7 +185,7 @@ Feature release — durability & cloud replication (EPIC-10).
**transactional outbox** and mirrored by a background worker with per-key
ordering, last-writer-wins coalescing, exponential-backoff retry, and a
dead-letter cap — resuming on boot and surviving remote outages. Configure via
`OB_REPLICATION_*` env vars or the `replication` module option.
`OPENBUCKET_REPLICATION_*` env vars or the `replication` module option.
- **Cold-object tiering** — objects not accessed within a policy window are
offloaded to the replication target to free local disk; a `GET` transparently
rehydrates them (read-through). Lifecycle `<Transition>` rules now drive tiering.
Expand Down
30 changes: 15 additions & 15 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -395,27 +395,27 @@ full admin. See the

### Async replication

Set `OB_REPLICATION_ENABLED=true` to asynchronously mirror every object
Set `OPENBUCKET_REPLICATION_ENABLED=true` to asynchronously mirror every object
mutation to an external S3-compatible bucket (see the embedded
[library README](./libs/nestjs/README.md#async-replication-to-an-external-s3-target)
for how it works). When enabled, `OB_REPLICATION_BUCKET` and both credentials are
for how it works). When enabled, `OPENBUCKET_REPLICATION_BUCKET` and both credentials are
required together (a partial config refuses to boot).

| Variable | Required\* | Default | Notes |
| --------------------------------------- | ---------- | ------------- | --------------------------------------------------------- |
| `OB_REPLICATION_ENABLED` | | `false` | Master switch. Off ⇒ zero cost, outbox stays empty. |
| `OB_REPLICATION_ENDPOINT` | | — | S3-compatible endpoint (R2/B2/MinIO). Omit for real AWS S3. `http://` warns (plaintext). |
| `OB_REPLICATION_REGION` | | `us-east-1` | Target region. |
| `OB_REPLICATION_BUCKET` | ✅ | — | Remote target bucket (must already exist). |
| `OB_REPLICATION_ACCESS_KEY_ID` | ✅ | — | Target credential. |
| `OB_REPLICATION_SECRET_ACCESS_KEY` | ✅ | — | Target credential (never logged; redacted). |
| `OB_REPLICATION_FORCE_PATH_STYLE` | | `true` | `true` for MinIO/S3-compat; `false` for AWS. |
| `OB_REPLICATION_MAX_ATTEMPTS` | | `12` | Dead-letter cap before an intent → `failed`. |
| `OB_REPLICATION_DRAIN_INTERVAL_MS` | | `5000` | Background drain tick interval (≥ 1000). |
| `OB_REPLICATION_BATCH_KEYS` | | `50` | Distinct keys drained per tick. |
| `OB_REPLICATION_LARGE_OBJECT_THRESHOLD_BYTES` | | `67108864` | Stream via multipart above this (64 MiB). |

\* Required only when `OB_REPLICATION_ENABLED=true`.
| `OPENBUCKET_REPLICATION_ENABLED` | | `false` | Master switch. Off ⇒ zero cost, outbox stays empty. |
| `OPENBUCKET_REPLICATION_ENDPOINT` | | — | S3-compatible endpoint (R2/B2/MinIO). Omit for real AWS S3. `http://` warns (plaintext). |
| `OPENBUCKET_REPLICATION_REGION` | | `us-east-1` | Target region. |
| `OPENBUCKET_REPLICATION_BUCKET` | ✅ | — | Remote target bucket (must already exist). |
| `OPENBUCKET_REPLICATION_ACCESS_KEY_ID` | ✅ | — | Target credential. |
| `OPENBUCKET_REPLICATION_SECRET_ACCESS_KEY` | ✅ | — | Target credential (never logged; redacted). |
| `OPENBUCKET_REPLICATION_FORCE_PATH_STYLE` | | `true` | `true` for MinIO/S3-compat; `false` for AWS. |
| `OPENBUCKET_REPLICATION_MAX_ATTEMPTS` | | `12` | Dead-letter cap before an intent → `failed`. |
| `OPENBUCKET_REPLICATION_DRAIN_INTERVAL_MS` | | `5000` | Background drain tick interval (≥ 1000). |
| `OPENBUCKET_REPLICATION_BATCH_KEYS` | | `50` | Distinct keys drained per tick. |
| `OPENBUCKET_REPLICATION_LARGE_OBJECT_THRESHOLD_BYTES` | | `67108864` | Stream via multipart above this (64 MiB). |

\* Required only when `OPENBUCKET_REPLICATION_ENABLED=true`.

The admin console's **Replication** page (and the `/api/admin/replication` API)
shows replication health — pending/failed depth and lag — and offers a
Expand Down
24 changes: 12 additions & 12 deletions apps/docs/docs/concepts/durability.md
Original file line number Diff line number Diff line change
Expand Up @@ -74,8 +74,8 @@ so it never starves request traffic.

```bash
# Enable the scrubber (standalone):
OB_INTEGRITY_SCRUB_ENABLED=true
OB_INTEGRITY_SCRUB_MAX_BYTES_PER_TICK=1073741824 # 1 GiB/tick (default)
OPENBUCKET_INTEGRITY_SCRUB_ENABLED=true
OPENBUCKET_INTEGRITY_SCRUB_MAX_BYTES_PER_TICK=1073741824 # 1 GiB/tick (default)
```

When a blob is `corrupt` **and** a replication target is configured, the scrubber
Expand All @@ -97,11 +97,11 @@ resumes on boot after a crash or a remote outage — local reads and writes keep
working while the remote is unreachable, and the backlog drains on recovery.

```bash
OB_REPLICATION_ENABLED=true
OB_REPLICATION_BUCKET=my-remote-mirror # must already exist
OB_REPLICATION_ACCESS_KEY_ID=…
OB_REPLICATION_SECRET_ACCESS_KEY=…
# Omit OB_REPLICATION_ENDPOINT for real AWS S3; set it for R2/B2/MinIO.
OPENBUCKET_REPLICATION_ENABLED=true
OPENBUCKET_REPLICATION_BUCKET=my-remote-mirror # must already exist
OPENBUCKET_REPLICATION_ACCESS_KEY_ID=…
OPENBUCKET_REPLICATION_SECRET_ACCESS_KEY=…
# Omit OPENBUCKET_REPLICATION_ENDPOINT for real AWS S3; set it for R2/B2/MinIO.
```

:::note[Plaintext over the wire]
Expand Down Expand Up @@ -132,11 +132,11 @@ Beyond the on-demand backup/restore endpoints, OpenBucket can write **`.zip`
snapshots on a schedule** and prune them by a retention policy:

```bash
OB_SCHEDULED_BACKUP_ENABLED=true
OB_SCHEDULED_BACKUP_SCOPE=instance # or 'buckets' (one .zip per bucket)
OB_SCHEDULED_BACKUP_CRON='0 3 * * *' # OR OB_SCHEDULED_BACKUP_INTERVAL_MINUTES
OB_SCHEDULED_BACKUP_KEEP_LAST=7 # retention floor
OB_SCHEDULED_BACKUP_MAX_AGE_DAYS=30 # union with keep-last
OPENBUCKET_SCHEDULED_BACKUP_ENABLED=true
OPENBUCKET_SCHEDULED_BACKUP_SCOPE=instance # or 'buckets' (one .zip per bucket)
OPENBUCKET_SCHEDULED_BACKUP_CRON='0 3 * * *' # OR OPENBUCKET_SCHEDULED_BACKUP_INTERVAL_MINUTES
OPENBUCKET_SCHEDULED_BACKUP_KEEP_LAST=7 # retention floor
OPENBUCKET_SCHEDULED_BACKUP_MAX_AGE_DAYS=30 # union with keep-last
```

Each snapshot is written **atomically** (stream to `<final>.part`, fsync, rename)
Expand Down
20 changes: 10 additions & 10 deletions apps/docs/docs/guides/backup-and-restore.md
Original file line number Diff line number Diff line change
Expand Up @@ -78,14 +78,14 @@ snapshot to your replication target.
### Standalone (env)

```bash
OB_SCHEDULED_BACKUP_ENABLED=true
OB_SCHEDULED_BACKUP_SCOPE=instance # or `buckets` (one .zip per bucket)
OB_SCHEDULED_BACKUP_CRON="0 3 * * *" # 03:00 daily — XOR with INTERVAL_MINUTES
OB_SCHEDULED_BACKUP_KEEP_LAST=7 # keep the newest N (default 7)
OB_SCHEDULED_BACKUP_MAX_AGE_DAYS=30 # also keep anything younger (default 30)
OB_SCHEDULED_BACKUP_PUSH_TO_REPLICATION=false
# OB_SCHEDULED_BACKUP_DIR=/var/lib/openbucket/backups # default <dataDir>/backups
# OB_SCHEDULED_BACKUP_INTERVAL_MINUTES=1440 # instead of a cron
OPENBUCKET_SCHEDULED_BACKUP_ENABLED=true
OPENBUCKET_SCHEDULED_BACKUP_SCOPE=instance # or `buckets` (one .zip per bucket)
OPENBUCKET_SCHEDULED_BACKUP_CRON="0 3 * * *" # 03:00 daily — XOR with INTERVAL_MINUTES
OPENBUCKET_SCHEDULED_BACKUP_KEEP_LAST=7 # keep the newest N (default 7)
OPENBUCKET_SCHEDULED_BACKUP_MAX_AGE_DAYS=30 # also keep anything younger (default 30)
OPENBUCKET_SCHEDULED_BACKUP_PUSH_TO_REPLICATION=false
# OPENBUCKET_SCHEDULED_BACKUP_DIR=/var/lib/openbucket/backups # default <dataDir>/backups
# OPENBUCKET_SCHEDULED_BACKUP_INTERVAL_MINUTES=1440 # instead of a cron
```

### Embedded (`forRoot`)
Expand Down Expand Up @@ -136,15 +136,15 @@ it can never launch a second overlapping snapshot — a concurrent request retur

## Push snapshots off-box

Set `pushToReplication` (env `OB_SCHEDULED_BACKUP_PUSH_TO_REPLICATION=true`) to
Set `pushToReplication` (env `OPENBUCKET_SCHEDULED_BACKUP_PUSH_TO_REPLICATION=true`) to
copy each finished snapshot to your configured [replication
target](./replication-and-tiering.md) under a reserved prefix — so a lost data
volume doesn't take your backups with it.

:::note[Enable replication first]
If you set the push flag but replication is off, the runner logs a boot-time
warning and writes snapshots locally only — the push is a no-op until you
configure `OB_REPLICATION_*`.
configure `OPENBUCKET_REPLICATION_*`.
:::

:::warning[Snapshots are plaintext]
Expand Down
28 changes: 14 additions & 14 deletions apps/docs/docs/guides/replication-and-tiering.md
Original file line number Diff line number Diff line change
Expand Up @@ -23,16 +23,16 @@ target is a durable copy.
### Standalone (env)

```bash
OB_REPLICATION_ENABLED=true
OB_REPLICATION_BUCKET=openbucket-mirror
OB_REPLICATION_ACCESS_KEY_ID=...
OB_REPLICATION_SECRET_ACCESS_KEY=...
OB_REPLICATION_ENDPOINT=https://<accountid>.r2.cloudflarestorage.com # omit for AWS S3
OB_REPLICATION_REGION=us-east-1
OB_REPLICATION_FORCE_PATH_STYLE=true # true for MinIO/R2/B2; false for AWS S3
OPENBUCKET_REPLICATION_ENABLED=true
OPENBUCKET_REPLICATION_BUCKET=openbucket-mirror
OPENBUCKET_REPLICATION_ACCESS_KEY_ID=...
OPENBUCKET_REPLICATION_SECRET_ACCESS_KEY=...
OPENBUCKET_REPLICATION_ENDPOINT=https://<accountid>.r2.cloudflarestorage.com # omit for AWS S3
OPENBUCKET_REPLICATION_REGION=us-east-1
OPENBUCKET_REPLICATION_FORCE_PATH_STYLE=true # true for MinIO/R2/B2; false for AWS S3
```

`OB_REPLICATION_BUCKET` and both credentials are required when replication is on —
`OPENBUCKET_REPLICATION_BUCKET` and both credentials are required when replication is on —
a partial config fails at boot. The target bucket must already exist.

### Embedded (`forRoot`)
Expand All @@ -44,8 +44,8 @@ OpenBucketModule.forRoot({
replication: {
bucket: 'openbucket-mirror',
credentials: {
accessKeyId: process.env.OB_REPLICATION_ACCESS_KEY_ID!,
secretAccessKey: process.env.OB_REPLICATION_SECRET_ACCESS_KEY!,
accessKeyId: process.env.OPENBUCKET_REPLICATION_ACCESS_KEY_ID!,
secretAccessKey: process.env.OPENBUCKET_REPLICATION_SECRET_ACCESS_KEY!,
},
endpoint: 'https://<accountid>.r2.cloudflarestorage.com', // omit for AWS S3
forcePathStyle: true,
Expand All @@ -66,14 +66,14 @@ background drain worker then mirrors those intents to the remote:
- **Last-writer-wins coalescing** — a burst of writes to one key collapses to the
latest state.
- **Retry with exponential backoff** — a transient remote failure is retried up
to a **dead-letter cap** (`OB_REPLICATION_MAX_ATTEMPTS`, default 12).
to a **dead-letter cap** (`OPENBUCKET_REPLICATION_MAX_ATTEMPTS`, default 12).
- **Resume on boot** — the worker picks up pending intents after a restart and
survives remote outages: nothing is lost, it just backs up in the outbox and
drains when the remote returns.

Tuning knobs: `OB_REPLICATION_DRAIN_INTERVAL_MS` (default 5000),
`OB_REPLICATION_BATCH_KEYS` (distinct keys per tick, default 50), and
`OB_REPLICATION_LARGE_OBJECT_THRESHOLD_BYTES` (switch to multipart streaming above
Tuning knobs: `OPENBUCKET_REPLICATION_DRAIN_INTERVAL_MS` (default 5000),
`OPENBUCKET_REPLICATION_BATCH_KEYS` (distinct keys per tick, default 50), and
`OPENBUCKET_REPLICATION_LARGE_OBJECT_THRESHOLD_BYTES` (switch to multipart streaming above
this size, default 64 MiB).

:::warning[The wire carries plaintext]
Expand Down
8 changes: 4 additions & 4 deletions apps/docs/docs/guides/securing-openbucket.md
Original file line number Diff line number Diff line change
Expand Up @@ -144,10 +144,10 @@ catch bit-rot or tampering at rest. It's **off by default** and strictly rate
limited so it never starves request traffic:

```bash
OB_INTEGRITY_SCRUB_ENABLED=true
OB_INTEGRITY_SCRUB_INTERVAL_MS=60000 # tick cadence
OB_INTEGRITY_SCRUB_MAX_OBJECTS_PER_TICK=1000 # per-tick object budget
OB_INTEGRITY_SCRUB_MAX_BYTES_PER_TICK=... # per-tick byte budget
OPENBUCKET_INTEGRITY_SCRUB_ENABLED=true
OPENBUCKET_INTEGRITY_SCRUB_INTERVAL_MS=60000 # tick cadence
OPENBUCKET_INTEGRITY_SCRUB_MAX_OBJECTS_PER_TICK=1000 # per-tick object budget
OPENBUCKET_INTEGRITY_SCRUB_MAX_BYTES_PER_TICK=... # per-tick byte budget
```

Each object gets a verdict (`unchecked` / `ok` / `corrupt`). When a
Expand Down
Loading
Loading