Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
15 changes: 15 additions & 0 deletions CHANGES.md
Original file line number Diff line number Diff line change
Expand Up @@ -126,6 +126,18 @@ To be released.
held activities call the outbox permanent failure handler with
`reason: "circuit-breaker-ttl"`. [[#620], [#778]]

- Added `benchmarkMode` to `createFederation()` and
`FederationBuilder.build()` for cooperative federation benchmarking.
When enabled, Fedify exposes `GET /.well-known/fedify/bench/stats`
for in-process OpenTelemetry metric snapshots and
`POST /.well-known/fedify/bench/trigger` for driving `sendActivity()`
to server-configured benchmark sink recipients. Benchmark mode also
defaults `allowPrivateAddress` to `true` when built-in loaders are used,
defaults `signatureTimeWindow` to `false`, reports queue depth through
the new `fedify.queue.depth` gauge, and adds explicit low-latency
buckets to the signature verification duration histogram.
[[#744], [#782], [#787]]

- Added OpenTelemetry metrics for ActivityPub fanout and activity
lifecycle events, complementing the per-recipient
`activitypub.delivery.*` counters and the per-task
Expand Down Expand Up @@ -248,6 +260,7 @@ To be released.
[#740]: https://github.com/fedify-dev/fedify/issues/740
[#741]: https://github.com/fedify-dev/fedify/issues/741
[#742]: https://github.com/fedify-dev/fedify/issues/742
[#744]: https://github.com/fedify-dev/fedify/issues/744
[#748]: https://github.com/fedify-dev/fedify/pull/748
[#752]: https://github.com/fedify-dev/fedify/issues/752
[#753]: https://github.com/fedify-dev/fedify/pull/753
Expand All @@ -261,6 +274,8 @@ To be released.
[#772]: https://github.com/fedify-dev/fedify/pull/772
[#777]: https://github.com/fedify-dev/fedify/pull/777
[#778]: https://github.com/fedify-dev/fedify/pull/778
[#782]: https://github.com/fedify-dev/fedify/issues/782
[#787]: https://github.com/fedify-dev/fedify/pull/787

### @fedify/fixture

Expand Down
1 change: 1 addition & 0 deletions docs/.vitepress/config.mts
Original file line number Diff line number Diff line change
Expand Up @@ -154,6 +154,7 @@ const MANUAL = {
{ text: "Linting", link: "/manual/lint.md" },
{ text: "Logging", link: "/manual/log.md" },
{ text: "OpenTelemetry", link: "/manual/opentelemetry.md" },
{ text: "Benchmarking", link: "/manual/benchmarking.md" },
{ text: "Deployment", link: "/manual/deploy.md" },
],
activeMatch: "/manual",
Expand Down
173 changes: 173 additions & 0 deletions docs/manual/benchmarking.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,173 @@
---
description: >-
Fedify can expose cooperative benchmark endpoints for measuring federation
workloads without requiring an external metrics backend.
---

Benchmarking
============

*This API is available since Fedify 2.3.0.*

Fedify can run as a cooperative benchmark target by enabling
`~FederationOptions.benchmarkMode`. This mode exposes local benchmark
endpoints under `/.well-known/fedify/bench/` and configures an in-process
OpenTelemetry metrics reader so benchmark clients can collect server-side
measurements without a separate metrics backend.

> [!WARNING]
> Do not enable `benchmarkMode` in production. It is intended for benchmark
> targets that you control.


Enabling benchmark mode
-----------------------

Enable `benchmarkMode` when creating the `Federation` object. If you use the
benchmark trigger endpoint, configure the sink inboxes on the server:

~~~~ typescript twoslash
import type { KvStore } from "@fedify/fedify";
// ---cut-before---
import { createFederation } from "@fedify/fedify";

const federation = createFederation<void>({
// ---cut-start---
kv: null as unknown as KvStore,
// ---cut-end---
benchmarkMode: {
triggerSinks: ["https://sink.example/inbox"],
},
});
~~~~

When enabled, Fedify changes only benchmark-target defaults:

- `~FederationOptions.allowPrivateAddress` defaults to `true`, unless a
custom document loader factory is configured.
- `~FederationOptions.signatureTimeWindow` defaults to `false`.
- Explicit `allowPrivateAddress` and `signatureTimeWindow` values still win.
- Inbox idempotency is unchanged. Benchmark clients that need repeated
deliveries should mint unique activity IDs.

If you provide `meterProvider` together with `benchmarkMode`, Fedify throws a
`TypeError`. OpenTelemetry metric readers have to be attached when a
`MeterProvider` is constructed, so benchmark mode owns its in-process provider.

If the same application code sometimes runs with benchmark mode and sometimes
runs with your normal OpenTelemetry pipeline, pass your application
`meterProvider` only when benchmark mode is off:

~~~~ typescript twoslash
import type { KvStore } from "@fedify/fedify";
import type { MeterProvider } from "@opentelemetry/api";
// ---cut-start---
declare const process: { env: Record<string, string | undefined> };
const kv = null as unknown as KvStore;
const meterProvider = null as unknown as MeterProvider;
// ---cut-end---
import { createFederation } from "@fedify/fedify";

const benchmarkEnabled = process.env.FEDIFY_BENCHMARK === "1";

const federation = createFederation<void>({
kv,
benchmarkMode: benchmarkEnabled
? { triggerSinks: ["https://sink.example/inbox"] }
: false,
meterProvider: benchmarkEnabled ? undefined : meterProvider,
});
~~~~


Benchmark stats endpoint
------------------------

`GET /.well-known/fedify/bench/stats` returns a versioned JSON snapshot of the
server-side metrics collected by the benchmark mode reader:

~~~~ json
{
"version": 1,
"source": "server",
"generatedAt": "2026-06-02T00:00:00.000Z",
"scopeMetrics": [],
"errors": []
}
~~~~

The `scopeMetrics` field contains serialized OpenTelemetry scope metrics.
Observable queue depth is included when configured queues implement
`MessageQueue.getDepth()`.


Benchmark trigger endpoint
--------------------------

`POST /.well-known/fedify/bench/trigger` asks the target application to call
`Context.sendActivity()` with an explicit sender, recipients, and activity.
This exercises the target's normal outbox and queue path.

The request body has this shape:

~~~~ json
{
"sender": { "identifier": "alice" },
"recipients": [
{
"@context": "https://www.w3.org/ns/activitystreams",
"type": "Service",
"id": "https://sink.example/actors/bob",
"inbox": "https://sink.example/inbox"
}
],
"activity": {
"@context": "https://www.w3.org/ns/activitystreams",
"type": "Create",
"id": "https://example.com/activities/bench-1",
"actor": "https://example.com/users/alice",
"object": {
"type": "Note",
"id": "https://example.com/notes/bench-1",
"content": "benchmark"
}
}
}
~~~~

The `sender` must be either `{ "identifier": string }` or
`{ "username": string }`. Recipients are parsed as ActivityPub actors and must
have `id` and `inbox` properties. The activity is parsed as an ActivityPub
`Activity`.

By default, every recipient inbox must appear in the server-configured
`~FederationBenchmarkOptions.triggerSinks` list. This keeps benchmark traffic
pointed at benchmark sink inboxes and prevents callers from choosing their own
allowlist. To bypass this guard for a controlled run, set
`~FederationBenchmarkOptions.allowUnsafeTriggerRecipients` to `true` in the
application configuration.

A successful trigger returns `202 Accepted`:

~~~~ json
{
"version": 1,
"activityId": "https://example.com/activities/bench-1",
"queueCorrelationId": "https://example.com/activities/bench-1",
"recipientCount": 1,
"inboxCount": 1
}
~~~~

The `queueCorrelationId` is the activity ID preserved on the queued fanout or
outbox work.


Metrics
-------

Benchmark mode uses the same Fedify metrics documented in
[*OpenTelemetry*](./opentelemetry.md), including queue task metrics, queue
depth, HTTP server metrics, and signature verification histograms. The
benchmark endpoints themselves are classified as `fedify.endpoint=benchmark`
in `fedify.http.server.request.*` metrics.
21 changes: 21 additions & 0 deletions docs/manual/federation.md
Original file line number Diff line number Diff line change
Expand Up @@ -275,6 +275,27 @@ Turned off by default.

[SSRF]: https://owasp.org/www-community/attacks/Server_Side_Request_Forgery

### `benchmarkMode`

*This API is available since Fedify 2.3.0.*

Whether to enable cooperative benchmark mode. When enabled, Fedify exposes
benchmark endpoints under `/.well-known/fedify/bench/` and configures an
in-process metrics reader for benchmark clients.

This mode changes only benchmark-target defaults:

- `allowPrivateAddress` defaults to `true`, unless a custom document loader
factory is configured.
- `signatureTimeWindow` defaults to `false`.
- Explicit option values still win.

> [!WARNING]
> Do not enable `benchmarkMode` in production.

See the [*Benchmarking* section](./benchmarking.md) for endpoint details and
safety rules.

### `userAgent`

*This API is available since Fedify 1.3.0.*
Expand Down
21 changes: 20 additions & 1 deletion docs/manual/opentelemetry.md
Original file line number Diff line number Diff line change
Expand Up @@ -370,6 +370,7 @@ Fedify records the following OpenTelemetry metrics:
| `fedify.queue.task.failed` | Counter | `{task}` | Counts queue tasks Fedify abandoned because processing threw. |
| `fedify.queue.task.duration` | Histogram | `ms` | Measures queue task processing duration in Fedify workers. |
| `fedify.queue.task.in_flight` | UpDownCounter | `{task}` | Tracks queue tasks currently in flight in this Fedify process. |
| `fedify.queue.depth` | Gauge | `{message}` | Reports queued, ready, and delayed queue depth when the queue backend supports it. |

### Metric attributes

Expand Down Expand Up @@ -841,6 +842,17 @@ Fedify records the following OpenTelemetry metrics:
Fedify process*, not cross-process totals. Aggregate it across
replicas in your metrics backend.

`fedify.queue.depth`
: `fedify.queue.depth.state` is always present and is one of `queued`,
`ready`, or `delayed`. `fedify.queue.role` is `inbox`, `outbox`,
`fanout`, or `shared`; `shared` means the same queue instance backs more
than one Fedify queue role, and `fedify.queue.roles` lists those roles as a
comma-separated string. `fedify.queue.backend` and
`fedify.queue.native_retrial` follow the same rules as the queue task
metrics. `fedify.federation.instance_id` is an opaque per-Federation
instance identifier that keeps queue depth series distinct when multiple
Federation instances share one [`MeterProvider`].

The `fedify.queue.task.*` metrics describe what Fedify's workers do with
queued messages. They complement the backend-side
[`MessageQueue.getDepth()` API](./mq.md#queue-depth-reporting), which
Expand All @@ -849,6 +861,10 @@ Reading both signals together (task throughput plus backlog depth)
makes it possible to distinguish a small, slow queue from a large, fast
one and to set alerting thresholds for delivery latency under load.

When [`benchmarkMode`](./benchmarking.md) is enabled, Fedify serves a
versioned snapshot of these in-process metrics from
`/.well-known/fedify/bench/stats`.

The `activitypub.inbox.activity`, `activitypub.outbox.activity`, and
`activitypub.fanout.recipients` metrics describe what is happening at
the *activity* level, complementing the per-recipient
Expand Down Expand Up @@ -951,16 +967,19 @@ for ActivityPub:
| `docloader.document_url` | string | The final URL of the fetched document (after following redirects). | `"https://example.com/object/1"` |
| `fedify.actor.identifier` | string | The identifier of the actor. | `"1"` |
| `fedify.endpoint` | string | The bounded endpoint category that classified an inbound HTTP request handled by `Federation.fetch()`. | `"actor"` |
| `fedify.federation.instance_id` | string | Opaque per-Federation instance identifier used to distinguish queue depth series on a shared `MeterProvider`. | `"fedify-1"` |
| `fedify.route.template` | string | The matched URI Template, with parameter names (not values). | `"/users/{identifier}"` |
| `fedify.inbox.recipient` | string | The identifier of the inbox recipient. | `"1"` |
| `fedify.object.type` | string | The URI of the object type. | `"https://www.w3.org/ns/activitystreams#Note"` |
| `fedify.object.values.{parameter}` | string[] | The argument values of the object dispatcher. | `["1", "2"]` |
| `fedify.collection.dispatcher` | string | The collection dispatcher family: `built_in` or `custom`. | `"built_in"` |
| `fedify.collection.cursor` | string | The cursor of the collection. | `"eyJpZCI6IjEiLCJ0eXBlIjoiT3JkZXJlZENvbGxlY3Rpb24ifQ=="` |
| `fedify.collection.items` | number | The number of materialized items in the collection response or page. It can be less than the total items. | `10` |
| `fedify.queue.role` | string | The Fedify queue role for the task: `inbox`, `outbox`, or `fanout`. | `"outbox"` |
| `fedify.queue.role` | string | The Fedify queue role: `inbox`, `outbox`, `fanout`, or `shared` for queue depth rows where one queue backs multiple roles. | `"outbox"` |
| `fedify.queue.backend` | string | The queue implementation's constructor name (best-effort backend identifier). | `"RedisMessageQueue"` |
| `fedify.queue.native_retrial` | boolean | Whether the queue backend declares `nativeRetrial`, meaning Fedify defers retry handling to the backend. | `true` |
| `fedify.queue.depth.state` | string | Queue depth count kind: `queued`, `ready`, or `delayed`. | `"queued"` |
| `fedify.queue.roles` | string | Comma-separated queue roles when one queue instance backs multiple roles. | `"fanout,inbox,outbox"` |
| `fedify.queue.task.attempt` | int | The zero-based attempt number recorded on `fedify.queue.task.enqueued`; non-zero for retry re-enqueues. | `1` |
| `fedify.queue.task.result` | string | The terminal outcome of queue task processing: `completed`, `failed`, or `aborted`. | `"failed"` |
| `http.redirect.url` | string | The redirect URL when a document fetch results in a redirect. | `"https://example.com/new-location"` |
Expand Down
2 changes: 1 addition & 1 deletion packages/fedify/package.json
Original file line number Diff line number Diff line change
Expand Up @@ -147,6 +147,7 @@
"@logtape/logtape": "catalog:",
"@opentelemetry/api": "catalog:",
"@opentelemetry/core": "catalog:",
"@opentelemetry/sdk-metrics": "catalog:",
"@opentelemetry/sdk-trace-base": "catalog:",
"@opentelemetry/semantic-conventions": "catalog:",
"byte-encodings": "catalog:",
Expand All @@ -159,7 +160,6 @@
"devDependencies": {
"@fedify/fixture": "workspace:*",
"@fedify/vocab-tools": "workspace:^",
"@opentelemetry/sdk-metrics": "catalog:",
"@std/assert": "jsr:^0.226.0",
"@std/path": "catalog:",
"@types/node": "^24.2.1",
Expand Down
Loading