Okapi

Reliable message delivery for Kotlin and Java services, using the transactional outbox pattern.

Okapi is a Kotlin/JVM library implementing the transactional outbox pattern. Messages are stored in the database within the same transaction as your business operation, then delivered asynchronously over HTTP or Kafka. This prevents messages from being lost between the database commit and the delivery attempt, without requiring distributed transactions.

Storage: PostgreSQL, MySQL 8+
Transports: HTTP webhooks, Kafka
Frameworks: Spring Boot autoconfiguration, or wire it by hand anywhere on the JVM
Kotlin-first, with a Java-friendly API
Apache-2.0, JDK 21+

Quick start (Spring Boot)

This example assumes an existing Spring Boot application connected to PostgreSQL, with a configured DataSource and PlatformTransactionManager.

1. Add the dependencies. The BOM keeps module versions aligned:

dependencies {
    implementation(platform("com.softwaremill.okapi:okapi-bom:1.0.0"))
    implementation("com.softwaremill.okapi:okapi-core")
    implementation("com.softwaremill.okapi:okapi-postgres")
    implementation("com.softwaremill.okapi:okapi-http")
    implementation("com.softwaremill.okapi:okapi-spring-boot")
    runtimeOnly("org.liquibase:liquibase-core")
}

Liquibase creates the okapi_outbox table on startup — no changelog edits on your side. See Database schema.

2. Provide a deliverer bean. This example uses HTTP. ServiceUrlResolver maps a logical service name to a base URL, so deployment topology stays out of your publishing code:

@Bean
fun httpDeliverer(): HttpMessageDeliverer =
    HttpMessageDeliverer(ServiceUrlResolver { serviceName ->
        when (serviceName) {
            "notification-service" -> "https://notifications.example.com"
            else -> error("Unknown service: $serviceName")
        }
    })

3. Publish inside your transaction. Inject SpringOutboxPublisher and call it right after your business write:

@Service
class OrderService(
    private val orderRepository: OrderRepository,
    private val outboxPublisher: SpringOutboxPublisher,
) {
    @Transactional
    fun placeOrder(order: Order) {
        orderRepository.save(order)

        outboxPublisher.publish(
            OutboxMessage("order.created", order.toJson()),
            httpDeliveryInfo {
                serviceName = "notification-service"
                endpointPath = "/webhooks/orders"
            },
        )
    }
}

The order row and the outbox row now commit together or not at all. Autoconfiguration takes care of scheduling, retries, delivery and cleanup.

SpringOutboxPublisher throws IllegalStateException if you call publish() outside an active read-write transaction. That is deliberate — an outbox write that can't commit atomically with your business data defeats the purpose of the pattern.

Examples

Runnable, self-contained applications live in okapi-examples. Each one is an independent Gradle project that consumes okapi as a published dependency, with its own docker-compose.yml for the databases and brokers it needs.

How it works

publish() writes a PENDING row to okapi_outbox in your transaction.
A background scheduler polls for pending rows (every second by default), claiming only the delivery types configured on that worker with FOR UPDATE SKIP LOCKED so workers do not process the same row concurrently.
Each row goes to the transport matching its delivery type. Success marks it DELIVERED; a retriable failure leaves it PENDING while retry attempts remain; an exhausted retry budget or permanent failure marks it FAILED.
A purger deletes delivered rows after a retention period.

Guarantees and limits

Duplicate delivery is possible. A crash between a successful delivery and the status update means the message may be sent again after restart. If processing a message more than once would cause unwanted effects, make the consumer idempotent. okapi sends your payload, your configured headers, and an x-outbox-id header to deduplicate on — see Deduplicating on x-outbox-id.
Best-effort ordering. Rows are claimed by due time (next_attempt_at, then id) within each delivery type: a new row is due when it is created, a retried one once its backoff has passed, so a failing row does not hold up newer ones. The processor rotates which type it checks first between batches; when that type has fewer rows than the batch size, the remaining capacity goes to the other types in alphabetical order, so under mixed backlog alphabetically earlier types get more of it. Every type is still claimed first at least once every N batches, where N is the number of delivery types the worker handles. Across types, rows are no longer claimed globally oldest first. Parallel delivery and retries can also change arrival order. Strict delivery ordering across types or within a type is not guaranteed.
Different workers can handle different delivery types. Each worker claims only its registered types, using at most two claim queries per batch regardless of the number of types. A custom OutboxStore used for processing must implement RouteAwareOutboxStore; otherwise processing fails before an unsafe claim. The older claimPending(limit) method remains available for callers that need it, but Okapi's processor does not use it. Apply the new index migration before starting updated workers if you manage the schema yourself.
Rolling upgrades need sequencing. Workers running an older Okapi version can still claim rows of a newly introduced delivery type. Upgrade or stop all old processors before publishing that type. An upgrade does not automatically retry rows already marked FAILED by older workers.
Failure classification is the transport's job. Each deliverer decides what is retriable. HTTP: 5xx, 429, 408 and connection errors are retriable; other responses and TLS errors are permanent. Kafka: broker-side retriable exceptions are retried; authorization and configuration errors are not.
On MySQL, scheduler transactions run at READ COMMITTED. The autoconfigured Spring TransactionRunner (with MysqlOutboxStore) and ExposedTransactionRunner (on a MySQL/MariaDB dialect) both use it. At MySQL's default REPEATABLE READ, the claim's gap locks would block every concurrent publish() until delivery finishes. If you supply your own TransactionRunner, an OutboxStore of your own, or a TransactionTemplate with an explicit isolation level, keep it at READ COMMITTED on MySQL. With a JtaTransactionManager, or a JpaTransactionManager without the Hibernate/EclipseLink JpaDialect, okapi cannot set it (startup logs a WARN): configure transaction_isolation = 'READ-COMMITTED' on the server or in the JDBC URL instead. READ COMMITTED requires binlog_format=ROW (the default since MySQL 5.7.7) or MIXED; with STATEMENT, InnoDB rejects the scheduler's writes with ERROR 1665.
Retry budget. okapi.processor.max-retries (default 5) counts retries after the first attempt — six attempts in total before a row becomes FAILED. FAILED is terminal. A retriable failure is retried after an exponential backoff with jitter: okapi.processor.initial-backoff (default 10s) before the first retry, backoff-multiplier (default 2) times longer for each further one, capped at max-backoff (default 10m) and shortened at random by up to backoff-jitter (default 0.2). With the defaults, a message survives roughly five minutes of downstream outage before it becomes FAILED; raise max-retries to cover longer ones (e.g. 10 retries span about 50 minutes). The backoff only applies to rows claimed by 2.0 workers.

Deduplicating on `x-outbox-id`

Every delivery carries an x-outbox-id header holding the outbox entry's UUID — the same value publish() returns. It is set by all transports (the name is the constant OutboxHeaders.OUTBOX_ID in okapi-core, so consumers can reference it without depending on a transport module), and the value is stable across retries: if okapi delivers the same entry twice, both copies carry the same id. A consumer that records ids it has already processed can therefore drop the repeat.

// Kafka consumer
val outboxId = record.headers().lastHeader(OutboxHeaders.OUTBOX_ID)?.let { String(it.value()) }
if (outboxId != null && !seenIds.add(outboxId)) return  // already processed, skip

// HTTP receiver (Spring MVC)
@PostMapping("/webhook")
fun receive(@RequestHeader("x-outbox-id") outboxId: String, @RequestBody payload: String) {
    if (!seenIds.add(outboxId)) return  // already processed, skip
    // ...
}

Notice that:

okapi sets the header last, so it overrides any x-outbox-id you set yourself in DeliveryInfo. Over HTTP your value is replaced outright. Kafka headers are multi-valued, so your value is still present in the record, but okapi's is the one appended last — which is why consumers must read it with lastHeader(...) (or take the last of headers(...)) rather than the first match. Pick a different header name if you need to pass an identifier of your own.
It does not deduplicate at the publish() level. The id identifies an outbox entry, not a business event. Calling publish() twice for the same event creates two entries with two different ids, and a consumer deduplicating on x-outbox-id will process both. Guarding against that is the publisher's job — deduplicate on a business key in the payload, or make the publish itself idempotent.

Configuration

In a typical single-DataSource application, all properties are optional. Multi-DataSource setups may require explicit qualifiers, as described below.

Processor

Property	Default	Description
`okapi.processor.enabled`	`true`	Set `false` to disable delivery entirely (e.g. on instances that only publish).
`okapi.processor.interval`	`1s`	How often the scheduler polls for pending entries.
`okapi.processor.batch-size`	`10`	Maximum entries claimed per worker per tick.
`okapi.processor.max-retries`	`5`	Retries after the initial attempt before an entry becomes `FAILED`.
`okapi.processor.initial-backoff`	`10s`	Delay before the first retry of a retriable failure.
`okapi.processor.backoff-multiplier`	`2.0`	Factor applied to the delay for each further retry (>= 1).
`okapi.processor.max-backoff`	`10m`	Upper bound on the delay between retries.
`okapi.processor.backoff-jitter`	`0.2`	Fraction (0 to 1) by which each delay is shortened at random, so entries that failed together do not retry together.
`okapi.processor.concurrency`	`1`	Parallel workers per tick, each claiming its own batch. Tune based on database capacity and delivery latency; see Performance.
`okapi.processor.transport-dispatch`	`parallel`	How a batch spanning several `MessageDeliverer` beans is dispatched: `parallel` (all groups but one go to a virtual thread each — N transports use N-1 extra threads — while the last runs inline on the calling thread, so the batch costs ~`max(Tᵢ)`) or `sequential` (one group after another, all on the calling thread). Use `sequential` only if a custom deliverer depends on caller thread context — MDC, security context, or a transaction-bound resource such as a connection obtained via `DataSourceUtils`, since a group on a virtual thread does not inherit the caller's transaction. Batches with a single delivery type always run inline, regardless of this setting.

Purger

Delivered entries are deleted on a schedule so they do not accumulate. FAILED entries are never purged and must be managed separately.

Property	Default	Description
`okapi.purger.enabled`	`true`	Set `false` to manage retention yourself (partitioning, external cron).
`okapi.purger.retention`	`7d`	How long delivered entries are kept.
`okapi.purger.interval`	`1h`	How often the purger runs.
`okapi.purger.batch-size`	`100`	Rows deleted per batch; each batch is its own transaction.

Schema

Property	Default	Description
`okapi.liquibase.enabled`	`true`	Set `false` if your application manages the outbox schema itself.
`okapi.liquibase.changelog-table`	`okapi_databasechangelog`	Liquibase tracking table for okapi's migrations.
`okapi.liquibase.changelog-lock-table`	`okapi_databasechangeloglock`	Liquibase lock table for okapi's migrations.

Data source and transactions

Property	Default	Description
`okapi.datasource-qualifier`	unset	Bean name of the outbox `DataSource`. When unset, the single or `@Primary` `DataSource` is used.
`okapi.transaction-manager-qualifier`	unset	Bean name of the outbox `PlatformTransactionManager`. When unset, it is resolved automatically. Set explicitly in multi-PTM setups — see Transactions.

Metrics

Property	Default	Description
`okapi.metrics.enabled`	`true`	Set `false` to disable okapi's Micrometer autoconfiguration. Logs a startup warning when disabled.
`okapi.metrics.refresh-interval`	`15s`	How often gauges poll the store. Each refresh runs two queries, wrapped in a single read-only transaction when a transaction manager is available.

Storage and transports

MySQL

Replace okapi-postgres with okapi-mysql; the okapi publishing API stays the same. Add rewriteBatchedStatements=true to your JDBC URL (jdbc:mysql://host:3306/db?rewriteBatchedStatements=true) so Connector/J can send a JDBC batch as a single multi-statement request. okapi cannot set this for you, since it doesn't own your DataSource.

Kafka

Provide a KafkaMessageDeliverer bean with your own producer, and publish with the matching builder:

@Bean
fun kafkaDeliverer(producer: KafkaProducer<String, String>): KafkaMessageDeliverer =
    KafkaMessageDeliverer(producer)

outboxPublisher.publish(
    OutboxMessage("order.created", order.toJson()),
    kafkaDeliveryInfo { topic = "order-events" },
)

You can register HTTP and Kafka deliverers together; okapi routes each entry by delivery type. You provide and configure the Kafka producer.

Without Spring Boot

okapi-core has no framework dependencies, so any JVM service can use it, from a Ktor app to a background worker. You assemble the pieces yourself — create the store, the deliverer, and the processor, start an OutboxScheduler, and hand it a TransactionRunner (a one-method interface wrapping a block in whatever transaction mechanism you already use). Copy the database-specific SQL linked in Database schema into your application's migrations.

Exposed

okapi-exposed bridges okapi's transaction and connection abstractions to Exposed: ExposedTransactionRunner, ExposedTransactionContextValidator, and ExposedConnectionProvider. Useful for Ktor and standalone Kotlin services.

Database schema

okapi ships Liquibase changelogs that create its table and indexes:

classpath:com/softwaremill/okapi/db/postgres/changelog.xml (from okapi-postgres)
classpath:com/softwaremill/okapi/db/mysql/changelog.xml (from okapi-mysql)

With okapi-spring-boot and Liquibase on the classpath, these run automatically against the configured DataSource at startup, tracked in dedicated Liquibase tables by default to avoid conflicts with the application's migration history.

If you use another migration tool, apply every SQL migration for your database, in order:

PostgreSQL: 001 table and initial indexes, 002 route-aware claim index, 003 TIMESTAMPTZ timestamp columns, 004 retry backoff column and claim index (its second changeset must run outside a transaction)
MySQL: 001 table and initial indexes, 002 route-aware claim index, 003 DATETIME(6) timestamp columns, 004 retry backoff column and claim index

For an existing installation, apply every new migration (up to 004) before starting updated workers. MySQL claims explicitly select the claim index from 004 and fail if it is missing. PostgreSQL's 002 and the index changeset of 004 use CREATE INDEX CONCURRENTLY, so run them outside a transaction when using another migration tool. The bundled Liquibase changelog handles this automatically. CREATE INDEX CONCURRENTLY waits for every transaction in the database that started before it, not only those on okapi_outbox, so long-running transactions (including old workers delivering a claimed batch) delay it, and with Liquibase they delay application startup while the changelog lock is held. Run it when no long transactions are open, or apply it ahead of the rollout.

PostgreSQL's 003 converts the timestamp columns from TIMESTAMP (the okapi JVM's local wall-clock time) to TIMESTAMPTZ. It reads existing values in the session TimeZone; pgjdbc sets that to the JVM default zone, so the bundled Liquibase changelog converts them correctly when it runs inside the application. When applying it by hand, run it inside a transaction (psql -1) after SET TIME ZONE '<zone your okapi JVMs run in>' in the same session. It takes an ACCESS EXCLUSIVE lock: it first waits (up to 30 s, then fails and retries on the next start) for open transactions on okapi_outbox, queuing every other statement on the table meanwhile, and then, unless the session zone is UTC (PostgreSQL 12+), rewrites the table while reads and writes wait. 1.x instances keep working after it, apart from one failed poll per pooled connection. See UPGRADING.md for when the in-app conversion uses the wrong zone. See UPGRADING.md.

MySQL's 003 converts the timestamp columns from TIMESTAMP(6), which ends in January 2038, to DATETIME(6) holding UTC. Stop every 1.x instance, publishers included, before it runs: 1.x writes these columns in the JVM zone. It copies the table; it first waits (up to 30 s, then fails and retries on the next start) for open transactions on okapi_outbox to finish, queuing every other statement on the table meanwhile, and then blocks writes, including publish(), until the copy finishes. See UPGRADING.md, including when rows written by 1.x can read back shifted.

okapi stores messages in the fixed okapi_outbox table. When the built-in Liquibase integration is used, it also uses two dedicated migration tracking tables:

Table	Purpose
`okapi_outbox`	Outbox entries. Name is fixed.
`okapi_databasechangelog`	Liquibase history for okapi's migrations (configurable; Liquibase integration only).
`okapi_databasechangeloglock`	Liquibase lock for okapi's migrations (configurable; Liquibase integration only).

Transactions

Each processor worker runs its claim, delivery and state update inside one transaction, which keeps FOR UPDATE SKIP LOCKED active until delivery state is saved. Each purge batch also runs in its own transaction. With a single DataSource and PlatformTransactionManager, no additional transaction configuration is needed.

Multiple data sources or transaction managers? Set both okapi.datasource-qualifier and okapi.transaction-manager-qualifier to the beans used for the outbox. okapi fails fast when it detects a mismatch. If the selected transaction manager does not expose its DataSource, okapi cannot verify the pairing and logs a warning instead.

Wiring schedulers by hand? TransactionRunner is a required constructor parameter, with no default:

OutboxScheduler(
    outboxProcessor = processor,
    transactionRunner = transactionRunner,
    config = OutboxSchedulerConfig(...),
)

Observability

For Spring Boot metrics with Prometheus, add:

implementation("com.softwaremill.okapi:okapi-micrometer")
implementation("org.springframework.boot:spring-boot-starter-actuator")
runtimeOnly("io.micrometer:micrometer-registry-prometheus")

Expose the Prometheus endpoint:

management:
  endpoints:
    web:
      exposure:
        include: health,prometheus

Metrics are then available at /actuator/prometheus.

Metric	Type	Description
`okapi.entries.delivered`	Counter	Successfully delivered entries
`okapi.entries.retry.scheduled`	Counter	Failed attempts rescheduled for retry
`okapi.entries.failed`	Counter	Permanently failed entries
`okapi.batch.duration`	Timer	Processing time per batch; under the scheduler from the claim until the transaction has committed (not including listener callbacks)
`okapi.entries.count`	Gauge	Current entry count (tag: `status`)
`okapi.entries.lag.seconds`	Gauge	Age of the oldest entry (tag: `status`)

Aggregating across instances. Counters and timers are per-instance, so sum them:

sum by (job) (rate(okapi_entries_delivered_total[5m]))

Gauges represent shared database state and are emitted by every instance. Do not sum them across instances; aggregate by Prometheus job and status:

max by (job, status) (okapi_entries_count)

Without Spring Boot, construct MicrometerOutboxListener and MicrometerOutboxMetrics with your MeterRegistry. Refresh gauges with OutboxMetricsRefresher or your own scheduler.

For custom reactions to delivery events, implement OutboxProcessorListener. OutboxProcessor takes a single listener; combine several with a composite of your own. The scheduler calls the listener after the processing transaction has committed, outside of it, and not for a batch whose commit failed, so every event describes persisted state and counters do not over-count when a commit fails. Counters therefore count committed status changes, not delivery attempts: a send whose transaction then rolled back (a failed commit, or a block Exposed retried) is not counted, though the message may have reached the consumer. If you drive OutboxProcessor yourself, call processBatch inside your transaction and publishEvents after it commits; processNext does both at once, so inside a transaction its events precede the commit.

Modules

The runtime modules build on okapi-core; okapi-bom only aligns their versions. Pick a storage module, one or more transports, and a framework adapter if you want one.

From Reliable Message Delivery: the Transactional Outbox Pattern With Okapi.

Module	Purpose
`okapi-core`	Abstractions, processing loop, scheduling, retry policy. No framework dependencies.
`okapi-postgres`	PostgreSQL storage via plain JDBC (`FOR UPDATE SKIP LOCKED`)
`okapi-mysql`	MySQL 8+ storage via plain JDBC
`okapi-http`	HTTP webhook delivery (JDK `HttpClient`)
`okapi-kafka`	Kafka topic publishing
`okapi-spring-boot`	Spring Boot autoconfiguration — selects a store module and wires registered deliverers and metrics
`okapi-exposed`	Exposed ORM integration for transactions and connections
`okapi-micrometer`	Micrometer counters, timers and gauges
`okapi-bom`	Version alignment for all of the above

Compatibility

Dependency	Supported	Notes
Java	21+	Required
Spring Boot	3.5.x, 4.0.x	`okapi-spring-boot`
Kafka Clients	3.9.x, 4.x	Included transitively by `okapi-kafka`; you can override the version in your build.
Exposed	1.x	`okapi-exposed`

okapi-spring-boot does not bring Spring Boot transitively; your application controls the Spring Boot version.

The storage modules use plain JDBC. With Spring Boot, they participate in the transaction selected through a PlatformTransactionManager; okapi-exposed provides adapters for Exposed-managed transactions.

Performance

Performance depends on the selected transport, database, batch size, concurrency and downstream latency. See benchmarks/ for methodology and measured results.

Building

./gradlew build                  # Build and test all modules (Docker required — Testcontainers)
./gradlew ktlintFormat           # Format code — mandatory before committing
./gradlew :okapi-benchmarks:jmh  # Run JMH benchmarks (~30 min, see benchmarks/README.md)

Contributing

All suggestions are welcome. Take a look at the open issues and pick one, or report your own.

If you are unsure why or how something works, ask on Discourse or open an issue. That usually means the documentation or the code is unclear, and fixing it helps everyone.

When your PR is ready, see our guide to preparing a good PR.

Commercial support

okapi is built and maintained by SoftwareMill. We offer commercial development services — get in touch to learn more.

Navigation Menu

Repository navigation

About

Repository files navigation

Okapi

Quick start (Spring Boot)

Examples

How it works

Guarantees and limits

Deduplicating on `x-outbox-id`

Configuration

Processor

Purger

Schema

Data source and transactions

Metrics

Storage and transports

MySQL

Kafka

Without Spring Boot

Exposed

Database schema

Transactions

Observability

Modules

Compatibility

Performance

Building

Contributing

Commercial support

License

About

Releases

Packages

Used by

Contributors

Languages

Name		Name	Last commit message	Last commit date
.github		.github
benchmarks		benchmarks
buildSrc		buildSrc
docs/images		docs/images
gradle		gradle
okapi-benchmarks		okapi-benchmarks
okapi-bom		okapi-bom
okapi-core		okapi-core
okapi-exposed		okapi-exposed
okapi-http		okapi-http
okapi-integration-tests		okapi-integration-tests
okapi-kafka		okapi-kafka
okapi-micrometer		okapi-micrometer
okapi-mysql		okapi-mysql
okapi-postgres		okapi-postgres
okapi-spring-boot		okapi-spring-boot
.editorconfig		.editorconfig
.gitignore		.gitignore
CHANGELOG.md		CHANGELOG.md
LICENSE		LICENSE
README.md		README.md
UPGRADING.md		UPGRADING.md
build.gradle.kts		build.gradle.kts
gradle.properties		gradle.properties
gradlew		gradlew
gradlew.bat		gradlew.bat
settings.gradle.kts		settings.gradle.kts

Latest commit

History

Folders and files

Repository files navigation

Okapi

Quick start (Spring Boot)

Examples

How it works

Guarantees and limits

Deduplicating on x-outbox-id

Configuration

Processor

Purger

Schema

Data source and transactions

Metrics

Storage and transports

MySQL

Kafka

Without Spring Boot

Exposed

Database schema

Transactions

Observability

Modules

Compatibility

Performance

Building

Contributing

Commercial support

License

About

Topics

Resources

Stars

Watchers

Forks

Releases

Packages

Used by

Contributors

Languages

Deduplicating on `x-outbox-id`