sexp-store

sexp-store persists portable Common Lisp data as readable top-level forms. It provides atomic single-form snapshots and crash-tolerant append logs.

Snapshots

(sexp-store:snapshot-write #P"state.sexp" '(:state :version 1))

(multiple-value-bind (form sole-form-p)
    (sexp-store:snapshot-read #P"state.sexp")
  (values form sole-form-p))

Snapshot publication writes a complete temporary file beside the target and renames it over the target. The default Unix mode is 0600 on supported POSIX environments. Pass :mode nil when permissions are managed separately.

Append logs

(sexp-store:log-append
 #P"events.sexp"
 '(:event :id 1)
 :initial-forms '((:events :version 1)))

• log-read returns the complete forms and a second value for an incomplete final form
• log-append repairs such a tail atomically before publishing the new form
• A malformed complete form signals store-error
• log-map visits one complete form at a time and returns a file position that a later call may use as :start-position

Applications that have already verified the tail may pass :repair-tail-p nil for constant-time appends to a large log. Request repair again after loading the log following an interrupted write.

All reads bind *read-eval* to nil. Logs are for one serialized writer and any number of readers.

Versioned records

A record is one form shaped (TAG :version VERSION . PROPERTIES). Field descriptions replace hand-rolled validators:

(sexp-store:snapshot-write
 #P"job.sexp"
 (sexp-store:make-record :job 2 :name "sync" :retries 3))

(sexp-store:snapshot-read-record
 #P"job.sexp"
 :tag :job
 :versions '(1 2)
 :fields (list (list :indicator :name
                     :validate (lambda (value)
                                 (and (stringp value)
                                      (plusp (length value))))
                     :required t)
               (list :indicator :retries
                     :validate #'integerp
                     :required '(2))))

• :required lists the versions that must carry the property, or t for all
• :validate is applied to present values only
• :allow-other-keys nil rejects properties beyond :version and the described fields
• record-check validates an in-memory form and returns a reason on failure
• record-version, record-property, and record-property-present-p read validated forms without getf pitfalls on malformed input

Transactions

Construct a log-store or snapshot-store with explicit :pathname, :lock-pathname, an :initial-state factory and a :validator predicate. Use store-read for validated fresh state, or store-transact for one locked read/modify/publish operation. Locks exclude local threads and cooperating processes through ls-flock.

For a log-store, supply :header, :header-validator, and a :reducer of state and record. An optional :finalizer converts folded state to its public value. For a snapshot-store, supply :decoder and :encoder callbacks. The updater receives fresh public state and returns changes, a result, and whether to write. Log changes are a list of records; snapshot changes are replacement state. A log batch is validated and published in one operation, including removal of an incomplete final form. Existing empty or malformed files are errors, rather than missing stores.

(sexp-store:store-transact
 store
 (lambda (current)
   (declare (ignore current))
   (values (list '(:item :version 1 :id "a" :value 42)) :updated t))
 :publish (lambda (committed) (setf cached-state committed)))

The optional :publish callback runs after successful persistence. Keep working state private until then. A publication-callback failure occurs after commit and cannot roll back the file. Return values are the updater result and committed public state. A read-only transaction also returns fresh state.

record-finite-p validates acyclic portable data while allowing shared substructure. record-shape-p additionally requires a keyword-tagged property record. Stores reject duplicate keys by default; :duplicate-keys :first enables explicit first-value compatibility. record-check accepts :allow-duplicate-keys t for the same policy. Domain callbacks own field and version validation.

Segmented logs

Use segment-map for one header-prefixed segment and segments-map for an ordered list of physical paths. Supply :header-function, which receives a pathname and header and returns the positive start sequence and whether an empty segment is allowed. Supply :record-sequence to extract each record’s sequence. Optional :validate-record and :validate-first-record callbacks express the record schema and checkpoint policy.

Replay requires contiguous sequences within and between segments. Torn final forms are ignored, including in sealed segments; complete malformed forms are errors. segment-map returns byte position, tail flag, record count, start sequence and next sequence. segments-map returns active tail flag, total count and next sequence. Callback conditions propagate unchanged.

segment-publish atomically creates a header and first record. Supply :lock-pathname or hold an external writer lease. An optional zero-argument :occupied-p predicate includes related product storage in the occupancy check. A preexisting segment is a conflict, even when identical. If publication signals after creating the file, exact readback permits recovery. Adopt active state only after return. log-repair-tail removes only an incomplete final form under the writer lock and returns whether replacement was needed.

Create-only publication uses an atomic hard link on the supported SBCL POSIX backend, rejecting any occupied target entry, including dangling symlinks. Replacements use same-directory rename. Operations flush Lisp output and support process-crash recovery; power-loss persistence through filesystem synchronization barriers is outside this contract.

Derived snapshots

Use revision-write before authoritative source mutations under the writer lock. Hold that same lock across cache reconstruction and publication. Otherwise a reader can stamp old source data with the writer’s new revision while the writer is between invalidation and mutation. revision-read returns zero for absent or malformed revisions. Supply the product’s tag and optional version to both calls.

Use sidecar-rebuild with :lock-pathname naming the source writer lock and a zero-argument :build callback that reads source data and its token under that lock. Supply :encode, :source-token and :value-token as for sidecar-write. The lock covers reconstruction through publication. A nil build result skips publication; lock, build and publication errors propagate.

sidecar-read accepts :decode, a zero-argument :source-token callback and a :value-token extractor. Tokens are compared with equal before and after reading. Malformed, stale or concurrently invalidated cache data returns nil. Optional :validate adds domain constraints. Under the shared-lock protocol, cache hits need no lock. sidecar-write accepts a prebuilt value with :encode and the same token callbacks, declining stale publication. Hold the source lock from before constructing that value until publication completes, or build from an immutable committed snapshot with its matching token.

file-revision returns pathname, byte size and write date. Combine these with an explicit revision and source locking, since size and date cannot distinguish same-size edits within one clock tick.

Tests

(asdf:test-system :sexp-store)

Part of the Lambda Symbolics library shelf.