Skip to content

Conversation

@cb1kenobi
Copy link
Contributor

@cb1kenobi cb1kenobi commented Sep 18, 2025

Adding support for the transaction log store.

Fixes #217.

@github-actions
Copy link

github-actions bot commented Sep 18, 2025

📊 Benchmark Results

encoding.bench.ts

Key encoding > ordered-binary keys - strings (100 records)

Implementation Rank Operations/sec Mean (ms) Min (ms) Max (ms) RME (%) Samples
🥇 lmdb 1 25.01K ops/sec 39.98 35.51 452.973 1.08 12,505
🥈 rocksdb 2 4.87K ops/sec 205.237 189.762 1,945.808 1.10 2,437

Key encoding > ordered-binary keys - numbers (100 records)

Implementation Rank Operations/sec Mean (ms) Min (ms) Max (ms) RME (%) Samples
🥇 lmdb 1 27.20K ops/sec 36.77 32.77 367.693 0.951 13,600
🥈 rocksdb 2 5.09K ops/sec 196.355 185.307 853.663 0.452 2,547

Key encoding > ordered-binary keys - mixed types (100 records)

Implementation Rank Operations/sec Mean (ms) Min (ms) Max (ms) RME (%) Samples
🥇 lmdb 1 24.32K ops/sec 41.11 36.56 414.666 1.02 12,162
🥈 rocksdb 2 4.97K ops/sec 201.347 188.215 940.782 0.540 2,484

Value encoding > msgpack values - strings (100 records)

Implementation Rank Operations/sec Mean (ms) Min (ms) Max (ms) RME (%) Samples
🥇 lmdb 1 17.29K ops/sec 57.84 54.62 443.085 0.559 8,644
🥈 rocksdb 2 4.47K ops/sec 223.817 210.625 972.93 0.468 2,234

Value encoding > msgpack values - numbers (100 records)

Implementation Rank Operations/sec Mean (ms) Min (ms) Max (ms) RME (%) Samples
🥇 lmdb 1 25.83K ops/sec 38.72 34.65 361.345 0.935 12,915
🥈 rocksdb 2 4.72K ops/sec 211.786 191.793 3,227.254 1.38 2,362

Value encoding > msgpack values - arrays (100 records)

Implementation Rank Operations/sec Mean (ms) Min (ms) Max (ms) RME (%) Samples
🥇 lmdb 1 9.20K ops/sec 108.673 95.92 1,160.386 1.61 4,601
🥈 rocksdb 2 3.59K ops/sec 278.636 258.151 1,162.479 0.658 1,795

Value encoding > msgpack values - small objects (100 records)

Implementation Rank Operations/sec Mean (ms) Min (ms) Max (ms) RME (%) Samples
🥇 lmdb 1 5.90K ops/sec 169.618 151.958 1,076.555 1.75 2,948
🥈 rocksdb 2 2.73K ops/sec 366.896 343.753 959.386 0.652 1,363

Value encoding > msgpack values - large objects (100 records)

Implementation Rank Operations/sec Mean (ms) Min (ms) Max (ms) RME (%) Samples
🥇 lmdb 1 648.47 ops/sec 1,542.083 1,413.22 6,725.547 3.61 325
🥈 rocksdb 2 643.55 ops/sec 1,553.874 1,405.851 3,567.458 2.03 322

get-sync.bench.ts

getSync() > random keys - small key size (100 records)

Implementation Rank Operations/sec Mean (ms) Min (ms) Max (ms) RME (%) Samples
🥇 lmdb 1 20.74K ops/sec 48.22 41.70 3,386.326 1.77 10,370
🥈 rocksdb 2 4.64K ops/sec 215.545 186.735 11,679.395 6.48 2,320

get.bench.ts

get() > rocksdb - random vs sequential keys (100 records)

Implementation Rank Operations/sec Mean (ms) Min (ms) Max (ms) RME (%) Samples
🥇 sequential 1 4.27K ops/sec 234.161 222.73 896.669 0.531 2,136
🥈 random 2 4.09K ops/sec 244.261 228.997 1,121.343 0.838 2,048

get() > random keys - max 1978 lmdb key size (100 records)

Implementation Rank Operations/sec Mean (ms) Min (ms) Max (ms) RME (%) Samples
🥇 rocksdb 1 3.19K ops/sec 313.034 278.886 2,391.898 1.98 1,598

get() > rocksdb - async vs sync

Implementation Rank Operations/sec Mean (ms) Min (ms) Max (ms) RME (%) Samples
🥇 sync 1 4.33K ops/sec 231.059 210.658 3,578.13 1.97 2,164
🥈 async 2 3.69K ops/sec 270.693 252.801 1,364.966 0.744 1,848

put-sync.bench.ts

putSync() > random keys - insert - small key size (100 records)

Implementation Rank Operations/sec Mean (ms) Min (ms) Max (ms) RME (%) Samples
🥇 rocksdb 1 2.38K ops/sec 419.474 399.446 599.339 0.186 1,192
🥈 lmdb 2 4.16 ops/sec 240,394.866 232,713.336 258,826.37 2.77 10.00

putSync() > random keys - update - small key size (100 records)

Implementation Rank Operations/sec Mean (ms) Min (ms) Max (ms) RME (%) Samples
🥇 rocksdb 1 2.39K ops/sec 418.793 391.229 600.566 0.175 1,194
🥈 lmdb 2 4.20 ops/sec 237,842.519 227,383.654 256,644.273 2.30 10.00

putSync() > random keys - insert - max 1978 lmdb key size (100 records)

Implementation Rank Operations/sec Mean (ms) Min (ms) Max (ms) RME (%) Samples
🥇 rocksdb 1 1.02K ops/sec 978.326 874.229 1,281.017 0.476 512
🥈 lmdb 2 3.95 ops/sec 252,844.753 240,549.96 275,323.093 2.56 10.00

putSync() > random keys - update - max 1978 lmdb key size (100 records)

Implementation Rank Operations/sec Mean (ms) Min (ms) Max (ms) RME (%) Samples
🥇 rocksdb 1 1.02K ops/sec 976.195 868.193 1,605.613 0.863 513
🥈 lmdb 2 3.92 ops/sec 255,335.987 249,175.14 274,551.116 2.57 10.00

putSync() > sequential keys - insert (100 records)

Implementation Rank Operations/sec Mean (ms) Min (ms) Max (ms) RME (%) Samples
🥇 rocksdb 1 2.40K ops/sec 416.795 386.007 577.319 0.193 1,200
🥈 lmdb 2 4.09 ops/sec 244,209.064 237,304.019 265,022.048 2.42 10.00

putSync() > put 100KB value (100 records)

Implementation Rank Operations/sec Mean (ms) Min (ms) Max (ms) RME (%) Samples
🥇 rocksdb 1 49.02 ops/sec 20,397.818 17,924.832 24,622.96 3.70 25.00
🥈 lmdb 2 2.68 ops/sec 373,590.853 365,534.185 395,292.189 1.88 10.00

putSync() > put 1MB value (100 records)

Implementation Rank Operations/sec Mean (ms) Min (ms) Max (ms) RME (%) Samples
🥇 rocksdb 1 5.24 ops/sec 190,758.567 183,563.113 204,675.734 2.29 10.00
🥈 lmdb 2 1.94 ops/sec 516,411.057 415,543.058 595,843.281 7.93 10.00

putSync() > get 10MB value (100 records)

Implementation Rank Operations/sec Mean (ms) Min (ms) Max (ms) RME (%) Samples
🥇 lmdb 1 0.24 ops/sec 4,198,095.819 3,296,360.847 5,022,380.181 9.57 10.00
🥈 rocksdb 2 0.14 ops/sec 7,269,942.892 5,958,958.856 8,881,560.658 11.39 10.00

put.bench.ts

put > small dataset (100 records)

Implementation Rank Operations/sec Mean (ms) Min (ms) Max (ms) RME (%) Samples
🥇 rocksdb 1 2.07K ops/sec 482.604 450.295 800.082 0.378 1,037
🥈 lmdb 2 3.32 ops/sec 301,156.251 285,454.004 328,746.459 3.06 10.00

put > async vs sync overhead

Implementation Rank Operations/sec Mean (ms) Min (ms) Max (ms) RME (%) Samples
🥇 sync 1 2.24K ops/sec 446.86 421.006 537.473 0.190 1,119
🥈 async 2 2.06K ops/sec 486.373 457.819 1,641.692 0.553 1,029

ranges.bench.ts

getRange() > small range (100 records, 50 range)

Implementation Rank Operations/sec Mean (ms) Min (ms) Max (ms) RME (%) Samples
🥇 lmdb 1 23.40K ops/sec 42.73 37.32 1,153.877 1.22 11,702
🥈 rocksdb 2 2.89K ops/sec 345.741 297.073 2,566.425 3.27 1,447

getRange() > full scan vs range scan

Implementation Rank Operations/sec Mean (ms) Min (ms) Max (ms) RME (%) Samples
🥇 lmdb range scan 1 20.08K ops/sec 49.81 42.82 1,263.555 1.32 10,039
🥈 lmdb full scan 2 11.29K ops/sec 88.60 75.86 865.487 1.46 5,644
🥉 rocksdb range scan 3 3.05K ops/sec 327.463 282.913 2,428.881 2.54 1,527
rocksdb full scan 4 1.66K ops/sec 602.959 525.347 4,243.486 3.27 830

getKeys() > keys only (100 records, 50 range)

Implementation Rank Operations/sec Mean (ms) Min (ms) Max (ms) RME (%) Samples
🥇 lmdb 1 53.19K ops/sec 18.80 16.31 7,837.979 3.61 26,597
🥈 rocksdb 2 4.76K ops/sec 210.096 184.105 1,085.823 1.09 2,380

Reverse iteration > reverse range (100 records, 50 range)

Implementation Rank Operations/sec Mean (ms) Min (ms) Max (ms) RME (%) Samples
🥇 lmdb 1 542.19K ops/sec 1.84 1.21 6,217.151 6.55 271,096
🥈 rocksdb 2 2.99K ops/sec 334.963 301.016 1,422.961 1.59 1,493

Reverse iteration > rocksdb - reverse vs forward

Implementation Rank Operations/sec Mean (ms) Min (ms) Max (ms) RME (%) Samples
🥇 forward 1 3.19K ops/sec 313.186 280.435 1,690.917 1.85 1,597
🥈 reverse 2 2.94K ops/sec 340.012 309.944 2,045.028 1.80 1,471

Range query patterns > prefix scan performance

Implementation Rank Operations/sec Mean (ms) Min (ms) Max (ms) RME (%) Samples
🥇 lmdb 1 30.60K ops/sec 32.67 28.71 1,092.238 1.17 15,303
🥈 rocksdb 2 3.45K ops/sec 289.984 263.341 1,721.494 1.66 1,725

Sparse data patterns > sparse - range over gaps

Implementation Rank Operations/sec Mean (ms) Min (ms) Max (ms) RME (%) Samples
🥇 lmdb 1 58.80K ops/sec 17.01 14.69 1,595.353 1.45 29,402
🥈 rocksdb 2 6.26K ops/sec 159.775 144.674 1,281.157 1.11 3,130

Sparse data patterns > sparse - prefix with gaps

Implementation Rank Operations/sec Mean (ms) Min (ms) Max (ms) RME (%) Samples
🥇 lmdb 1 120.48K ops/sec 8.30 6.95 3,238.826 2.64 60,241
🥈 rocksdb 2 12.38K ops/sec 80.78 70.84 3,296.165 1.53 6,190

remove-sync.bench.ts

removeSync() > random keys - small key size (100 records)

Implementation Rank Operations/sec Mean (ms) Min (ms) Max (ms) RME (%) Samples
🥇 lmdb 1 13.16K ops/sec 75.98 69.39 395.644 0.461 6,581
🥈 rocksdb 2 2.69K ops/sec 372.229 353.804 523.494 0.185 1,344

removeSync() > sequential keys - small key size (100 records)

Implementation Rank Operations/sec Mean (ms) Min (ms) Max (ms) RME (%) Samples
🥇 lmdb 1 12.67K ops/sec 78.94 67.60 708.767 1.36 6,334
🥈 rocksdb 2 2.69K ops/sec 371.456 337.512 616.601 0.213 1,347

removeSync() > rocksdb - random vs sequential keys (100 records)

Implementation Rank Operations/sec Mean (ms) Min (ms) Max (ms) RME (%) Samples
🥇 random 1 2.76K ops/sec 362.378 332.58 600.357 0.194 1,380
🥈 sequential 2 2.76K ops/sec 362.899 342.265 530.113 0.161 1,378

removeSync() > random keys - max 1978 lmdb key size (100 records)

Implementation Rank Operations/sec Mean (ms) Min (ms) Max (ms) RME (%) Samples
🥇 lmdb 1 9.66K ops/sec 103.554 88.79 1,616.251 2.59 4,829
🥈 rocksdb 2 1.11K ops/sec 901.582 797.359 1,099.287 0.389 555

removeSync() > random access pattern (100 records)

Implementation Rank Operations/sec Mean (ms) Min (ms) Max (ms) RME (%) Samples
🥇 lmdb 1 11.72K ops/sec 85.33 75.61 586.529 1.28 5,860
🥈 rocksdb 2 2.73K ops/sec 366.72 351.957 523.422 0.187 1,364

removeSync() > non-existent keys (100 records)

Implementation Rank Operations/sec Mean (ms) Min (ms) Max (ms) RME (%) Samples
🥇 lmdb 1 12.78K ops/sec 78.24 74.60 325.331 0.275 6,391
🥈 rocksdb 2 2.73K ops/sec 365.791 349.499 614.086 0.337 1,367

transaction-sync.bench.ts

transaction sync > optimistic > simple put operations (100 records)

Implementation Rank Operations/sec Mean (ms) Min (ms) Max (ms) RME (%) Samples
🥇 rocksdb 1 582.24 ops/sec 1,717.512 1,610.314 4,332.538 1.61 292
🥈 lmdb 2 4.21 ops/sec 237,549.219 233,336.091 259,589.562 2.37 10.00

transaction sync > optimistic > batch operations (100 records)

Implementation Rank Operations/sec Mean (ms) Min (ms) Max (ms) RME (%) Samples
🥇 rocksdb 1 2.92K ops/sec 342.125 312.162 1,917.91 0.651 1,462
🥈 lmdb 2 343.60 ops/sec 2,910.372 2,102.534 7,995.192 4.02 172

transaction sync > optimistic > read-write operations (100 records)

Implementation Rank Operations/sec Mean (ms) Min (ms) Max (ms) RME (%) Samples
🥇 rocksdb 1 339.63 ops/sec 2,944.386 2,638.975 5,696.946 1.99 170
🥈 lmdb 2 4.19 ops/sec 238,696.867 232,862.997 260,027.519 2.33 10.00

transaction sync > optimistic > concurrent non-conflicting operations (100 records)

Implementation Rank Operations/sec Mean (ms) Min (ms) Max (ms) RME (%) Samples
🥇 rocksdb 1 583.12 ops/sec 1,714.922 1,641.762 3,196.548 1.08 292
🥈 lmdb 2 4.21 ops/sec 237,738.56 229,801.694 278,612.686 4.37 10.00

transaction sync > optimistic > rollback operations (100 records)

Implementation Rank Operations/sec Mean (ms) Min (ms) Max (ms) RME (%) Samples
🥇 lmdb 1 16.43K ops/sec 60.87 58.21 342.697 0.390 8,214
🥈 rocksdb 2 5.64K ops/sec 177.237 166.62 879.476 0.320 2,822

transaction sync > optimistic > rocksdb - large transaction vs many small

Implementation Rank Operations/sec Mean (ms) Min (ms) Max (ms) RME (%) Samples
🥇 rocksdb 1 2.98K ops/sec 335.84 287.806 477.495 0.213 1,489
🥈 rocksdb 2 598.07 ops/sec 1,672.055 1,588.225 4,095.174 1.35 300

transaction sync > optimistic > lmdb - large transaction vs many small

Implementation Rank Operations/sec Mean (ms) Min (ms) Max (ms) RME (%) Samples
🥇 lmdb 1 344.52 ops/sec 2,902.603 2,121.241 8,551.096 4.75 173
🥈 lmdb 2 4.20 ops/sec 237,854.334 211,872.502 255,987.877 4.10 10.00

transaction sync > optimistic > empty transaction overhead

Implementation Rank Operations/sec Mean (ms) Min (ms) Max (ms) RME (%) Samples
🥇 lmdb 1 793.39K ops/sec 1.26 1.09 1,673.367 0.797 396,696
🥈 rocksdb 2 96.75K ops/sec 10.34 9.00 2,211.975 1.46 48,377

transaction sync > optimistic > transaction with only reads (100 records)

Implementation Rank Operations/sec Mean (ms) Min (ms) Max (ms) RME (%) Samples
🥇 lmdb 1 1.55K ops/sec 646.511 589.344 4,325.035 3.39 774
🥈 rocksdb 2 274.42 ops/sec 3,644.018 3,413.889 11,521.025 3.81 138

transaction sync > pessimistic > simple put operations (100 records)

Implementation Rank Operations/sec Mean (ms) Min (ms) Max (ms) RME (%) Samples
🥇 rocksdb 1 532.36 ops/sec 1,878.414 1,819.025 2,640.7 0.429 267
🥈 lmdb 2 4.27 ops/sec 234,176.86 217,763.745 254,171.794 3.57 10.00

transaction.bench.ts

transaction > optimistic > simple put operations (100 records)

Implementation Rank Operations/sec Mean (ms) Min (ms) Max (ms) RME (%) Samples
🥇 rocksdb 1 289.37 ops/sec 3,455.726 2,827.035 5,545.969 2.10 145
🥈 lmdb 2 3.25 ops/sec 307,755.982 298,163.7 323,125.261 2.19 10.00

transaction > optimistic > batch operations (100 records)

Implementation Rank Operations/sec Mean (ms) Min (ms) Max (ms) RME (%) Samples
🥇 rocksdb 1 2.31K ops/sec 432.43 334.04 528.327 0.410 1,157
🥈 lmdb 2 289.89 ops/sec 3,449.63 2,034.496 9,566.125 7.27 145

transaction > optimistic > read-write operations (100 records)

Implementation Rank Operations/sec Mean (ms) Min (ms) Max (ms) RME (%) Samples
🥇 rocksdb 1 190.71 ops/sec 5,243.503 3,851.979 11,714.057 6.81 96.00
🥈 lmdb 2 3.40 ops/sec 293,975.408 288,739.35 298,039.515 0.903 10.00

transaction > optimistic > concurrent non-conflicting operations (100 records)

Implementation Rank Operations/sec Mean (ms) Min (ms) Max (ms) RME (%) Samples
🥇 rocksdb 1 917.83 ops/sec 1,089.532 935.797 5,823.538 3.56 459
🥈 lmdb 2 252.18 ops/sec 3,965.457 2,256.463 8,092.857 3.64 127

transaction > optimistic > rollback operations (100 records)

Implementation Rank Operations/sec Mean (ms) Min (ms) Max (ms) RME (%) Samples
🥇 rocksdb 1 5.18K ops/sec 192.989 183.751 751.955 0.509 2,592
🥈 lmdb 2 278.83 ops/sec 3,586.378 3,141.295 6,992.51 3.55 140

transaction > optimistic > rocksdb - large transaction vs many small

Implementation Rank Operations/sec Mean (ms) Min (ms) Max (ms) RME (%) Samples
🥇 rocksdb 1 2.24K ops/sec 446.332 376.329 709.558 0.308 1,121
🥈 rocksdb 2 323.98 ops/sec 3,086.602 2,709.747 7,843.838 3.68 162

transaction > optimistic > lmdb - large transaction vs many small

Implementation Rank Operations/sec Mean (ms) Min (ms) Max (ms) RME (%) Samples
🥇 lmdb 1 279.40 ops/sec 3,579.148 3,090.53 6,763.679 3.71 141
🥈 lmdb 2 3.48 ops/sec 287,570.058 272,868.396 301,767.085 2.67 10.00

transaction > optimistic > empty transaction overhead

Implementation Rank Operations/sec Mean (ms) Min (ms) Max (ms) RME (%) Samples
🥇 lmdb 1 39.81K ops/sec 25.12 16.79 5,446.186 3.75 19,907
🥈 rocksdb 2 39.25K ops/sec 25.48 20.30 625.316 0.842 19,626

transaction > optimistic > transaction with only reads (100 records)

Implementation Rank Operations/sec Mean (ms) Min (ms) Max (ms) RME (%) Samples
🥇 lmdb 1 339.66 ops/sec 2,944.105 2,566.568 19,180.406 7.02 170
🥈 rocksdb 2 123.13 ops/sec 8,121.544 7,252.258 13,157.981 3.59 62.00

transaction > pessimistic > simple put operations (100 records)

Implementation Rank Operations/sec Mean (ms) Min (ms) Max (ms) RME (%) Samples
🥇 rocksdb 1 233.91 ops/sec 4,275.126 3,826.178 6,717.684 1.85 117
🥈 lmdb 2 3.11 ops/sec 321,908.249 311,249.276 339,934.139 2.43 10.00

worker-get-sync.bench.ts

Worker > random keys - small key size (100 records, 1 worker)

Implementation Rank Operations/sec Mean (ms) Min (ms) Max (ms) RME (%) Samples
🥇 lmdb 1 15.05K ops/sec 66.47 56.69 6,907.922 3.17 7,523
🥈 rocksdb 2 3.75K ops/sec 266.516 220.284 1,440.086 1.00 1,877

Worker > random keys - small key size (100 records, 2 workers)

Implementation Rank Operations/sec Mean (ms) Min (ms) Max (ms) RME (%) Samples
🥇 lmdb 1 14.23K ops/sec 70.26 60.54 6,996.116 3.21 7,117
🥈 rocksdb 2 3.61K ops/sec 276.973 231.477 1,524.063 1.03 1,806

Worker > random keys - small key size (100 records, 10 workers)

Implementation Rank Operations/sec Mean (ms) Min (ms) Max (ms) RME (%) Samples
🥇 lmdb 1 6.86K ops/sec 145.748 118.249 10,985.131 4.88 3,431
🥈 rocksdb 2 1.75K ops/sec 572.922 492.662 4,572.103 2.02 873

worker-put-sync.bench.ts

putSync() > random keys - small key size (100 records, 1 worker)

Implementation Rank Operations/sec Mean (ms) Min (ms) Max (ms) RME (%) Samples
🥇 rocksdb 1 1.96K ops/sec 509.342 459.65 2,717.6 1.09 982
🥈 lmdb 2 3.93 ops/sec 254,416.711 243,646.156 276,818.499 3.11 10.00

putSync() > random keys - small key size (100 records, 2 workers)

Implementation Rank Operations/sec Mean (ms) Min (ms) Max (ms) RME (%) Samples
🥇 rocksdb 1 1.46K ops/sec 683.597 634.753 3,168.132 1.06 732
🥈 lmdb 2 1.84 ops/sec 543,829.079 526,743.916 579,790.127 2.09 10.00

putSync() > random keys - small key size (100 records, 10 workers)

Implementation Rank Operations/sec Mean (ms) Min (ms) Max (ms) RME (%) Samples
🥇 rocksdb 1 800.37 ops/sec 1,249.422 1,083.601 4,910.949 1.81 401
🥈 lmdb 2 1.10 ops/sec 907,961.318 799,045.21 1,108,209.673 7.14 10.00

Results from commit 9d259c5

@cb1kenobi cb1kenobi marked this pull request as ready for review November 12, 2025 00:08
@cb1kenobi cb1kenobi marked this pull request as draft November 12, 2025 17:35
cb1kenobi and others added 9 commits November 12, 2025 12:18
- Add null pointer checks for entry data before accessing
- Add bounds checking for txnHeader index
- Add detailed error logging in Windows WriteFile operations
- Skip empty iovecs in Windows implementation
- Validate iovec pointers before WriteFile calls

These defensive checks should help diagnose and prevent access violations
on Windows Node.js 18, where the zero-copy optimization may encounter
stricter memory validation.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Transaction Log: Store with auto-rotation

3 participants