Repliq

Repliq is a local-first, CRDT-based data sync engine for Rust. Build apps that work offline, sync automatically between devices, and merge data without conflicts.

Features

• Conflict-Free Replication: Built on proven CRDT (Conflict-free Replicated Data Type) algorithms
• Offline-First: All data is stored locally; sync happens in the background
• Multiple CRDT Types: LWW-Set, LWW-Map (more coming soon)
• Persistent Storage: Local data persistence using sled (embedded database)
• Network Sync: WebSocket-based sync layer for multi-device support
• Rust-Native: Fast, safe, and memory-efficient

Project Structure

repliq/
├── repliq-core/          # Core CRDT implementations
├── repliq-storage/       # Local persistence layer (sled)
├── repliq-network/       # WebSocket sync protocol
├── repliq-cli/           # CLI tool for running sync servers
└── examples/
    └── notes-app/        # Example notes application

Quick Start

1. Install

Add Repliq to your Cargo.toml:

[dependencies]
repliq-core = { path = "path/to/repliq/repliq-core" }
repliq-storage = { path = "path/to/repliq/repliq-storage" }
repliq-network = { path = "path/to/repliq/repliq-network" }

2. Build the Project

cargo build --release

3. Run the Example Notes App

Add a note:

cargo run -p notes-app -- add "My First Note" "This is the content"

List all notes:

cargo run -p notes-app -- list

Start a sync server:

cargo run -p notes-app -- server --addr 127.0.0.1:8080

Sync with server (in another terminal):

cargo run -p notes-app -- --data-dir ./data2 sync --url ws://127.0.0.1:8080

Usage Examples

Basic LWW-Map Usage

use repliq_core::{LwwMap, CrdtState, ReplicaId};

// Create a new map with a unique replica ID
let replica_id = ReplicaId::new();
let mut map = LwwMap::new(replica_id);

// Add/update entries
map.set("user_name".to_string(), "Alice".to_string());
map.set("user_age".to_string(), "30".to_string());

// Get values
if let Some(name) = map.get(&"user_name".to_string()) {
    println!("Name: {}", name);
}

// Merge with another replica
let mut map2 = LwwMap::new(ReplicaId::new());
map2.set("user_email".to_string(), "alice@example.com".to_string());

map.merge(map2);
println!("Total entries: {}", map.len());

Basic LWW-Set Usage

use repliq_core::{LwwSet, CrdtState, ReplicaId};

let replica_id = ReplicaId::new();
let mut set = LwwSet::new(replica_id);

// Add elements
set.add("apple".to_string());
set.add("banana".to_string());

// Check membership
assert!(set.contains(&"apple".to_string()));

// Remove elements
set.remove("apple".to_string());
assert!(!set.contains(&"apple".to_string()));

// Get all elements
let elements = set.elements();
println!("Set contains: {:?}", elements);

Local Persistence

use repliq_storage::{SledStorage, Storage};
use repliq_core::LwwMap;

// Open storage
let backend = SledStorage::open("./data")?;
let storage = Storage::new(backend);

// Store CRDT state
let map = LwwMap::new(replica_id);
storage.put("my_map", &map)?;

// Load CRDT state
let loaded_map: Option<LwwMap<String, String>> = storage.get("my_map")?;

Network Sync - Server

use repliq_network::Server;

#[tokio::main]
async fn main() -> Result<()> {
    let server = Server::new("127.0.0.1:8080".to_string());
    server.run().await?;
    Ok(())
}

Network Sync - Client

use repliq_network::{Client, Message};
use repliq_core::ReplicaId;

#[tokio::main]
async fn main() -> Result<()> {
    let replica_id = ReplicaId::new();
    let mut client = Client::new(replica_id);
    
    client.connect("ws://127.0.0.1:8080").await?;
    
    client.run(|message| {
        match message {
            Message::Sync(sync_msg) => {
                println!("Received sync: {:?}", sync_msg);
            }
            _ => {}
        }
    }).await?;
    
    Ok(())
}

Architecture

CRDT Types

LWW-Map (Last-Write-Wins Map)

• Key-value store where the last write (by timestamp) wins
• Uses hybrid logical clocks for ordering
• Handles concurrent updates gracefully

LWW-Set (Last-Write-Wins Set)

• Set data structure with add/remove operations
• Elements are never truly deleted, just marked as removed
• Bias towards additions when timestamps are equal

Timestamp System

Repliq uses Hybrid Logical Clocks (HLC) that combine:

• Physical time (milliseconds since epoch)
• Logical counter (for events at the same physical time)

This ensures total ordering of events across replicas without requiring clock synchronization.

Conflict Resolution

Conflicts are resolved deterministically using:

1. Timestamp comparison: Later timestamp wins
2. Replica ID tie-breaking: Higher replica ID wins if timestamps are equal

How Sync Works

1. Local Operations: All changes are applied locally first and persisted
2. Operation Broadcast: When connected, operations are sent to peers via WebSocket
3. Merge on Receive: Incoming operations are merged using CRDT semantics
4. Eventual Consistency: All replicas converge to the same state

Device A                  Server                  Device B
   |                        |                        |
   |--- Sync(map_state) --->|                        |
   |                        |--- Broadcast --------->|
   |                        |                        | [Merge]
   |                        |<--- Sync(map_state) ---|
   |<--- Broadcast ---------|                        |
   | [Merge]                |                        |

Optional Features & Future Ideas

Delta-Based Sync

Instead of syncing full state, only send operations (deltas) since last sync:

// Track vector clocks per peer
// Send only operations newer than peer's last known state

Encryption

Encrypt local storage for sensitive data:

// Use age or sodiumoxide for encryption
// Encrypt before storing, decrypt after loading

Language Bindings

• JavaScript/TypeScript: WebAssembly bindings via wasm-bindgen
• Python: PyO3 bindings
• Go: CGO bindings (already used in the workspace)

Advanced CRDTs

• CRDT Text: Collaborative text editing (RGA, Automerge-style)
• Counter: Increment/decrement counters
• Registers: Multi-value register (MVR)

Libp2p Integration

Replace WebSocket with libp2p for:

• Peer-to-peer connections (no central server)
• NAT traversal
• Discovery mechanisms

Garbage Collection

Prune old tombstones and reduce memory usage:

// Keep operations for last N days only
// Compact state periodically

Testing

Run all tests:

cargo test --workspace

Run specific module tests:

cargo test -p repliq-core
cargo test -p repliq-storage
cargo test -p repliq-network

Benchmarking

cargo bench

Contributing

Contributions are welcome! Please:

1. Fork the repository
2. Create a feature branch
3. Add tests for new functionality
4. Ensure cargo test and cargo clippy pass
5. Submit a pull request

License

Licensed under either of:

• Apache License, Version 2.0 (LICENSE-APACHE)
• MIT license (LICENSE-MIT)

at your option.

References

• CRDTs: Consistency without concurrency control
• Hybrid Logical Clocks
• Automerge: A CRDT for collaborative applications

Acknowledgments

Inspired by:

• Automerge
• Yjs
• ElectricSQL
• RxDB