Discovery in v2: replacing GossipSub with DHT-native search

[tkircsi]

Discussed in #1968

^{Originally posted by tkircsi August 4, 2026}

Summary

Discovery results used to depend on when your node joined the network. A node that started
after a record was announced simply did not know about it, sometimes for a day and a half. This was
not a bug in one function — it followed from the architecture, so we changed the architecture.

Records and their labels are now published directly into the DHT, and searching is a live lookup
against the network rather than a scan of whatever announcements a node happened to overhear.
GossipSub is gone.

This is a breaking change, shipping in v2.0.0 as a flag-day cutover.

---

The problem

Discovery worked by broadcast. When a node published a record, it announced the record's labels
over GossipSub, and every subscribed peer cached that announcement locally. Searching meant
scanning your own cache.

That works only if you were listening at the moment of the announcement. Three consequences
followed:

Late joiners saw nothing. A node joining the network today learns nothing about anything
announced yesterday. It stays partially blind until each record is re-announced, which was every
36 hours.

Every restart opened a fresh gap. Persistent storage keeps what a node already learned, so a
restart didn't erase its view. But nothing is delivered while a node is down. Anything announced
during the outage was missed outright, and only surfaced at the next re-announcement — up to 36
hours later. A two-minute redeploy could cost a day of visibility into whatever changed meanwhile.
Nodes running in-memory storage fared worse: they lost the cache entirely and restarted blind. This
is what made the problem routine rather than occasional — it hit every existing node on every
deploy, not just newcomers.

Two nodes gave different answers. Because each node searched its own cache, and each cache
reflected a different slice of history, the same query returned different results depending on
which node you asked. There was no single answer to converge on.

Underneath all three: there was no way to pull. A node could only wait to be told.

---

What we changed

The DHT already solves "who has this?" — it just wasn't being used for it. Records were announced
to it, but nothing ever looked anything up. Discovery ran entirely on the broadcast layer bolted
alongside.

So we made labels first-class DHT keys. Publishing a record now advertises both the record's CID
and each of its labels, expanded to their ancestors, so a record tagged
/skills/natural_language_processing/text_completion is discoverable by searching for
/skills/natural_language_processing too.

Searching is then two steps: ask the DHT which peers hold a label, then ask those peers directly
what they have.

flowchart TB
  subgraph before["v1 — wait to be told"]
    direction LR
    A1[Node publishes] -->|GossipSub broadcast| S1[Subscribers cache it]
    S1 --> C1[Search scans the local cache]
    X1[Joined late? Restarted?] -.->|misses the broadcast| C1
  end

  subgraph after["v2 — pull from the network"]
    direction LR
    A2[Search /skills/A] -->|"FindProviders(hash of label)"| D2[(DHT)]
    D2 -->|peers holding it| P2[Peer 1, Peer 2, ...]
    P2 -->|"QueryRecords(full query)"| R2[Matching records]
  end

  before ~~~ after

Loading

The important shift is that nothing is cached in between. A search reaches live peers and asks them
what they currently hold, so the answer doesn't depend on what this node witnessed or how long it
has been running. Two nodes asking the same question now get the same answer.

This costs a network round trip where v1 did a local lookup. We think that's the right trade:
v1's local lookup was fast and frequently wrong.

It also fails earlier and more honestly. A peer that is offline now drops out at the discovery
step, rather than appearing in results and failing when you try to fetch from it.

Related fixes that came with it

• Restarts no longer lose advertisements. Nodes re-advertise everything they publish on
  startup. Previously a restarted node advertised nothing until its next scheduled cycle.
• Provider counts are real. They were counting announcements this node happened to see
  rather than actual providers, which is why two nodes ranked the same results differently.
• Publishing takes effect immediately. It was queued for a scheduler that ran hourly.
• Bulk publishing no longer stalls. Publishing more records than the internal queue could hold
  silently deferred the remainder to the next hourly cycle; a 500-record publish took roughly five
  hours. It now runs at full speed.

---

Storage: a different shape entirely

The two designs distribute the index very differently. (Record content is unaffected — your
content store and database are unchanged. This is only about the data used to find things.)

Under GossipSub, every node cached every announcement it saw. Each node ended up holding an index
of the entire network, and every new record published anywhere added an entry on every node.

Under the DHT, each key lives on the ~20 peers closest to it. A node stores pointers for its share
of the keyspace, not all of it. As the network grows it gains both more keys and more peers to
hold them, so the two scale together.

	v1 — GossipSub cache	v2 — DHT
what a node stores	every announcement it witnessed	its share of the keyspace, ~20 copies network-wide
grows with	total records in the network	records published by this node
per entry	label, CID, peer, timestamps	key → peer IDs

The practical difference is the scaling law. v1 per-node storage grows linearly with the network;
v2 per-node storage stays roughly flat as nodes are added.

Some arithmetic from the design — illustrative, not benchmarks — assuming ~100 records per node and
~10 labels per record once ancestors are expanded:

Team network, 10 nodes, 1,000 records. v1: ~10,000 entries per node. v2: ~1,500 keys, and since
there are fewer than 20 nodes every node holds nearly all of them. No real difference at this
scale — with fewer peers than the replication factor, everyone stores everything either way.

Public network, 1,000 nodes, 100,000 records. v1: ~1,000,000 entries on every node — each
one carrying a full copy of the global index. v2: roughly 100,000 record keys plus ~1,500 label
keys, replicated 20 times and spread across 1,000 nodes, so about 2,000 entries per node.

That's around a 500× difference, and the gap widens with every node added, because v1's number
keeps climbing while v2's does not. A workstation joining a large public network under v1 had to
carry the whole network's index; under v2 it carries roughly what it publishes.

Label keys in particular are bounded by the taxonomy rather than by record count. The OASF 1.1.0
skills taxonomy is 513 nodes, so a million records tagged with those skills still produce at most
513 skill keys — shared, not per record.

---

What this means if you run a node

v1 and v2 nodes cannot see each other. The protocol identifiers changed, so the two form
separate networks. They can run side by side on one host during migration. Recommended order:
bootstrap and relay infrastructure first, then pin and team servers so content is present before
anyone searches, then everything else.

Delete the routing datastore directory. No data migration. Your content store and database are
untouched, and each node re-advertises from its database on startup.

Republish whatever your node was serving. Push and publish stay separate, exactly as before:
push stores a record, publish makes it discoverable. What moved is where that state is kept.
Publication used to be recorded in the routing datastore; it is now a field on the record itself,
alongside everything else we know about it. Since the upgrade deletes that datastore and the new
field defaults to unpublished, records come back held but private. Publishing them again restores
discoverability.

Configuration removed: all gossipsub.* keys, and routing.autosync.*.

Autosync was removed. It only ever reacted to GossipSub announcements, so it had no trigger
left. Rather than ship something inert, we removed it. It's worth rebuilding against a real
requirement — as a poll of trusted peers, or as standing queries over the DHT — and the machinery
it needs is now in better shape than before.

---

Trade-offs we accepted

Search is best-effort, not exhaustive. A DHT lookup reaches the peers it can reach within a
time budget. Results stream in and completeness improves until that budget expires, so a slow or
partitioned network costs recall, not correctness. This is inherent to Kademlia rather than
something we chose, but it's worth stating plainly: the search API does not promise every match.

Each search costs network round trips. Discovery and a query per responding peer, against v1's
single local read. Bounded by concurrency limits and deadlines, and cached results are on the list
of future improvements.

Unpublishing is not instant. Removing a record stops it being advertised immediately, but
copies of the pointer already distributed to other peers expire on their own within about 48 hours.
Kademlia has no revocation. Unpublish takes effect locally at once and network-wide as those
expire.

---

Status

Implemented and validated. Unit tests pass under the race detector; end-to-end suites pass against
a real four-node network and a single-node deployment. #1967

Happy to go deeper on any part of this — particularly the migration sequence if you operate
infrastructure nodes.