Skip to content

ARCHITECTURE.md

Latest commit

 

History

History
361 lines (281 loc) · 15.9 KB

ARCHITECTURE.md

File metadata and controls

361 lines (281 loc) · 15.9 KB
Raw

brainstack Architecture

Version 2.0 · stack version 1.0.0 · see CHANGELOG for history and ADR-007 for the versioning decision.

The brain: how a plain folder of Markdown becomes a durable, connected, self-maintaining knowledge system, operated by an agent such as Hermes Agent. This document is the reference for every skill in the stack. If you only read one file, read this one.

Core insight

Memory is not a flat dump of context. Every fact, preference, decision and observation has a type, a confidence, a provenance and a lifecycle. Notes are not files — they are atomic nodes in a graph, connected by links that must be maintained or they silently rot.

The system is a three-layer pipeline, mirroring how memory actually works:

Layer 1 - REAL-TIME CAPTURE      (during conversation / reading)
Layer 2 - NIGHTLY CONSOLIDATION  (the dream cycle)
Layer 3 - CROSS-SESSION INTELLIGENCE (weekly distillation)

Each layer feeds the next. Capture without consolidation is a junk drawer; consolidation without capture is a library with no new books; intelligence without both is a blog no one reads.

Around this pipeline sits the agent layer: one orchestrator, several domain workers, and a credential boundary that decides what each may do.

┌────────────────────────────── AGENT LAYER ──────────────────────────────┐
│  Orchestrator (main agent)                                              │
│    loads context modes · delegates to workers · delivers to the human   │
│        │                                        │                       │
│        ▼                                        ▼                       │
│  Workers (isolated context)            Credential boundary              │
│  research · network · content ·        work flows upward,               │
│  finance · vault                       credentials do not flow down     │
└──────────────────────────────┬──────────────────────────────────────────┘
                               ▼
┌────────────────────────── KNOWLEDGE LAYER ──────────────────────────────┐
│  Capture → Atomic notes → Graph (Iron Law backlinks) → Retrieval        │
│  Memory (typed, decaying) · NOW.md · Tree Index · MOCs                  │
└──────────────────────────────┬──────────────────────────────────────────┘
                               ▼
┌────────────────────────── INFRASTRUCTURE LAYER ─────────────────────────┐
│  Plain Markdown + git · regenerable indexes · secrets outside the repo  │
└──────────────────────────────────────────────────────────────────────────┘

Layer 1 — Capture

The pipeline (target: under 30 seconds per item)

  1. Smart extraction — long documents (PDFs, articles, transcripts) become a summary + bullets. NEVER the full text. Full text is for the original file, not for notes.
  2. Atomic note — one note = one concept. A document with 5 themes becomes 5 notes. Titles are clear and unique.
  3. Wikilinks — 4-8 high-value links per note. A note without links is not a note; it is a tombstone.
  4. MOC update — update an existing Map of Content; create a new one only when 3+ notes exist on the theme.
  5. Tree Index — add the new concept/entity to the Tree Index if it's new.
  6. NOW log — one line in now/recent.md (the changelog).
  7. Log & preserve — record in log.md; the original stays immutable in 00-Inbox/ → moved to Processed/ after analysis.

What gets captured immediately

Trigger Action
Decision made Decision log entry (DEC-###) + memory
Correction received Memory entry with high confidence (0.9)
Preference discovered Memory entry, typed preference
Tool/workflow learning Memory entry + note in References/
Pattern seen 2nd/3rd time Memory entry with counter; 3rd occurrence → procedural memory
Stray thought / principle / insight Captured verbatim with an ID (SIG-###) — the original voice is the value
Deep-dive (article, thesis, framework) Reference note with standardized frontmatter

The Iron Law of backlinks

A note that mentions an entity (person, company, project) that has its own page MUST add a backlink to that entity's page.

A mention without a backlink is a broken synapse. The entity page gains a reverse-chronological "Referenced in" section. This is what turns a folder of notes into a graph — and it is the single most violated rule in every vault that doesn't enforce it. The dream cycle repairs violations nightly.

Layer 2 — The dream cycle (nightly consolidation)

Modeled on how biological memory consolidates during sleep. Runs nightly (cron or manual "good night" trigger), and is idempotent — running it twice is safe. It is the reason the system does not rot. See ADR-003.

  1. Dedupe — people, companies, concepts with duplicate pages; merge, archive the loser (never delete — provenance survives).
  2. Backlink repair — find mentions that violate the Iron Law; add the missing backlinks.
  3. Tree Index refresh — add new entities, retitle stale ones.
  4. NOW.md cleanup — prune essentials.md (today-only urgency), close resolved threads in threads.md, update recent.md.
  5. Contradiction detection — compare active memory entries on the same topic; resolve by recency + confidence (newest and most confident wins); the loser is archived with a superseded_by note, never deleted.
  6. Memory decay — apply TTL decay (see schema below). Stale facts lose confidence; expired facts are flagged, not silently removed.
  7. Next-day priorities — write the 3 priorities for tomorrow.

Agent layer

The knowledge layers are inert without an agent operating them. The agent layer has three parts:

Orchestrator + workers

The main agent (the orchestrator) does the reasoning and delivers results. For well-defined domains it delegates to workers — sub-agents with isolated context that write files and hand results upward. See ADR-008 and the agents skill.

Worker Domain Output
Research Deep dives, data collection report file
Network People CRM, prospecting CRM updates + summary
Content Posts, threads, newsletters drafts folder
Finance Portfolio, revenue, cash flow finance folder
Vault Maintenance, inbox, lint maintenance report

Rules:

  • Workers write files; the orchestrator reads and delivers. Workers never talk to the human directly.
  • Fresh-context isolation: each worker gets only its briefing card — never the full system state. Cheaper, faster, less leakage.
  • Model routing: cheap model does I/O (search, collection, lint), expensive model does reasoning (strategy, voice, judgment).

Context modes

The orchestrator changes behavior per domain by loading a context mode file at the start of each work block (see 00-Hermes/contexts/ in the vault template). A mode adjusts tone, data sources, and allowed tools — e.g. board work, content creation, development, personal finance. Modes keep one agent from behaving like a one-size-fits-all assistant.

The credential boundary

Work flows upward. Credentials do not flow downward.

  • Workers do not inherit MCP servers, messaging credentials, or public gateways automatically.
  • Publishing (Telegram, LinkedIn, X, WhatsApp, email) always goes through the main profile with explicit approval.
  • If a worker identifies an action it cannot perform, it hands the work upward — that is the boundary working, not a limitation.

This rule is the difference between a helpful delegation and a liability. It is enforced by the agents skill and documented in the AGENTS.md template so every agent in the system is trained on it. See ADR-005.

Typed memory schema

Compiler store (auto-generated from vault notes)

Type TTL Default confidence Heuristic
raw 7d 0.3 Lives in 00-Inbox/
working 30d 0.5 Default
semantic 365d 0.8 >500 chars, formal content
episodic 180d 0.7 Has date + meeting/call context
procedural 365d 0.8 "How to", workflow, playbook
preference 365d 0.6 "I like", "I prefer", style statements
decision 730d 0.7 "I decided", "approved", "signed"

Every entry: id, type, title, content(≤2000), source, confidence, created/updated, valid_until, ttl_days, tags, links, superseded_by, hash.

Curated entries (13 types)

The 7 above plus relationship, instruction, goal, commitment, context, event, observation, artifact, error — reserved for human curation. These live as Markdown files with YAML frontmatter: type, confidence, provenance, status, supersedes/superseded_by, tags, source.

Decay rules

Type Rule
preference -0.1 @ 30d, -0.2 @ 90d
decision -0.05 @ 180d
context -0.1 @ 14d
observation -0.15 @ 7d, -0.3 @ 30d
learning +0.05 per validation (reinforcement)

Decay is healthy: preferences change, observations age. TTL marks expiry but does not delete — the compiler keeps the store clean.

Conflict resolution

Conflict is not an error — it is an opportunity to refine.

  1. Detect contradiction between active entries of the same type/topic.
  2. Resolve by recency + confidence: newest and most confident wins.
  3. Loser is archived with a conflict note, never deleted.
  4. Unresolved conflict becomes silent contradiction — apply policy automatically.

Rules of thumb

  1. Every entry has a type.
  2. Archiving is not deleting — history preserves provenance.
  3. One entry = one concept.
  4. Fact ≠ opinion — "X is a good leader" is an observation, not semantic.
  5. Drafts enter with confidence < 0.5 so they don't inflate the signal.
  6. Hot cache (the agent's context) ≠ cold storage (the vault). Neither substitutes for the other.

Retrieval

Two complementary paths:

  • Semantic index (bin/semantic-index.py, bin/semantic-search.py) — embed notes locally, query by meaning. Hybrid search: keyword + vector. Indexes are regenerable and never committed (ADR-006).
  • Iron Law backlinks — the graph. Entity pages accumulate "Referenced in" entries, so any question that starts at a person/company/project walks the graph instead of searching strings.

Retrieval protocol (the skills enforce it):

  1. Brain-first: consult the vault before any external API.
  2. Synthesize + gap analysis: answer from the vault, then say what is missing and where to get it.
  3. Never answer from memory alone when the vault has the source.

The NOW.md system (live context)

File Function Cadence
now/essentials.md Urgent today, deadlines Daily; keep lean, remove what lost relevance
now/threads.md Open threads with owner/deadline/status Every thread; close resolved
now/recent.md Vault changelog (~10 entries) Every modification

NOW.md is what makes the vault usable in the morning: it answers "what is alive right now?" without searching.

Vault layout

00-Inbox/            Raw incoming (immutable; Processed/ after analysis)
00-Hermes/           The operating system (see below)
01-Wiki/             Atomic evergreen notes
  Articles/ Media/ Signals/ Briefings/ Meeting-Notes/
  now/               essentials.md, threads.md, recent.md
02-MOCs/             Maps of Content + Tree Index.md
People/              People and relationships
Organizations/       Companies, institutions, boards
Projects/            Active projects and engagements
References/          Deep-dive reference notes (standardized frontmatter)
00-SECRETS/          Encrypted secrets (*.age) — never plaintext, ADR-004
Archive/             Cold storage
log.md               Audit trail

00-Hermes/ is where the machine keeps its own memory — the agent's contexts, decisions, typed memory store, and worker briefing cards:

00-Hermes/
  contexts/          Context modes (board, content, dev, finance, ...)
  decisions/         DEC-### decision log
  memory/            Typed memory store (working/semantic/procedural/...)
  workers/           Worker briefing cards
  references/        Deep-dive reference notes

Every entity page has typed frontmatter (person, organization, meeting, financial, ...) with expected fields and edges — the schema is what makes automation reliable. The contract lives in vault-template/schemas/.

Decision log

Decisions are first-class citizens: DEC-### entries with context, options considered, rationale, and owner. A decision without a log entry didn't happen; a logged decision without a revisit date is a bet without a check-in. The log is the difference between "we discussed this" and "we decided this, and here is why".

Signals (stray thoughts, principles, insights) get SIG-### IDs so the Iron Law, references and distillation can anchor to them. The signal skill enforces this.

Guardrails

  1. Never copy a full document into a note.
  2. 00-Inbox/ is immutable; log.md records everything.
  3. Under 30s per item — if it takes longer, extract less.
  4. Never overwrite notes the human edits manually.
  5. Lean storage: contacts ≤ 5KB, conversations only summaries, raw emails never enter the vault.
  6. The brain belongs to the human. The agent maintains it, never owns it.
  7. Secrets never enter the repository (ADR-004).
  8. Indexes are regenerable; a broken index is deleted and rebuilt, not "repaired" (ADR-006).

Why plain Markdown + git

  • No lock-in: any editor, any agent, any OS.
  • Diffable: every change is reviewable; rollback is a git command.
  • Portable: the whole brain is a folder you can copy, back up, or gift.
  • Durable: Markdown will outlive every proprietary format you could pick.

The cost (no fancy database) is repaid by the consolidation layer, which reorganizes the mess nightly. Structure is a process, not a schema. See ADR-001.

Versioning and replication

The stack is versioned like the software it is: Semantic Versioning (VERSION

  • git tags), Keep a Changelog (CHANGELOG.md), and decision records (docs/adr/) for every structural choice (ADR-007). The release procedure is in RELEASING.md.

The stack replicates without its content: ./setup scaffolds the vault skeleton, installs skills and the engine, and wires the nightly cron. To stand it up somewhere new — another machine, another person, a team — follow REPLICATION.md. The architecture is the product; the knowledge belongs to its owner.

Further reading