feat(inventory): project homelab topology into graph - #68
Merged
Conversation
📝 WalkthroughWalkthroughRelease 1.13.1 projects cached homelab inventory into the investigation graph: schema and constants are widened, Migration 30 rebuilds constrained tables, a projection pipeline upserts entities/relationships/evidence, runtime refresh now triggers on interval, local compose/proxy file changes, and optional SSH docker-events, and docs/versions are updated. ChangesInventory projection into investigation graph
Estimated code review effort🎯 4 (Complex) | ⏱️ ~60 minutes Poem
🚥 Pre-merge checks | ✅ 4 | ❌ 1❌ Failed checks (1 warning)
✅ Passed checks (4 passed)
✏️ Tip: You can configure your own custom pre-merge checks in the settings. ✨ Finishing Touches📝 Generate docstrings
🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 6282738def
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| for service in &inventory.services { | ||
| let service_entity = upsert_service(&tx, service)?; | ||
| services.insert(service_key(service), service_entity.clone()); | ||
| services.insert(canonical_or_raw(&service.name), service_entity.clone()); |
There was a problem hiding this comment.
When an inventory has the same Compose/container service name on multiple hosts, this bare-name key overwrites the previous EntityRef. The compose-project and network projection later resolve project.services/network.members by the same bare canonical name, so both projects can be related to whichever duplicate service was inserted last, corrupting the topology graph for multi-host homelabs.
Useful? React with 👍 / 👎.
There was a problem hiding this comment.
Pull request overview
This PR expands Cortex’s investigation graph schema to represent homelab inventory topology (Compose projects, reverse proxies/domains, networks, storage, config artifacts), and adds background mechanisms to keep the inventory cache (and its graph projection) refreshed with lower latency triggers.
Changes:
- Added an inventory→graph projection pipeline (
db::graph_inventory) that upserts topology entities/edges and evidence into the graph tables. - Enhanced server-side inventory refresh with local config file watching and remote
docker eventsstreaming over SSH as refresh triggers. - Updated graph vocabulary/schema constraints, docs, and bumped version to
1.13.0.
Reviewed changes
Copilot reviewed 24 out of 25 changed files in this pull request and generated 3 comments.
Show a summary per file
| File | Description |
|---|---|
| src/runtime/inventory_refresh.rs | Adds watch/SSH event triggers and runs inventory refresh + graph projection. |
| src/runtime/inventory_refresh_tests.rs | Adds unit tests for watcher target selection and SSH args building. |
| src/runtime.rs | Wires the DB pool into the inventory refresh background task. |
| src/db/pool.rs | Bumps schema version and adds migration to rebuild graph tables with widened vocabulary. |
| src/db/pool_tests.rs | Extends graph schema tests to cover new entity/relationship/evidence vocabulary. |
| src/db/graph.rs | Extends graph vocabulary constants for new entity/relationship/reason codes. |
| src/db/graph_inventory/sql.rs | Implements SQL helpers for pruning and upserting inventory-projected graph rows. |
| src/db/graph_inventory.rs | Implements inventory→graph projection logic (entities, relationships, evidence, meta). |
| src/db/graph_inventory_tests.rs | Adds an integration-style test validating topology projection output. |
| src/db.rs | Exposes the new graph_inventory module. |
| src/cli/commands/graph.rs | Updates CLI graph entity type parsing to accept new types. |
| src/app/services/graph.rs | Hooks inventory projection into graph rebuild when cache is available. |
| server.json | Bumps published package version and container tag to v1.13.0. |
| README.md | Documents that server-side refresh projects inventory topology into the graph. |
| mcpb/manifest.json | Bumps MCP bundle manifest version to 1.13.0. |
| docs/mcp/TOOLS.md | Documents server-side refresh + projection behavior for MCP map. |
| docs/INVENTORY.md | Documents refresh cadence, new refresh triggers, and new env toggles/providers. |
| docs/contracts/investigation-graph.md | Updates graph contract vocabulary lists and inventory projection notes. |
| docs/contracts/current-schema.sql | Updates contract schema to reflect widened graph CHECK constraints. |
| docs/CLI.md | Notes that server-side refresh also projects inventory into the graph. |
| CHANGELOG.md | Adds 1.13.0 changelog entry describing topology projection and refresh triggers. |
| Cargo.toml | Bumps crate version to 1.13.0. |
| Cargo.lock | Bumps crate version entry to 1.13.0. |
| .env.example | Adds inventory/map env examples including refresh triggers and optional collectors. |
| .claude-plugin/plugin.json | Bumps plugin version to 1.13.0. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
Comment on lines
+63
to
+76
| refresh_and_project(&pool).await; | ||
| } | ||
| })) | ||
| } | ||
|
|
||
| async fn refresh_and_project(pool: &DbPool) { | ||
| let started = Instant::now(); | ||
| let config = crate::inventory::InventoryConfig::from_env(); | ||
| match crate::inventory::refresh_inventory(config.clone()).await { | ||
| Ok(report) => { | ||
| let projection = match crate::inventory::read_inventory_cache(&config) { | ||
| Ok(inventory) => crate::db::graph_inventory::project_inventory(pool, &inventory), | ||
| Err(error) => Err(error.context("read inventory cache for graph projection")), | ||
| }; |
![cubic-dev-ai[bot]](https://avatars.githubusercontent.com/in/1082092?s=60&v=4){kind=small}
There was a problem hiding this comment.
8 issues found across 25 files
Prompt for AI agents (unresolved issues)
Check if these issues are valid — if so, understand the root cause of each and fix them. If appropriate, use sub-agents to investigate and fix each issue separately.
<file name="server.json">
<violation number="1" location="server.json:10">
P3: CLAUDE.md header version still reads 1.12.0 after server.json was bumped to 1.13.0, creating a version inconsistency across version-bearing files.</violation>
</file>
Reply with feedback, questions, or to request a fix.
Re-trigger cubic
| "source": "github" | ||
| }, | ||
| "version": "1.12.0", | ||
| "version": "1.13.0", |
There was a problem hiding this comment.
P3: CLAUDE.md header version still reads 1.12.0 after server.json was bumped to 1.13.0, creating a version inconsistency across version-bearing files.
Prompt for AI agents
Check if this issue is valid — if so, understand the root cause and fix it. At server.json, line 10:
<comment>CLAUDE.md header version still reads 1.12.0 after server.json was bumped to 1.13.0, creating a version inconsistency across version-bearing files.</comment>
<file context>
@@ -7,11 +7,11 @@
"source": "github"
},
- "version": "1.12.0",
+ "version": "1.13.0",
"packages": [
{
</file context>
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 5
Caution
Some comments are outside the diff and can’t be posted inline due to platform limitations.
⚠️ Outside diff range comments (1)
src/db/pool_tests.rs (1)
452-475:⚠️ Potential issue | 🟡 Minor | ⚡ Quick winUse relationship-specific IDs for evidence inserts.
The test currently fetches
rel_idfrom an unorderedSELECT id FROM graph_relationships, so evidence may attach to the wrong relationship and weaken what this test proves.🛠️ Proposed fix
- let rel_id: i64 = conn - .query_row("SELECT id FROM graph_relationships", [], |row| row.get(0)) + let observed_rel_id: i64 = conn + .query_row( + "SELECT id FROM graph_relationships + WHERE relationship_key = 'source_ip:10.0.0.1:514->host:claimed-host'", + [], + |row| row.get(0), + ) + .unwrap(); + let proxy_rel_id: i64 = conn + .query_row( + "SELECT id FROM graph_relationships + WHERE relationship_key = 'reverse_proxy:example.tootie.tv->domain:example.tootie.tv'", + [], + |row| row.get(0), + ) .unwrap(); @@ - [rel_id], + [observed_rel_id], @@ - [rel_id], + [proxy_rel_id], @@ - [rel_id], + [observed_rel_id],🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@src/db/pool_tests.rs` around lines 452 - 475, The test currently grabs rel_id with an unordered "SELECT id FROM graph_relationships" which can attach evidence to the wrong relationship; change the query that sets rel_id to deterministically select the intended relationship (for example, add a WHERE clause that matches the unique fields of the relationship you just created or use the relationship's known unique key/attributes), so that the subsequent inserts that use rel_id (the two conn.execute calls inserting into graph_relationship_evidence) always target the correct relationship row.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@CHANGELOG.md`:
- Around line 10-28: Update the footer link references in CHANGELOG.md so the
new [1.13.0] section is reflected and the [Unreleased] link no longer points to
v1.10.1...HEAD; add anchor/compare links for 1.13.0, 1.12.0, and 1.11.0 (and
update the Unreleased compare range to v1.13.0...HEAD or remove it), ensuring
the release list footer matches the headings (e.g., update the `[Unreleased]`
target and add hrefs for `1.13.0`, `1.12.0`, `1.11.0` to the corresponding
GitHub compare tags).
In `@docs/contracts/investigation-graph.md`:
- Around line 204-206: The contract text is inconsistent: the earlier "current
v1 projection emits evidence rows for log/error_signature" list must include the
new kinds; update the canonical evidence-kinds list to contain log,
error_signature, source_inventory and app_inventory (i.e., add source_inventory
and app_inventory to the same enumeration that documents current v1 projection
evidence rows) and ensure the surrounding prose references that unified list so
the doc no longer contradicts itself.
In `@src/app/services/graph.rs`:
- Around line 417-431: The current branch treats every error from
read_inventory_cache(&config) as a benign "cache unavailable" and logs at debug;
instead, inspect the specific error returned by read_inventory_cache and only
downgrade to debug for a true "cache miss"/not-found variant, while emitting
warn/error for parse/corruption or I/O failures so projection failures are
visible; update the match on read_inventory_cache(&config) to pattern-match the
error variant (the error type returned by read_inventory_cache in
src/inventory/storage.rs) and call tracing::debug!(...) for NotFound/NoCache
variants but tracing::warn! or tracing::error!(...) for parse/read corruption
and other failures, then proceed to run_db(...,
db::graph_inventory::project_inventory, ...) only when inventory is Ok.
In `@src/db/graph_inventory/sql.rs`:
- Around line 337-344: The returned InventoryGraphStats currently hardcodes
source_row_count = 0 in graph_counts; instead query the stored source count from
the graph_projection_meta table (the value set by update_projection_meta) and
populate InventoryGraphStats.source_row_count with that value (or safely fall
back to 0 if no meta row exists), so callers of graph_counts / project_inventory
receive the actual source row count; locate this logic in graph_counts(conn:
&Connection) and read from graph_projection_meta (or reuse any existing helper
that reads projection meta) before constructing the InventoryGraphStats.
In `@src/runtime/inventory_refresh.rs`:
- Around line 222-226: The current stderr_task uses BufReader::read_to_end into
a Vec (sink) which can grow unbounded for long-lived SSH event streams; change
stderr handling to stream and discard or bounded-log the bytes instead of
accumulating them: replace the read_to_end on the BufReader (stderr_task) with a
loop or a call to tokio::io::copy that copies from the BufReader/reader into
tokio::io::sink() (or otherwise read into a fixed-size buffer and drop or emit
bounded logs), so the task consumes stderr without allowing Vec<u8> to grow
indefinitely.
---
Outside diff comments:
In `@src/db/pool_tests.rs`:
- Around line 452-475: The test currently grabs rel_id with an unordered "SELECT
id FROM graph_relationships" which can attach evidence to the wrong
relationship; change the query that sets rel_id to deterministically select the
intended relationship (for example, add a WHERE clause that matches the unique
fields of the relationship you just created or use the relationship's known
unique key/attributes), so that the subsequent inserts that use rel_id (the two
conn.execute calls inserting into graph_relationship_evidence) always target the
correct relationship row.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Organization UI
Review profile: ASSERTIVE
Plan: Pro
Run ID: 1f108cbd-1133-4353-ad3b-6a0dc5eaee52
⛔ Files ignored due to path filters (1)
Cargo.lockis excluded by!**/*.lockand included by**/*
📒 Files selected for processing (24)
.claude-plugin/plugin.json.env.exampleCHANGELOG.mdCargo.tomlREADME.mddocs/CLI.mddocs/INVENTORY.mddocs/contracts/current-schema.sqldocs/contracts/investigation-graph.mddocs/mcp/TOOLS.mdmcpb/manifest.jsonserver.jsonsrc/app/services/graph.rssrc/cli/commands/graph.rssrc/db.rssrc/db/graph.rssrc/db/graph_inventory.rssrc/db/graph_inventory/sql.rssrc/db/graph_inventory_tests.rssrc/db/pool.rssrc/db/pool_tests.rssrc/runtime.rssrc/runtime/inventory_refresh.rssrc/runtime/inventory_refresh_tests.rs
There was a problem hiding this comment.
7 issues found across 22 files (changes from recent commits).
You’re at about 92% of the monthly reviewed-line limit. You may want to disable incremental reviews to conserve quota. Reviews will continue until that limit is exceeded. If you need help avoiding interruptions, please contact contact@cubic.dev.
Prompt for AI agents (unresolved issues)
Check if these issues are valid — if so, understand the root cause of each and fix them. If appropriate, use sub-agents to investigate and fix each issue separately.
<file name="server.json">
<violation number="1" location="server.json:10">
P3: CLAUDE.md header version still reads 1.12.0 after server.json was bumped to 1.13.0, creating a version inconsistency across version-bearing files.</violation>
</file>
Reply with feedback, questions, or to request a fix.
Re-trigger cubic
There was a problem hiding this comment.
Actionable comments posted: 1
Caution
Some comments are outside the diff and can’t be posted inline due to platform limitations.
⚠️ Outside diff range comments (1)
src/db/graph_inventory/sql.rs (1)
284-329:⚠️ Potential issue | 🟠 Major | ⚡ Quick winClear stale projection metadata on successful runs.
mark_projection_degraded()persistsis_degradedandlast_error, but the success path here only rewrites the three table counts. After the first transient projection failure, later successful refreshes will still report the graph as degraded, andgraph_counts()will keep returning the stalesource_row_countfromgraph_projection_metabecause that column is never refreshed either. Reset the degraded fields here and persist the current source count as part of the successful update.🛠️ Suggested fix
pub(super) fn update_projection_meta( conn: &Connection, counts: &InventoryGraphStats, ) -> Result<()> { conn.execute( "UPDATE graph_projection_meta - SET entity_count = ?1, - relationship_count = ?2, - evidence_count = ?3, + SET source_row_count = ?1, + entity_count = ?2, + relationship_count = ?3, + evidence_count = ?4, + is_degraded = 0, + last_error = NULL, updated_at = strftime('%Y-%m-%dT%H:%M:%fZ', 'now') WHERE id = 1", params![ + counts.source_row_count, counts.entity_count, counts.relationship_count, counts.evidence_count ], )?;Also populate
counts.source_row_countbefore callingupdate_projection_meta(), otherwise the returnedInventoryGraphStatsstays wrong even after a successful projection.🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@src/db/graph_inventory/sql.rs` around lines 284 - 329, The update only writes entity/relationship/evidence counts but never clears degraded state or updates source_row_count, so a later successful run still reports the projection as degraded and returns stale source_row_count; modify update_projection_meta to also set source_row_count = ?4, is_degraded = 0 and last_error = NULL (and updated_at) and accept counts.source_row_count as a parameter, and ensure the caller populates InventoryGraphStats.source_row_count before calling update_projection_meta; keep mark_projection_degraded as-is for failures.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@src/runtime/inventory_refresh.rs`:
- Around line 105-108: The call to
crate::db::graph_inventory::mark_inventory_projection_failed is currently
ignored (let _ = ...), which can hide DB write failures; replace the discard
with explicit error handling: capture the Result returned by
mark_inventory_projection_failed (from graph_inventory), match on Ok/Err, and on
Err log the failure with context (include the original projection error and the
DB error), and optionally increment a metric or trigger a retry/backoff path so
projection health isn't left stale; ensure the code references
mark_inventory_projection_failed, projection_pool, and the error.to_string()
value when constructing the log/metric.
---
Outside diff comments:
In `@src/db/graph_inventory/sql.rs`:
- Around line 284-329: The update only writes entity/relationship/evidence
counts but never clears degraded state or updates source_row_count, so a later
successful run still reports the projection as degraded and returns stale
source_row_count; modify update_projection_meta to also set source_row_count =
?4, is_degraded = 0 and last_error = NULL (and updated_at) and accept
counts.source_row_count as a parameter, and ensure the caller populates
InventoryGraphStats.source_row_count before calling update_projection_meta; keep
mark_projection_degraded as-is for failures.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Organization UI
Review profile: ASSERTIVE
Plan: Pro
Run ID: 0bb2dba0-b3cb-470e-a471-309dca0a0f6c
⛔ Files ignored due to path filters (1)
Cargo.lockis excluded by!**/*.lockand included by**/*
📒 Files selected for processing (21)
.claude-plugin/plugin.json.env.exampleCHANGELOG.mdCLAUDE.mdCargo.tomlREADME.mddocs/CLI.mddocs/INVENTORY.mddocs/contracts/current-schema.sqldocs/contracts/investigation-graph.mddocs/mcp/TOOLS.mdmcpb/manifest.jsonserver.jsonsrc/app/services/graph.rssrc/db/graph_inventory.rssrc/db/graph_inventory/sql.rssrc/db/graph_inventory_tests.rssrc/db/pool_tests.rssrc/runtime.rssrc/runtime/inventory_refresh.rssrc/runtime/inventory_refresh_tests.rs
35 tasks
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Verification
Notes
Summary by cubic
Projects the homelab inventory cache into the investigation graph with a richer topology model, scoped evidence, and near‑real‑time updates. Preserves graph rebuild health by degrading gracefully on inventory issues and keeping remote Docker events opt‑in.
New Features
graph rebuild; remote Dockereventsover SSH are opt‑in viaCORTEX_INVENTORY_REMOTE_DOCKER_EVENTS=true.Bug Fixes
Written for commit bcb0103. Summary will update on new commits.
Summary by CodeRabbit
New Features
Bug Fixes
Documentation