[data] Ingestion correctness & performance: embedding-dim default, O(n^2) edges, no write transaction, dup parser

[rahulkatiyar19955]

Severity

Medium

Summary

A cluster of correctness and performance issues in the ingestion/graph-write path: an embedding-dimension default that mismatches the local embedder, an O(n²) concurrent-edge generator, per-write Neo4j sessions with no shared transaction (partial-write corruption + event-loop blocking), an O(n·m) anomaly-to-log matcher, misleading Hz rounding, and ~250 lines of duplicated parser logic that has diverged.

Findings

• init_indexes default embedding_dim=1536 mismatches the local MiniLM default (384) — backend/app/services/neo4j_client.py:19 (vs backend/app/services/embeddings.py:33-36) — When no OpenAI key is set, the default embedder is MiniLM at 384 dims, but a default-created vector index is built at 1536. Written vectors then don't match the index, so db.index.vector.queryNodes errors and every search silently falls back to full-text. Fix: make embedding_dim required (remove the misleading 1536 default), or default it to embedding_service.get_embedding_dimension().

• CONCURRENT_WITH generation is O(n²) and unbounded for clustered timestamps — backend/app/services/causal_rules.py:119-136 — The nested loop emits an edge for every pair within 50ms; the break only bounds the scan when timestamps are well spread. Bursts of same-timestamp logs (or the long-bag 0.0 timestamp bug) make it full O(n²) and write a huge number of edges into Neo4j, with no per-node cap. Fix: cap concurrent neighbors per log, or skip CONCURRENT_WITH when a timestamp cluster exceeds a threshold; ensure numeric timestamps so the break actually bounds the scan.

• Per-write Neo4j sessions with no shared transaction — partial-write corruption + blocks the event loop — backend/app/services/neo4j_client.py:14-17 and all write_* methods — Each write_* opens its own session for a single statement, so a mid-sequence failure (e.g. write_edges) leaves a partially written graph (Session + logs committed, edges missing) with no rollback. The blocking sync driver also runs directly inside the async ingestion task, blocking the event loop. Fix: let write_* accept an optional session/tx so ingestion can wrap all writes in one execute_write transaction; run blocking driver calls via asyncio.to_thread (or switch to the async driver).

• _derive_anomalies nearest-log match is O(n·m) — backend/app/services/parser.py:303-316 — For each anomaly it does min(log_t_pairs, key=…), i.e. O(anomalies × logs), which is slow on real bags. Fix: pre-sort logs by time and binary-search the nearest log, or bucket logs by time.

• hz uses post-rounding and can mislead on sub-second bags — backend/app/services/parser.py:605 and mcap_parser/parser.py:428 — hz = round(msgs/dur, 2); a tiny dur yields wildly inflated Hz, and dur == 0 reports hz = 0.0 even when messages exist. Fix: guard tiny/zero durations and report null Hz below a sane threshold.

• Heavy parser logic duplicated across two files with subtle divergences — backend/app/services/parser.py and mcap_parser/parser.py (_extract_sensor_info, _TFParser, diagnostics decode, sensor/severity maps; plus id-numbering and _derive_anomalies differ) — ~250 duplicated lines that have drifted apart (the mcap_parser service sorts and renumbers ids while the inline _parse_mcap does not; _derive_anomalies is keyed differently), so ingestion behavior depends on whether the parser service is up. Fix: extract the shared parsing into one importable module used by both, or always go through the service and delete the inline copy.

Project-rule reference

n/a