[discussion] Catalog-audit use case — is it in scope for the agent's design?

[shawnxiao105-afk]

Hi @shirshanka — sorry to land a design question in the issue tracker (Discussions are disabled, so this seemed the cleanest place; happy to relocate if there's a better one).

Context: I'm a data-governance practitioner exploring the Analytics Agent as part of evaluating where DataHub could help our team's catalog quality work — multi-source metadata, manual ownership, uneven description coverage. Loved the recent blog post; the "passive accumulation → active evolution" framing resonates strongly.

While playing with the agent locally against datahub docker quickstart sample data, I noticed something I'd like to understand the design intent behind, before going further.

The system prompt seems to anchor the workflow around what I'd call an analyst-question shape:

• The goal statement at the top frames the agent's job as answering "data questions" terminating in SQL execution.
• Tool definitions explicitly mark search_documents as "USE THIS FIRST".
• Workflow Step 1 reinforces it: "Always call search_documents before doing anything else."

The "Documentation = intent / catalog = existence" principle further down separates the two cleanly, but the operational workflow doesn't seem to give catalog-existence a peer-level path — it appears mostly as scaffolding toward SQL.

The behavior I observed (against quickstart sample data):

• Asked "tell me about SampleHiveDataset". The dataset exists in DataHub (verified via direct GMS call) but has no description, glossary term, or domain assignment.
• The agent ran search_documents, got zero hits, and concluded the dataset "doesn't exist in the catalog" — without falling through to an entity-level search.
• /improve-context (per SKILL.md Step 1) explicitly scopes itself to the conversation history, so datasets that nobody has asked about — often the highest-governance-value ones — never enter the improvement candidate set.

My question, purely about design intent (not asking for any fix at this point):

Is the agent intentionally scoped to "analyst asking business questions," with catalog audit / proactive governance treated as a separate concern (perhaps a different surface entirely)? Or is the analyst use case the phase-1 build-out, with the governance angle planned but not yet shaped?

The blog framing read to me like it covers the governance use case, but the implementation seems to keep "active evolution" bounded by what users happen to query. Wanted to understand whether that's deliberate scoping (and if so, the reasoning) or a phase-1 limitation.

Happy to file smaller, behavior-specific issues separately if useful — asked the bigger question first so I don't propose anything that's already considered out of scope.

Thanks for any color. This is the most interesting AI-over-metadata project I've come across, and I'd like to understand its shape correctly before contributing in earnest.

[shirshanka]

Hi @shawnxiao105-afk — thanks for doing the reading before asking. Direct answer: the analyst-question shape in this repo is intentional, not phase-1 scaffolding. You read the system prompt correctly — search_documents-first, the "USE THIS FIRST" marker, the Step 1 ordering, and the conversation-scoped /improve-context skill are all by design for this surface.

The shape of it: Analytics Agent (OSS) is scoped to analyst workflows — someone arrives with a question, the agent grounds in documentation and the catalog, writes SQL, answers. The catalog is treated as authoritative about existence (per the principle further down in the prompt), but the operational loop is anchored to a user-supplied question. /improve-context is deliberately conversation-bounded because its job is to harden the analyst loop where it actually got used — not to crawl the catalog for unrelated gaps.

Governance and policy workflows live on a different surface: Ask DataHub, a DataHub Cloud feature that runs inside DataHub UI, Slack, Teams, and a Chrome extension. The feature guide is the best entry point: https://docs.datahub.com/docs/features/feature-guides/ask-datahub

That's the surface where Context Documents, Glossary Terms, Domains, policy-anchored answers, quality assertions, and impact analysis all come together into one conversational interface. Worth being precise on scope, though: Ask DataHub is query-driven — it answers questions grounded in your metadata graph plus organizational knowledge. The "actively crawl the catalog for ungoverned datasets and surface them as candidates" shape — the one your /improve-context observation points at — isn't currently a feature of either surface, and not for lack of demand: it's a deliberate position to keep humans in the loop and anchor on policy documents rather than have AI autonomously roam the catalog. Whether that balance shifts is a real product question, just not one the OSS repo is the place to answer.

I should also call out: the relationship between this repo and Ask DataHub isn't documented as crisply in our README as it should be. Your question is partly a symptom of that — fair feedback to take.

Separately, one thing in your repro is a real bug regardless of scope: the agent concluded "doesn't exist in the catalog" off a zero-hit search_documents call. That's a false negative even within the analyst lane — search_documents searches business documentation, not the entity catalog. If documentation returns nothing, the agent should fall through to an entity-level search before declaring non-existence. Would you mind filing that as a separate bug? Your repro (SampleHiveDataset, GMS-verified, no description) is exactly what I'd want in the issue.

On contributing — the analyst-lane here is wide open and your PR #59 already fits cleanly. If your interest is governance specifically, the Ask DataHub docs above are the right next read; happy to chat further on this thread or at shirshanka@acryl.io. Either way, glad you're here.