Give it a client's domain, get 10–20 tracking prompts you can run against AI search engines to measure whether that client shows up.
To know if ChatGPT or Perplexity recommends your product, you need questions to ask them — the questions your actual buyers would type. Writing those by hand takes a strategist a day per client and the result is unrepeatable. This generates them from evidence: the client's own docs, pricing tiers, GitHub issues and real search-autocomplete data. Every prompt traces back to a verbatim string in that client's evidence capsule. Nothing is invented.
promptgen run mastra.aiOne command, cold start — no prior database rows, no client config, no manual
ordering of stages. It fetches, restates, derives buyer personas, builds the
capsule, generates, validates through five stages, and writes
exports/mastra.ai.prompts.json.
AI chat is a product-discovery surface at search-engine scale, and OpenAI has now measured it. How People Use ChatGPT (Chatterji, Cunningham, Deming, Hitzig, Ong, Shan & Wadman — OpenAI, Duke and Harvard, September 2025) classified a representative sample of consumer conversations and reports:
- 700 million users sending 18 billion messages a week as of July 2025 — around 10% of the world's adult population.
- ~49% of messages are Asking: seeking information or clarification to
inform a decision.
Doingis 40%,Expressing11% — and Asking is growing faster than the others. Practical GuidanceandSeeking Informationare together almost half of all usage. The paper describesSeeking Informationas covering searches for "information about people, current events, products, and recipes," and "appears to be a very close substitute for web search."- The paper's own summary: ChatGPT "provides economic value through decision support."
If half of that traffic is people asking what to use, then whether a client surfaces in the answer is a measurable fact with revenue attached — and it cannot be measured without a fixed, repeatable set of questions to ask each month. Producing that set, grounded in evidence rather than guesswork, is what this tool does.
- Why track this · Quickstart · What you get
- Who this works for · Requirements
- Setup · Configuration · Commands
- Reading a run · Exit codes · Client config
- How it works · Connectors · Adding a connector
- Development · Layout
- Troubleshooting
From a clean clone to a finished export:
git clone git@github.com:Twix1288/Manicule_PromtpGeneratorCLI.git
cd Manicule_PromtpGeneratorCLI
uv sync # installs into .venv
docker compose up -d # Postgres on :55432
cp .env.example .env # then set ANTHROPIC_API_KEY
uv run promptgen migrate # create the schema
uv run promptgen run mastra.ai # → exports/mastra.ai.prompts.jsonuv sync puts a promptgen executable on the venv's path, so after
source .venv/bin/activate you can drop the uv run prefix. Every example
below omits it.
No API key? promptgen signals mastra.ai and promptgen capsule mastra.ai
run in full without one — everything except model synthesis is deterministic
Python. That is the fastest way to see what the system knows about a client.
A JSON array of records in the eval service's exact schema — 11 fields, no more, checked at the boundary before the file is written:
{
"ID": "6c4d3196-6547-442d-8620-4a776c334243",
"Topic": "Agents",
"Prompt": "Best agent framework for enterprise VPC deployments with predictable cost — how does Mastra compare to LangGraph and Vercel AI SDK?",
"Tags": "enterprise-comparison, vpc, data-residency, langgraph, vercel-ai-sdk",
"Regions": "Global",
"Language": "en",
"Platforms": "chatgpt, perplexity, google-ai-overviews",
"Personas": "Enterprise",
"Analysis Types": "Visibility, Sentiment",
"Created": "9/4/2026, 08:29 PM",
"Updated": "9/4/2026, 08:29 PM"
}The set is spread across four archetypes, so you measure different failure modes rather than the same one twenty times:
| Archetype | Asks like | Catches |
|---|---|---|
| Explicit | "How does Mastra compare to LangGraph on X?" | Whether you win a head-to-head you were named in |
| Implicit | "Our agent stops after one tool call — what do people use?" | Whether you surface for a symptom, unprompted. Must never name the brand. |
| Command | "Write a TypeScript workflow that pauses for human approval…" | Whether you are the default in generated code |
| Multi-Turn | "[Context: top 3 frameworks…] Which of those runs in our VPC?" | Whether you drop out of a shortlist you were just on |
Plus category-tracking templates (Evaluate Mastra on Agents) built in Python,
not asked of a model — so the string is byte-stable across runs and gives you a
real time series. A prompt the model rewords each month charts a different
question each month.
Three sample exports are committed in exports/ if you want to read
output without running anything.
Built for B2B SaaS and developer tools. It runs against any domain without crashing, but output quality tracks how much public, structured evidence the company publishes — and for a consumer brand, half the evidence sources return nothing.
Signals gathered per capsule slot, one run each against eight real domains (2–4 September 2026; n=1 per client, so indicative rather than a benchmark):
| Client | identity | capability | query | competitor | pain | persona |
|---|---|---|---|---|---|---|
| railway.app | 10 | 12 | 24 | 5 | 8 | 4 |
| vercel.com | 10 | 12 | 24 | 4 | 8 | 3 |
| mastra.ai | 9 | 12 | 24 | 3 | 8 | 4 |
| stripe.com | 9 | 12 | 24 | 2 | 6 | 2 |
| render.com | 9 | 9 | 24 | 5 | 3 | — |
| figma.com | 6 | — | 24 | 5 | — | 4 |
| nike.com | 6 | — | 24 | 6 | — | — |
| patagonia.com | 1 | — | 24 | 4 | — | — |
What holds everywhere. query is 24 for every client — Google autocomplete
works for any brand with search volume — and competitor holds up too. So every
company gets a usable topic axis and a usable archetype spread.
What needs a developer-shaped company. Three sources go empty as you move
down the table — capability and pain at Figma, persona one row later at
Nike:
| Slot | Needs | Empty when |
|---|---|---|
capability |
a docs site the crawler recognizes | there are no developer docs |
pain |
a public GitHub repo | there is no public issue tracker |
persona |
a pricing page with named tiers | pricing is not tiered, or not public |
identity degrades separately, on brand ambiguity: patagonia.com scores 1
because the name is a region before it is a company, and the competitor filters
spend their effort discarding national parks.
nike.com still shipped 20 records, and they read like real buyer questions
— "My arches ache and my ankles roll inward after three miles, which stability
running shoes should I shortlist?" But every one of them carries the same
persona, General Buyer, because there were no pricing tiers to derive segments
from. The ICP axis — one of the three this tool is built on — collapses to a
single segment for a consumer brand. That is a thinner product, not an
equivalent one.
- Recoverable: the persona axis. Supply
clients/{domain}.personas.jsonand it comes back — which is the intended path anyway, see Client config. - Not recoverable from config:
capabilityandpain. Those need a source connected — a support inbox, a reviews feed, a docs-analytics export. Naming which one, per client, is what the coverage report is for.
Run against a domain with almost nothing (example.com, 3 signals, 1 kind):
topics 0 · category 0 · 0 capabilities
quotas {'Explicit': 0, 'Command': 0, 'Implicit': 0, 'Multi-Turn': 0}
persona: general-buyer [generic_fallback]
exit=0
Zero topics, zero quotas, the persona flagged generic_fallback, exit 0. No
crash, and no invented market to fill the gap. Shipping nothing is a correct
answer for a client with nothing to ground a prompt in — the one thing the
system will not do is fabricate evidence to hit a number.
| Python | 3.11 (pinned: >=3.11,<3.12) |
| uv | dependency management — curl -LsSf https://astral.sh/uv/install.sh | sh |
| PostgreSQL | 16 recommended. Use the bundled docker-compose.yml, or any local instance. |
| Anthropic API key | Only for generate and run. Everything else works without it. |
uv syncThen a database. Either the bundled compose file:
docker compose up -d # postgres:16 on localhost:55432…or a local Postgres:
createdb promptgenThen configure and create the schema:
cp .env.example .env # every variable is documented inline
promptgen migrate # applies migrations/*.sql in order
promptgen status # shows applied vs pendingMigrations are numbered SQL files applied inside a transaction, with a checksum recorded per file — re-running is a no-op, and an edited migration is refused rather than silently skipped.
All configuration is environment variables, read from .env. .env is
gitignored; never commit real credentials.
| Variable | Required | Purpose |
|---|---|---|
DATABASE_URL |
yes | Postgres DSN, asyncpg driver. Every stage except synthesis runs off cached rows here. |
ANTHROPIC_API_KEY |
for synthesis | Default provider. Also powers pain-point restatement at ingest. |
NIM_API_KEY |
no | Free OpenAI-compatible endpoint, selected with --provider nim. |
NIM_MODEL |
no | Defaults to deepseek-ai/deepseek-v4-pro-0813. |
GITHUB_TOKEN |
no | Raises the GitHub API rate limit. The issues adapter works without it. |
SQL_ECHO |
no | Set to 1 to log every SQL statement. |
Without an Anthropic key, signals and capsule still run in full — only
archetype synthesis needs a model. The category-tracking templates are built in
Python and ship even when every model call fails.
run is the composite. The granular commands are how you debug one stage, and
each reads what the previous one cached.
Everything, from a cold database. Fetch → restate → derive personas → build capsule → generate → validate → export.
promptgen run mastra.ai
promptgen run mastra.ai --total 15 --refresh --timings| Flag | Default | Meaning |
|---|---|---|
--total <int> |
20 |
Target prompt count. Drives per-topic quotas and ICP floors. |
--model <str> |
claude-opus-5 |
Model id for synthesis. |
--provider <str> |
anthropic |
anthropic ships the deliverable; nim is a free endpoint for iteration. |
--effort <str> |
low |
low | default | medium | high. |
--refresh |
off | Re-fetch signals even if the 24h cache is warm. |
--timings |
off | Per-call latency table, split into prefill and decode. |
-v, --verbose |
off | INFO-level logging. |
Fetch, normalize, gate and persist a client's raw signals. This is the only stage that fetches from the client's own sources — everything downstream reads the cache.
promptgen signals mastra.ai
promptgen signals mastra.ai --no-store --json--store / --no-store (default store) · --json · -v, --verbose
Build the evidence capsule and generation directives from cache. Never
fetches — run signals first. Needs no API key, so it is the cheapest way to
inspect what the system knows about a client.
promptgen capsule mastra.ai
promptgen capsule mastra.ai --xml-only > capsule.xml--total <int> (default 20) · --xml-only
Build the capsule from cache, then synthesize and validate. Needs a model.
promptgen generate mastra.ai
promptgen generate mastra.ai --provider nim --no-store # free, no writes
promptgen generate mastra.ai --review # human read harness| Flag | Default | Meaning |
|---|---|---|
--total <int> |
20 |
Target prompt count. |
--model <str> |
claude-opus-5 |
Model id. |
--provider <str> |
anthropic |
anthropic | nim. |
--effort <str> |
low |
Reasoning effort. |
--store / --no-store |
store | Persist the capsule and every candidate. |
--json |
off | Machine-readable output instead of tables. |
--review |
off | Read harness: ask-shape distribution, brand leaks, vendor voice. |
--timings |
off | Per-call latency table. |
--force-partial |
off | Export even though a topic call failed every retry. |
-v, --verbose |
off | INFO-level logging. |
Load a client-supplied ICP config. See Client config.
promptgen personas import clients/vercel.com.personas.jsonApply pending migrations, or show applied vs pending.
run prints five sections, in this order:
- Coverage — which capsule slots filled, from which source, and how many signals survived the pipeline (not how many were fetched). Cells that came back empty are named, along with the archetypes those empty cells gated off, per topic. This is the generator-problem-vs-data-problem answer, and it is the artifact that tells a customer which source to connect next.
- Topics — derived capability areas with evidence counts and per-topic archetype quotas.
- Personas — the buyer segments, and where they came from: a client config, or inferred from pricing tiers.
- Delivery accounting — shipped vs. target, with every deduction named and attributed: rejected at stage 1, brand leaks at stage 2, near-duplicates at stage 4 (with the colliding string and the surviving sibling), quota cuts and ICP-floor backfills at stage 5, and anything over the 20-record ceiling.
- Export path.
A count that reads as success while describing a shortfall is treated as a defect in the same class as a wrong prompt. If the number is short, the run says exactly where each missing prompt went.
| Code | Meaning |
|---|---|
0 |
Success — including a below-floor run. Shipping fewer grounded prompts is correct for a thin client, and failing CI over it would make correct degradation look broken. |
1 |
A crash, no cached signals, an export that fails the format contract, or a partial run. |
2 |
No usable model credential. |
A partial run is one where a topic call failed every retry. It is persisted
as partial, refuses to export without --force-partial, and prints which
topics died. A failed call and a genuinely thin client produce the same-looking
output, so they must not produce the same exit code — that is the one property
a measurement system cannot give up.
Personas are an input. A client knows who it sells to; inferring from a pricing page is the fallback, not the plan.
clients/{domain}.personas.json
See clients/_template.personas.json —
every field is documented inline. Import it with promptgen personas import.
Without one, personas are derived from the client's pricing tiers and the run says so. Nothing fails. Pricing tiers supply each persona's vocabulary either way: the features gated behind a tier are the words that segment uses.
adapters ──► raw_signals ──► build_capsule() ──► synthesis ──► validation ──► export
(the cache) (no network) (per topic) (5 stages)
Adapters return list[Signal]. Nothing downstream knows which source
produced one. Live vs. fixture is chosen at construction, in the registry —
there is no if DEMO: in any pipeline path.
build_capsule() runs with the network disabled, structurally: capsule/
imports no adapter. It reads cached rows and nothing else.
A missing source is an empty slot, reported in coverage. Never fabricated. An archetype with no evidence gets a smaller quota, or zero.
What promptgen run <domain> actually does, in order.
1. Fetch — the only stage that touches the client's sources.
Connectors run in waves (see below), each returning list[Signal]. A signal is
one normalized fact: its verbatim text, the source that produced it, a
kind, a namespaced ref for provenance, and a weight of 1–10. Every field
is validated in Signal.__post_init__, so a malformed signal fails at the
adapter that made it rather than three stages later.
2. Restate — one gate for every pain source.
Issue titles are engineering artifacts ("Redis internal URL causes deployment
build to fail"), not things a buyer would say. restate_pains rewrites them as
buyer conditions. It is a pipeline stage, not adapter-internal, so a second
pain source added later clears the same bar automatically.
3. Persist — raw_signals is the cache layer.
Rows are upserted, then rows a source stopped reporting are swept — but only
for sources that actually succeeded, so a failed fetch can never delete good
cached data. Everything downstream reads Postgres, never an API.
4. Build the capsule — with the network structurally unavailable.
capsule/ imports no adapter, so build_capsule() cannot fetch. It reads
cached rows and produces: an evidence capsule as XML, derived topics, buyer
personas, and a directives block carrying per-topic archetype quotas and ICP
floors. Quotas are computed from evidence counts, so a slot with nothing in it
shrinks the archetypes it grounds — down to zero.
5. Synthesize — one model call per topic, fanned out concurrently. The static prefix is byte-identical for every client and sits behind a cache breakpoint; only the capsule and directives vary. Candidates stream back and are validated as they arrive, so a duplicate is caught on arrival rather than in a batch pass at the end.
6. Validate — five stages, then a sixth for the templates. See Validation. Every drop is attributed: which stage took it, which rule fired, and which sibling survived.
7. Export — the contract is checked at the boundary.
export_records verifies the field set and required values before writing,
because this file is the deliverable: everything upstream can be right and still
produce a record the eval service rejects.
A connector ("adapter") is one source of evidence. The whole design goal is that nothing downstream knows which source produced a signal — so adding one is a local change, and losing one degrades the output instead of breaking it.
Subclass Adapter (or HttpAdapter if you need HTTP) and declare four things:
| Member | What it is |
|---|---|
slot |
Which capsule slot this fills — a Slot enum member. |
source |
A bare token ("github", "pricing"). Must match Signal.source on everything you emit, and prefixes every ref. |
timeout |
Wall-clock ceiling, enforced by the runner. Default 10s. |
wave |
Fetch order. Default 0. |
Then implement one method:
async def fetch(self, client: Client) -> list[Signal]:
"""Return normalized signals. MUST NOT raise: on failure return []."""Use self.signal(...) to build them — it supplies source and kind from the
class, so an adapter cannot emit a signal that disagrees with the adapter that
made it.
HttpAdapter receives its client in __init__, not in fetch. That is
deliberate: the argument to fetch says which company, the constructor says
with what resources.
Wave 0 adapters read the network and nothing else. A wave-1 adapter is
primed with every wave-0 signal first, via prime(signals), because its
input is the other adapters' output — you cannot ask what buyers search for a
category before you know the category.
Within a wave everything runs concurrently. Across waves it is ordered. The
dependency is declared on the class, which is what keeps fetch(client)
unchanged: the ordering constraint lives in the runner, not in the signature.
fetch must not raise. The runner wraps every call anyway and turns each
outcome into one of four states, all of which appear in the coverage table:
| Status | Means |
|---|---|
ok |
Signals returned. |
empty |
Fetched and parsed, and the source legitimately has nothing. Raise NoDataError to say this explicitly. |
timeout |
Exceeded the adapter's own budget, or the 25s run budget. |
error |
FetchError (DNS, TLS, 4xx/5xx) or an unexpected exception, which is logged with a traceback. |
No adapter failure stops the run. A wave-0 adapter that errors contributes nothing to priming and does not block wave 1. Cancellation is never swallowed.
adapters/registry.py is the single place
that decides which adapter fills a slot. There is no demo, use_fixtures, or
mode parameter anywhere — the branch happens once, at wiring time, and
everything downstream receives an already-resolved tuple.
Equally important: the condition is never a mode flag. It is the presence of a
real thing — a credential, a /docs route, a public repo — which is what makes
the demo path and the production path the same code path.
The worked example is a review source for consumer brands, which is the gap the
coverage table names for nike.com and patagonia.com.
1. Write the adapter.
# promptgen/adapters/reviews.py
class PublicReviewAdapter(HttpAdapter):
slot = Slot.PAIN_POINTS # routes to community_pain_points
source = "reviews"
timeout = 12.0
async def fetch(self, client: Client) -> list[Signal]:
found = await try_fetch(self._http, client.url("reviews"), expect=HTML_TYPES)
if found is None:
return [] # honest empty slot, reported as such
return [
self.signal(complaint, entity="review", ident=review_id, weight=6)
for review_id, complaint in parse(found.text)
]2. Register it — one line in build_adapters():
GitHubIssuesAdapter(http),
PublicReviewAdapter(http), # the non-tech counterpart3. That is the whole wiring. Everything else already works, because nothing downstream is aware of sources:
- signals persist and cache like any other,
- they pass the same pain-restatement gate (step 2 above), because it is a pipeline stage rather than something GitHub's adapter owns,
- they raise
community_pain_pointscoverage, which raises the Implicit quota, - and they appear in the coverage table under their own
sourcename.
4. Test it offline. Adapters are constructed with their HTTP client, so a
test passes httpx.MockTransport and needs no network — see
tests/test_http.py and the fixtures in
tests/fixtures/.
Routing downstream is by Signal.kind, not by Adapter.slot, so one
adapter can feed two slots — the pricing adapter already emits both persona
and capability signals. Emit the kind you mean and it lands in the right place.
To add a new slot rather than fill an existing one: add a Slot member and
its SLOT_KINDS entry in adapters/base.py, give
it a cap in capsule/builder.py, and decide in
capsule/directives.py which archetypes it
grounds. ARCHETYPE_EVIDENCE is the table that maps evidence to quota, and it
is the reason a new source automatically buys more prompts rather than needing
the generator changed.
The coverage report exists to answer "which source should this customer connect next", per client. The four that would fill the most:
| Source | Fills | Unlocks |
|---|---|---|
| Support inbox / helpdesk | pain_points |
Implicit, for any company without a public issue tracker |
| Public reviews | pain_points |
the same, for consumer brands |
| Search Console | search_intent |
real query volume instead of autocomplete |
| Docs analytics / site search | capabilities |
which capabilities buyers actually look for |
| Slot | Source |
|---|---|
identity |
homepage: llms.txt → ld+json → og:* → <h1> |
official_capabilities |
docs tree, 6-tier discovery |
user_search_intent |
Google autocomplete, three seed families |
competitive_landscape |
autocomplete "<brand> vs", filtered by silence / reciprocity / peer graph |
community_pain_points |
GitHub issues, restated as buyer conditions |
buyer_personas |
client config, else pricing tiers |
| Stage | Checks |
|---|---|
| 1 schema | fields, topic resolves, persona label resolves, per-archetype word caps (25 Explicit / 35 the rest), ≤2 constraints, verbatim provenance |
| 2 brand leak | Implicit / Command / Multi-Turn must not name the brand |
| 3 quality judge | not built |
| 4 diversity | lexical cosine ≥ 0.62 drops near-paraphrases. Not embeddings — no provider is configured, so the threshold is deliberately lower than the spec's semantic 0.85. |
| 5 set level | per-topic quotas, ICP floors with reported shortfalls, 20-record ceiling. The ceiling runs after the floor and can undo its backfill; the run reports when it does. |
Sentiment audits (Evaluate {brand} on {topic}) and category head terms are
built in Python, not asked of a model — that is what makes them byte-stable and
so worth charting month to month. The cost is that they pass through none of
the five stages above: a template pastes whatever string it is handed straight
into the export.
validation/synthesized.py is their
gate. It refuses a label that is a docs artifact (Agent setup workflow (skill.md)), a list tail (auth more), or an instruction rather than a
capability area (Authenticate, Choose project goal).
Every rule was written against a string that actually reached an export, and
each is checked against every cached client for false positives — Deploy hooks
is a real Render feature and has to survive, which is why a leading verb
proves nothing and only a much narrower list applies in that position. A dropped
label is logged, never silent.
ruff check . # lint FIRST: pytest does not exercise the CLI's
# database paths and stays green through a missing
# import there. Ruff does not.
mypy promptgen scripts # strict, and clean
pytest -q # 312 tests, ~2sCI runs all three on every push against a real Postgres service. No test reaches the network — everything except synthesis is deterministic and testable offline.
Two things skip rather than fail, so the suite is useful on any machine:
- Without Postgres, the database tests skip. The rest still run. (With it,
the suite creates and drops a throwaway
promptgen_testdatabase.) - Without
ANTHROPIC_API_KEY, one test skips — the static prefix's token budget, which has to be measured with the real tokenizer rather than estimated at chars/4. So a clean run reports311 passed, 1 skipped, and that is what CI shows.
Each needs a live DATABASE_URL; only the last one spends money.
python scripts/reconcile_run.py /tmp/run.out 66 # DB rows vs the run's own accounting
python scripts/cross_run_stability.py 59 60 61 64 65 66 # is the set consistent across runs?
python scripts/answerability.py 66 # do the prompts elicit a recommendation?| Path | What lives there |
|---|---|
promptgen/adapters/ |
One module per source. Each returns list[Signal] and nothing more. |
promptgen/capsule/ |
Cache → capsule XML + generation directives. Imports no adapter. |
promptgen/synthesis/ |
Model clients, and master-prompt.md — the static cacheable prefix, loaded verbatim. |
promptgen/validation/ |
The five stages: per-candidate (schema, brand_leak) and set-level (setlevel). |
promptgen/pipeline.py |
Streams candidates through stages 2 and 4, then applies stage 5. Prints nothing. |
promptgen/reporting.py |
Everything the CLI prints. Renders finished values; computes none. |
promptgen/cli.py |
Typer commands and the order they run in. |
migrations/ |
Numbered SQL, applied by promptgen migrate. |
clients/ |
Optional per-client ICP configs. |
exports/ |
Generated deliverables. Three samples committed. |
docs/ |
STATE.md, the brief, and the working notes the code cites. |
no cached signals — run: promptgen signals <domain>
capsule and generate read from the cache and never fetch. Run
promptgen signals <domain> first, or use promptgen run, which does it for
you.
Anthropic credit balance exhausted
Top up, or re-run with --provider nim to use the free endpoint. Per-topic work
already cached is not repeated.
export refused: partial run
A topic call failed every retry, so the export would be short for a reason that
has nothing to do with the client. Re-run it, or pass --force-partial if a
short export is genuinely what you want.
Connection refused, or Postgres unavailable
Check DATABASE_URL points at where Postgres actually is. The compose file uses
port 55432, not 5432, so it cannot collide with a system Postgres — and
.env.example ships the local DSN by default, so switching to Docker means
switching that line too.
promptgen: command not found
Either source .venv/bin/activate, or prefix commands with uv run.
Pending migrations
promptgen status lists applied vs pending; promptgen migrate applies them.
docs/STATE.md — settled findings with the file and
function establishing each, the open decisions with their arithmetic, what is
measured versus what is asserted, and the validation experiment scoped
but not run.
Read it before changing anything about quotas, ranking, or the prompt spec: several plausible-looking changes are already known not to move shipped output.
Also in docs/: the original design brief, the
implementation plan, and the
build guide whose per-slot caps the code cites by name.