Skip to content

About

Trial Week

Resources

Stars

1 star

Watchers

0 watching

Forks

Repository files navigation

Manicule Prompt Gen

CI Python 3.11

Give it a client's domain, get 10–20 tracking prompts you can run against AI search engines to measure whether that client shows up.

To know if ChatGPT or Perplexity recommends your product, you need questions to ask them — the questions your actual buyers would type. Writing those by hand takes a strategist a day per client and the result is unrepeatable. This generates them from evidence: the client's own docs, pricing tiers, GitHub issues and real search-autocomplete data. Every prompt traces back to a verbatim string in that client's evidence capsule. Nothing is invented.

promptgen run mastra.ai

One command, cold start — no prior database rows, no client config, no manual ordering of stages. It fetches, restates, derives buyer personas, builds the capsule, generates, validates through five stages, and writes exports/mastra.ai.prompts.json.


Why track this

AI chat is a product-discovery surface at search-engine scale, and OpenAI has now measured it. How People Use ChatGPT (Chatterji, Cunningham, Deming, Hitzig, Ong, Shan & Wadman — OpenAI, Duke and Harvard, September 2025) classified a representative sample of consumer conversations and reports:

  • 700 million users sending 18 billion messages a week as of July 2025 — around 10% of the world's adult population.
  • ~49% of messages are Asking: seeking information or clarification to inform a decision. Doing is 40%, Expressing 11% — and Asking is growing faster than the others.
  • Practical Guidance and Seeking Information are together almost half of all usage. The paper describes Seeking Information as covering searches for "information about people, current events, products, and recipes," and "appears to be a very close substitute for web search."
  • The paper's own summary: ChatGPT "provides economic value through decision support."

If half of that traffic is people asking what to use, then whether a client surfaces in the answer is a measurable fact with revenue attached — and it cannot be measured without a fixed, repeatable set of questions to ask each month. Producing that set, grounded in evidence rather than guesswork, is what this tool does.


Contents


Quickstart

From a clean clone to a finished export:

git clone git@github.com:Twix1288/Manicule_PromtpGeneratorCLI.git
cd Manicule_PromtpGeneratorCLI

uv sync                             # installs into .venv
docker compose up -d                # Postgres on :55432
cp .env.example .env                # then set ANTHROPIC_API_KEY

uv run promptgen migrate            # create the schema
uv run promptgen run mastra.ai      # → exports/mastra.ai.prompts.json

uv sync puts a promptgen executable on the venv's path, so after source .venv/bin/activate you can drop the uv run prefix. Every example below omits it.

No API key? promptgen signals mastra.ai and promptgen capsule mastra.ai run in full without one — everything except model synthesis is deterministic Python. That is the fastest way to see what the system knows about a client.


What you get

A JSON array of records in the eval service's exact schema — 11 fields, no more, checked at the boundary before the file is written:

{
  "ID": "6c4d3196-6547-442d-8620-4a776c334243",
  "Topic": "Agents",
  "Prompt": "Best agent framework for enterprise VPC deployments with predictable cost — how does Mastra compare to LangGraph and Vercel AI SDK?",
  "Tags": "enterprise-comparison, vpc, data-residency, langgraph, vercel-ai-sdk",
  "Regions": "Global",
  "Language": "en",
  "Platforms": "chatgpt, perplexity, google-ai-overviews",
  "Personas": "Enterprise",
  "Analysis Types": "Visibility, Sentiment",
  "Created": "9/4/2026, 08:29 PM",
  "Updated": "9/4/2026, 08:29 PM"
}

The set is spread across four archetypes, so you measure different failure modes rather than the same one twenty times:

Archetype Asks like Catches
Explicit "How does Mastra compare to LangGraph on X?" Whether you win a head-to-head you were named in
Implicit "Our agent stops after one tool call — what do people use?" Whether you surface for a symptom, unprompted. Must never name the brand.
Command "Write a TypeScript workflow that pauses for human approval…" Whether you are the default in generated code
Multi-Turn "[Context: top 3 frameworks…] Which of those runs in our VPC?" Whether you drop out of a shortlist you were just on

Plus category-tracking templates (Evaluate Mastra on Agents) built in Python, not asked of a model — so the string is byte-stable across runs and gives you a real time series. A prompt the model rewords each month charts a different question each month.

Three sample exports are committed in exports/ if you want to read output without running anything.


Who this works for

Built for B2B SaaS and developer tools. It runs against any domain without crashing, but output quality tracks how much public, structured evidence the company publishes — and for a consumer brand, half the evidence sources return nothing.

Signals gathered per capsule slot, one run each against eight real domains (2–4 September 2026; n=1 per client, so indicative rather than a benchmark):

Client identity capability query competitor pain persona
railway.app 10 12 24 5 8 4
vercel.com 10 12 24 4 8 3
mastra.ai 9 12 24 3 8 4
stripe.com 9 12 24 2 6 2
render.com 9 9 24 5 3 —
figma.com 6 — 24 5 — 4
nike.com 6 — 24 6 — —
patagonia.com 1 — 24 4 — —

What holds everywhere. query is 24 for every client — Google autocomplete works for any brand with search volume — and competitor holds up too. So every company gets a usable topic axis and a usable archetype spread.

What needs a developer-shaped company. Three sources go empty as you move down the table — capability and pain at Figma, persona one row later at Nike:

Slot Needs Empty when
capability a docs site the crawler recognizes there are no developer docs
pain a public GitHub repo there is no public issue tracker
persona a pricing page with named tiers pricing is not tiered, or not public

identity degrades separately, on brand ambiguity: patagonia.com scores 1 because the name is a region before it is a company, and the competitor filters spend their effort discarding national parks.

What that costs in practice

nike.com still shipped 20 records, and they read like real buyer questions — "My arches ache and my ankles roll inward after three miles, which stability running shoes should I shortlist?" But every one of them carries the same persona, General Buyer, because there were no pricing tiers to derive segments from. The ICP axis — one of the three this tool is built on — collapses to a single segment for a consumer brand. That is a thinner product, not an equivalent one.

  • Recoverable: the persona axis. Supply clients/{domain}.personas.json and it comes back — which is the intended path anyway, see Client config.
  • Not recoverable from config: capability and pain. Those need a source connected — a support inbox, a reviews feed, a docs-analytics export. Naming which one, per client, is what the coverage report is for.

The floor

Run against a domain with almost nothing (example.com, 3 signals, 1 kind):

topics 0 · category 0 · 0 capabilities
quotas {'Explicit': 0, 'Command': 0, 'Implicit': 0, 'Multi-Turn': 0}
persona: general-buyer  [generic_fallback]
exit=0

Zero topics, zero quotas, the persona flagged generic_fallback, exit 0. No crash, and no invented market to fill the gap. Shipping nothing is a correct answer for a client with nothing to ground a prompt in — the one thing the system will not do is fabricate evidence to hit a number.


Requirements

Python 3.11 (pinned: >=3.11,<3.12)
uv dependency management — curl -LsSf https://astral.sh/uv/install.sh | sh
PostgreSQL 16 recommended. Use the bundled docker-compose.yml, or any local instance.
Anthropic API key Only for generate and run. Everything else works without it.

Setup

uv sync

Then a database. Either the bundled compose file:

docker compose up -d       # postgres:16 on localhost:55432

…or a local Postgres:

createdb promptgen

Then configure and create the schema:

cp .env.example .env       # every variable is documented inline
promptgen migrate          # applies migrations/*.sql in order
promptgen status           # shows applied vs pending

Migrations are numbered SQL files applied inside a transaction, with a checksum recorded per file — re-running is a no-op, and an edited migration is refused rather than silently skipped.


Configuration

All configuration is environment variables, read from .env. .env is gitignored; never commit real credentials.

Variable Required Purpose
DATABASE_URL yes Postgres DSN, asyncpg driver. Every stage except synthesis runs off cached rows here.
ANTHROPIC_API_KEY for synthesis Default provider. Also powers pain-point restatement at ingest.
NIM_API_KEY no Free OpenAI-compatible endpoint, selected with --provider nim.
NIM_MODEL no Defaults to deepseek-ai/deepseek-v4-pro-0813.
GITHUB_TOKEN no Raises the GitHub API rate limit. The issues adapter works without it.
SQL_ECHO no Set to 1 to log every SQL statement.

Without an Anthropic key, signals and capsule still run in full — only archetype synthesis needs a model. The category-tracking templates are built in Python and ship even when every model call fails.


Commands

run is the composite. The granular commands are how you debug one stage, and each reads what the previous one cached.

promptgen run <domain>

Everything, from a cold database. Fetch → restate → derive personas → build capsule → generate → validate → export.

promptgen run mastra.ai
promptgen run mastra.ai --total 15 --refresh --timings
Flag Default Meaning
--total <int> 20 Target prompt count. Drives per-topic quotas and ICP floors.
--model <str> claude-opus-5 Model id for synthesis.
--provider <str> anthropic anthropic ships the deliverable; nim is a free endpoint for iteration.
--effort <str> low low | default | medium | high.
--refresh off Re-fetch signals even if the 24h cache is warm.
--timings off Per-call latency table, split into prefill and decode.
-v, --verbose off INFO-level logging.

promptgen signals <domain>

Fetch, normalize, gate and persist a client's raw signals. This is the only stage that fetches from the client's own sources — everything downstream reads the cache.

promptgen signals mastra.ai
promptgen signals mastra.ai --no-store --json

--store / --no-store (default store) · --json · -v, --verbose

promptgen capsule <domain>

Build the evidence capsule and generation directives from cache. Never fetches — run signals first. Needs no API key, so it is the cheapest way to inspect what the system knows about a client.

promptgen capsule mastra.ai
promptgen capsule mastra.ai --xml-only > capsule.xml

--total <int> (default 20) · --xml-only

promptgen generate <domain>

Build the capsule from cache, then synthesize and validate. Needs a model.

promptgen generate mastra.ai
promptgen generate mastra.ai --provider nim --no-store   # free, no writes
promptgen generate mastra.ai --review                    # human read harness
Flag Default Meaning
--total <int> 20 Target prompt count.
--model <str> claude-opus-5 Model id.
--provider <str> anthropic anthropic | nim.
--effort <str> low Reasoning effort.
--store / --no-store store Persist the capsule and every candidate.
--json off Machine-readable output instead of tables.
--review off Read harness: ask-shape distribution, brand leaks, vendor voice.
--timings off Per-call latency table.
--force-partial off Export even though a topic call failed every retry.
-v, --verbose off INFO-level logging.

promptgen personas import <path>

Load a client-supplied ICP config. See Client config.

promptgen personas import clients/vercel.com.personas.json

promptgen migrate · promptgen status

Apply pending migrations, or show applied vs pending.


Reading a run

run prints five sections, in this order:

  1. Coverage — which capsule slots filled, from which source, and how many signals survived the pipeline (not how many were fetched). Cells that came back empty are named, along with the archetypes those empty cells gated off, per topic. This is the generator-problem-vs-data-problem answer, and it is the artifact that tells a customer which source to connect next.
  2. Topics — derived capability areas with evidence counts and per-topic archetype quotas.
  3. Personas — the buyer segments, and where they came from: a client config, or inferred from pricing tiers.
  4. Delivery accounting — shipped vs. target, with every deduction named and attributed: rejected at stage 1, brand leaks at stage 2, near-duplicates at stage 4 (with the colliding string and the surviving sibling), quota cuts and ICP-floor backfills at stage 5, and anything over the 20-record ceiling.
  5. Export path.

A count that reads as success while describing a shortfall is treated as a defect in the same class as a wrong prompt. If the number is short, the run says exactly where each missing prompt went.


Exit codes

Code Meaning
0 Success — including a below-floor run. Shipping fewer grounded prompts is correct for a thin client, and failing CI over it would make correct degradation look broken.
1 A crash, no cached signals, an export that fails the format contract, or a partial run.
2 No usable model credential.

A partial run is one where a topic call failed every retry. It is persisted as partial, refuses to export without --force-partial, and prints which topics died. A failed call and a genuinely thin client produce the same-looking output, so they must not produce the same exit code — that is the one property a measurement system cannot give up.


Client config (optional)

Personas are an input. A client knows who it sells to; inferring from a pricing page is the fallback, not the plan.

clients/{domain}.personas.json

See clients/_template.personas.json — every field is documented inline. Import it with promptgen personas import.

Without one, personas are derived from the client's pricing tiers and the run says so. Nothing fails. Pricing tiers supply each persona's vocabulary either way: the features gated behind a tier are the words that segment uses.


How it works

adapters ──► raw_signals ──► build_capsule() ──► synthesis ──► validation ──► export
             (the cache)     (no network)        (per topic)   (5 stages)

Adapters return list[Signal]. Nothing downstream knows which source produced one. Live vs. fixture is chosen at construction, in the registry — there is no if DEMO: in any pipeline path.

build_capsule() runs with the network disabled, structurally: capsule/ imports no adapter. It reads cached rows and nothing else.

A missing source is an empty slot, reported in coverage. Never fabricated. An archetype with no evidence gets a smaller quota, or zero.

Step by step

What promptgen run <domain> actually does, in order.

1. Fetch — the only stage that touches the client's sources. Connectors run in waves (see below), each returning list[Signal]. A signal is one normalized fact: its verbatim text, the source that produced it, a kind, a namespaced ref for provenance, and a weight of 1–10. Every field is validated in Signal.__post_init__, so a malformed signal fails at the adapter that made it rather than three stages later.

2. Restate — one gate for every pain source. Issue titles are engineering artifacts ("Redis internal URL causes deployment build to fail"), not things a buyer would say. restate_pains rewrites them as buyer conditions. It is a pipeline stage, not adapter-internal, so a second pain source added later clears the same bar automatically.

3. Persist — raw_signals is the cache layer. Rows are upserted, then rows a source stopped reporting are swept — but only for sources that actually succeeded, so a failed fetch can never delete good cached data. Everything downstream reads Postgres, never an API.

4. Build the capsule — with the network structurally unavailable. capsule/ imports no adapter, so build_capsule() cannot fetch. It reads cached rows and produces: an evidence capsule as XML, derived topics, buyer personas, and a directives block carrying per-topic archetype quotas and ICP floors. Quotas are computed from evidence counts, so a slot with nothing in it shrinks the archetypes it grounds — down to zero.

5. Synthesize — one model call per topic, fanned out concurrently. The static prefix is byte-identical for every client and sits behind a cache breakpoint; only the capsule and directives vary. Candidates stream back and are validated as they arrive, so a duplicate is caught on arrival rather than in a batch pass at the end.

6. Validate — five stages, then a sixth for the templates. See Validation. Every drop is attributed: which stage took it, which rule fired, and which sibling survived.

7. Export — the contract is checked at the boundary. export_records verifies the field set and required values before writing, because this file is the deliverable: everything upstream can be right and still produce a record the eval service rejects.


Connectors

A connector ("adapter") is one source of evidence. The whole design goal is that nothing downstream knows which source produced a signal — so adding one is a local change, and losing one degrades the output instead of breaking it.

The contract

Subclass Adapter (or HttpAdapter if you need HTTP) and declare four things:

Member What it is
slot Which capsule slot this fills — a Slot enum member.
source A bare token ("github", "pricing"). Must match Signal.source on everything you emit, and prefixes every ref.
timeout Wall-clock ceiling, enforced by the runner. Default 10s.
wave Fetch order. Default 0.

Then implement one method:

async def fetch(self, client: Client) -> list[Signal]:
    """Return normalized signals. MUST NOT raise: on failure return []."""

Use self.signal(...) to build them — it supplies source and kind from the class, so an adapter cannot emit a signal that disagrees with the adapter that made it.

HttpAdapter receives its client in __init__, not in fetch. That is deliberate: the argument to fetch says which company, the constructor says with what resources.

Waves

Wave 0 adapters read the network and nothing else. A wave-1 adapter is primed with every wave-0 signal first, via prime(signals), because its input is the other adapters' output — you cannot ask what buyers search for a category before you know the category.

Within a wave everything runs concurrently. Across waves it is ordered. The dependency is declared on the class, which is what keeps fetch(client) unchanged: the ordering constraint lives in the runner, not in the signature.

The failure model

fetch must not raise. The runner wraps every call anyway and turns each outcome into one of four states, all of which appear in the coverage table:

Status Means
ok Signals returned.
empty Fetched and parsed, and the source legitimately has nothing. Raise NoDataError to say this explicitly.
timeout Exceeded the adapter's own budget, or the 25s run budget.
error FetchError (DNS, TLS, 4xx/5xx) or an unexpected exception, which is logged with a traceback.

No adapter failure stops the run. A wave-0 adapter that errors contributes nothing to priming and does not block wave 1. Cancellation is never swallowed.

The registry is the only branch point

adapters/registry.py is the single place that decides which adapter fills a slot. There is no demo, use_fixtures, or mode parameter anywhere — the branch happens once, at wiring time, and everything downstream receives an already-resolved tuple.

Equally important: the condition is never a mode flag. It is the presence of a real thing — a credential, a /docs route, a public repo — which is what makes the demo path and the production path the same code path.


Adding a connector

The worked example is a review source for consumer brands, which is the gap the coverage table names for nike.com and patagonia.com.

1. Write the adapter.

# promptgen/adapters/reviews.py
class PublicReviewAdapter(HttpAdapter):
    slot = Slot.PAIN_POINTS      # routes to community_pain_points
    source = "reviews"
    timeout = 12.0

    async def fetch(self, client: Client) -> list[Signal]:
        found = await try_fetch(self._http, client.url("reviews"), expect=HTML_TYPES)
        if found is None:
            return []                      # honest empty slot, reported as such
        return [
            self.signal(complaint, entity="review", ident=review_id, weight=6)
            for review_id, complaint in parse(found.text)
        ]

2. Register it — one line in build_adapters():

GitHubIssuesAdapter(http),
PublicReviewAdapter(http),      # the non-tech counterpart

3. That is the whole wiring. Everything else already works, because nothing downstream is aware of sources:

  • signals persist and cache like any other,
  • they pass the same pain-restatement gate (step 2 above), because it is a pipeline stage rather than something GitHub's adapter owns,
  • they raise community_pain_points coverage, which raises the Implicit quota,
  • and they appear in the coverage table under their own source name.

4. Test it offline. Adapters are constructed with their HTTP client, so a test passes httpx.MockTransport and needs no network — see tests/test_http.py and the fixtures in tests/fixtures/.

Filling a different slot

Routing downstream is by Signal.kind, not by Adapter.slot, so one adapter can feed two slots — the pricing adapter already emits both persona and capability signals. Emit the kind you mean and it lands in the right place.

To add a new slot rather than fill an existing one: add a Slot member and its SLOT_KINDS entry in adapters/base.py, give it a cap in capsule/builder.py, and decide in capsule/directives.py which archetypes it grounds. ARCHETYPE_EVIDENCE is the table that maps evidence to quota, and it is the reason a new source automatically buys more prompts rather than needing the generator changed.

Named gaps

The coverage report exists to answer "which source should this customer connect next", per client. The four that would fill the most:

Source Fills Unlocks
Support inbox / helpdesk pain_points Implicit, for any company without a public issue tracker
Public reviews pain_points the same, for consumer brands
Search Console search_intent real query volume instead of autocomplete
Docs analytics / site search capabilities which capabilities buyers actually look for

The capsule slots

Slot Source
identity homepage: llms.txt → ld+json → og:* → <h1>
official_capabilities docs tree, 6-tier discovery
user_search_intent Google autocomplete, three seed families
competitive_landscape autocomplete "<brand> vs", filtered by silence / reciprocity / peer graph
community_pain_points GitHub issues, restated as buyer conditions
buyer_personas client config, else pricing tiers

Validation

Stage Checks
1 schema fields, topic resolves, persona label resolves, per-archetype word caps (25 Explicit / 35 the rest), ≤2 constraints, verbatim provenance
2 brand leak Implicit / Command / Multi-Turn must not name the brand
3 quality judge not built
4 diversity lexical cosine ≥ 0.62 drops near-paraphrases. Not embeddings — no provider is configured, so the threshold is deliberately lower than the spec's semantic 0.85.
5 set level per-topic quotas, ICP floors with reported shortfalls, 20-record ceiling. The ceiling runs after the floor and can undo its backfill; the run reports when it does.

The templates get a gate of their own

Sentiment audits (Evaluate {brand} on {topic}) and category head terms are built in Python, not asked of a model — that is what makes them byte-stable and so worth charting month to month. The cost is that they pass through none of the five stages above: a template pastes whatever string it is handed straight into the export.

validation/synthesized.py is their gate. It refuses a label that is a docs artifact (Agent setup workflow (skill.md)), a list tail (auth more), or an instruction rather than a capability area (Authenticate, Choose project goal).

Every rule was written against a string that actually reached an export, and each is checked against every cached client for false positives — Deploy hooks is a real Render feature and has to survive, which is why a leading verb proves nothing and only a much narrower list applies in that position. A dropped label is logged, never silent.


Development

ruff check .              # lint FIRST: pytest does not exercise the CLI's
                          # database paths and stays green through a missing
                          # import there. Ruff does not.
mypy promptgen scripts    # strict, and clean
pytest -q                 # 312 tests, ~2s

CI runs all three on every push against a real Postgres service. No test reaches the network — everything except synthesis is deterministic and testable offline.

Two things skip rather than fail, so the suite is useful on any machine:

  • Without Postgres, the database tests skip. The rest still run. (With it, the suite creates and drops a throwaway promptgen_test database.)
  • Without ANTHROPIC_API_KEY, one test skips — the static prefix's token budget, which has to be measured with the real tokenizer rather than estimated at chars/4. So a clean run reports 311 passed, 1 skipped, and that is what CI shows.

Analysis scripts

Each needs a live DATABASE_URL; only the last one spends money.

python scripts/reconcile_run.py /tmp/run.out 66            # DB rows vs the run's own accounting
python scripts/cross_run_stability.py 59 60 61 64 65 66    # is the set consistent across runs?
python scripts/answerability.py 66                         # do the prompts elicit a recommendation?

Repository layout

Path What lives there
promptgen/adapters/ One module per source. Each returns list[Signal] and nothing more.
promptgen/capsule/ Cache → capsule XML + generation directives. Imports no adapter.
promptgen/synthesis/ Model clients, and master-prompt.md — the static cacheable prefix, loaded verbatim.
promptgen/validation/ The five stages: per-candidate (schema, brand_leak) and set-level (setlevel).
promptgen/pipeline.py Streams candidates through stages 2 and 4, then applies stage 5. Prints nothing.
promptgen/reporting.py Everything the CLI prints. Renders finished values; computes none.
promptgen/cli.py Typer commands and the order they run in.
migrations/ Numbered SQL, applied by promptgen migrate.
clients/ Optional per-client ICP configs.
exports/ Generated deliverables. Three samples committed.
docs/ STATE.md, the brief, and the working notes the code cites.

Troubleshooting

no cached signals — run: promptgen signals <domain> capsule and generate read from the cache and never fetch. Run promptgen signals <domain> first, or use promptgen run, which does it for you.

Anthropic credit balance exhausted Top up, or re-run with --provider nim to use the free endpoint. Per-topic work already cached is not repeated.

export refused: partial run A topic call failed every retry, so the export would be short for a reason that has nothing to do with the client. Re-run it, or pass --force-partial if a short export is genuinely what you want.

Connection refused, or Postgres unavailable Check DATABASE_URL points at where Postgres actually is. The compose file uses port 55432, not 5432, so it cannot collide with a system Postgres — and .env.example ships the local DSN by default, so switching to Docker means switching that line too.

promptgen: command not found Either source .venv/bin/activate, or prefix commands with uv run.

Pending migrations promptgen status lists applied vs pending; promptgen migrate applies them.


Where to start reading

docs/STATE.md — settled findings with the file and function establishing each, the open decisions with their arithmetic, what is measured versus what is asserted, and the validation experiment scoped but not run.

Read it before changing anything about quotas, ranking, or the prompt spec: several plausible-looking changes are already known not to move shipped output.

Also in docs/: the original design brief, the implementation plan, and the build guide whose per-slot caps the code cites by name.

About

Trial Week

Resources

Stars

1 star

Watchers

0 watching

Forks

Releases

Packages

Contributors

Languages