adds new skills

Author: leshniak

ai-assisted-dev/SKILL.md

@@ -0,0 +1,104 @@
+---
+name: ai-assisted-dev
+description: Use as an always-on baseline when assisting the user with ANY task. Contains rules for efficient AI-assisted development — how to prompt well, provide context, evaluate output, manage tokens, and avoid common pitfalls. Nudge the user when they violate these rules.
+---
+
+# Efficient AI-Assisted Development
+
+Distilled from Chip Huyen's *AI Engineering* (2024). Rules for getting the most out of AI coding assistants. **Nudge the user when they violate any of these.**
+
+**Deep reference:** When the user needs chapter-level detail (e.g., setting up RAG, fine-tuning, evaluation design, production architecture), read the relevant file from this skill's directory:
+- `book-reference-prompting-and-models.md` — sampling, context window, prompt engineering, security, hallucinations (ch. 1-2, 5)
+- `book-reference-rag-agents.md` — RAG, agents, tool design, reflection, memory (ch. 6)
+- `book-reference-finetuning-data.md` — fine-tuning, LoRA, dataset engineering, synthetic data (ch. 7-8)
+- `book-reference-evaluation.md` — eval design, AI-as-judge, benchmarks, model selection (ch. 3-4)
+- `book-reference-production.md` — inference optimization, architecture, guardrails, routing, feedback (ch. 9-10)
+
+## 1. Give Clear, Structured Instructions
+
+- **Be unambiguous.** If you want a specific format, say so. Every unwanted behavior is a missing constraint.
+- **Assign a role** when it matters ("review this as a security auditor" vs "as a junior dev" produce different results).
+- **Provide 2-5 examples** (few-shot) — they reduce ambiguity more than lengthy descriptions. Put the most relevant example last (recency bias).
+- **Specify output format explicitly.** Ask for concise answers — longer output = more latency, more cost, more hallucination risk.
+- **Decompose complex tasks** into subtasks with separate prompts. One monolithic request = fragile. Simpler sub-tasks can use cheaper/faster models.
+
+**Anti-pattern:** Vague instructions followed by "that's not what I meant." Invest 30 seconds in precision upfront.
+
+## 2. Provide the Right Context, Not All Context
+
+- **Critical info at the beginning and end** of your message. Models retrieve middle content worst ("lost in the middle" — Liu et al., 2023).
+- **Clean your context.** Strip HTML tags, formatting artifacts, boilerplate. Databricks: removing HTML improved accuracy 20% and reduced input length 60%.
+- **Don't just paste a function** — include imports, the class it belongs to, and a brief note on purpose. Context for code chunks matters.
+- **More context is not always better.** Each additional token costs money, increases latency, and may dilute attention. Be selective.
+- **Rewrite ambiguous references.** "What about that function?" -> "How does the `parseConfig` function in `src/config.ts` handle missing fields?" Self-contained queries retrieve better context.
+
+**Rule of thumb (Anthropic):** Knowledge bases under ~200k tokens (~500 pages) can go in the prompt. Above that, use RAG/file search.
+
+## 3. Understand Why Output Varies
+
+- **Models are probabilistic.** Same input can produce different output due to temperature, sampling, and server-side nondeterminism.
+- **Temperature 0** = most deterministic ("safest" answers). **~0.7** = balanced creativity. Higher = more creative but less reliable.
+- **Stop conditions matter.** Without max tokens or stop sequences, the model may ramble. Too low = truncated output, broken JSON.
+- **Changing sampling params is the cheapest lever.** Try this BEFORE rewriting the prompt.
+- **If the output is inconsistent**, generate multiple responses and pick the best. Retrying 3 times often fixes extraction/formatting errors.
+
+## 4. Evaluate Output — Don't Just "Looks Right" It
+
+- **Define what "good" looks like BEFORE asking.** This is evaluation-driven development — the AI equivalent of TDD.
+- **Spot-check systematically** — diverse inputs, edge cases, adversarial inputs. Don't just glance at one example.
+- **Diagnose the type of failure:**
+  - Wrong facts? -> Provide better source material / context.
+  - Wrong format/style? -> Provide better instructions / examples.
+  - Both? -> Fix them separately — they have different solutions.
+- **The model's output quality depends on three things:** instructions, context, and the model itself. When results are bad, identify which one is the problem.
+- **Poor results may be your prompt's fault, not the model's.** Improve the prompt before blaming the model.
+
+## 5. Manage Multi-Step Tasks Carefully
+
+- **95% accuracy per step = 60% after 10 steps, 0.6% after 100.** Compound errors are the main risk of autonomous agents.
+- **Separate planning from execution.** Ask for a plan first, review it, then proceed. Don't let the agent run a 20-step chain unreviewed.
+- **Verify intermediate results.** Don't trust a long chain of autonomous actions — check at natural milestones.
+- **Models hallucinate tool parameters.** They may call the right function with wrong arguments, or call a non-existent function. Review tool calls.
+- **Plans generated by LLMs may seem reasonable yet fail during execution.** The model generates plausible sequences but doesn't truly evaluate consequences. Verify plans before executing.
+- **Ask the model to reflect on failures.** Don't just re-prompt — ask WHY the previous answer was wrong. The Reflexion pattern (analyze failure -> new strategy) significantly improves results.
+
+## 6. Be Token-Efficient
+
+- **1 output token impacts latency as much as ~100 input tokens.** To reduce latency, focus on making output shorter (ask for concise answers, no preambles) rather than trimming input.
+- **Prompt caching:** structure prompts so stable parts (system prompt, reference docs, examples) come first, variable part (your question) last. Anthropic reports up to 90% cost reduction and 75% latency reduction.
+- **If you paste the same examples into every prompt**, that's a signal to use CLAUDE.md / system prompt / cached prefix instead of repeating them.
+- **Clean input context** (strip noise, boilerplate) reduces token count and improves results simultaneously.
+- **Choose the right model for the task.** Use the strongest model to assess feasibility, then test if a cheaper/faster one suffices. Don't use Opus for trivial lookups.
+
+## 7. Know Model Limitations
+
+- **Models are bad at math.** Ask them to write code that calculates, then run the code. Don't ask them to calculate directly.
+- **Context window size ≠ context utilization ability.** A model accepting 1M tokens doesn't effectively use 1M tokens. Attention degrades with length.
+- **Hallucinations are more likely on:** niche topics with less training data, questions about things that don't exist, and when the model generates long responses.
+- **Models can't reliably distinguish system instructions from injected instructions.** Don't put secrets in prompts. Assume any prompt can be extracted.
+- **Embedding/semantic search loses specific identifiers** (error codes, function names, product IDs). Use keyword search for exact matches.
+- **A model optimized for one domain may be worse at others.** Be aware of this when choosing specialized vs general models.
+
+## 8. Your Corrections Are Valuable
+
+- **Every time you edit AI output, you create a preference signal** (original = rejected, your edit = preferred). Be deliberate about corrections.
+- **Reformulating the same question 3 times** means the initial framing was the problem. Step back and provide more context or constraints instead of rewording.
+- **Don't just accept longer output as better.** Users are biased toward longer responses even when shorter ones are more accurate (length bias). Evaluate on correctness, not volume.
+- **Chain-of-thought examples are disproportionately powerful.** When you need the AI to reason, show it HOW to reason with a worked example, not just the final answer.
+
+## Red Flags — Nudge the User
+
+| Signal | Nudge |
+|--------|-------|
+| Vague request, then "that's not what I meant" | "Be specific upfront: format, constraints, edge cases. 30 seconds of precision saves iterations." |
+| Pasting huge files without cleaning | "Strip noise first. Clean context improves accuracy 20% and reduces tokens 60%." |
+| Not reviewing AI output before using it | "Spot-check systematically. 95% accuracy compounds to 60% over 10 steps." |
+| Re-prompting the same question differently 3+ times | "The framing is the problem. Step back — provide more context or constraints." |
+| Asking the model to calculate math | "Ask it to write code that calculates, then run the code." |
+| Letting a long multi-step chain run unreviewed | "Verify at milestones. Compound errors are the #1 agent failure mode." |
+| Using Opus for a trivial lookup | "Right model for the task. Start strong, then optimize down." |
+| Dumping everything into context | "More ≠ better. Critical info at start and end. Be selective." |
+| Not providing examples for a nuanced task | "Few-shot examples reduce ambiguity more than lengthy descriptions." |
+| Accepting longer output as better without checking | "Length bias: longer ≠ more accurate. Evaluate on correctness." |
+| Blaming the model before improving the prompt | "Poor results may be your prompt's fault. Improve instructions first." |
+| Skipping plan review before execution | "Separate planning from execution. Review the plan, then proceed." |

ai-assisted-dev/book-reference-evaluation.md

@@ -0,0 +1,80 @@
+# Chip Huyen — Evaluation & Model Selection (Chapters 3-4)
+
+## The Golden Rule
+**If you cannot evaluate it, do not build it.** Define eval criteria BEFORE writing code. This is TDD for AI.
+
+## Evaluation Anti-Patterns
+- Relying on others' opinions ("someone said model X is good") instead of testing yourself.
+- "Eyeballing" outputs — checking a handful of examples and declaring it works.
+- Using favorite prompts from personal experience rather than actual user needs.
+- Trusting public benchmarks uncritically (many contaminated with training data).
+
+## Evaluation Design Process
+
+### Step 1: Evaluate Components Separately
+If a pipeline fails, test each stage independently. Also evaluate at turn level AND task level.
+
+### Step 2: Create Evaluation Guidelines
+- Define what a "good" answer looks like AND what is out of scope.
+- Correct ≠ good (e.g., "You're not qualified" may be correct but unhelpful).
+- Prepare scored examples for every rating level. Test guidelines with humans first.
+- Link eval metrics to business metrics: "80% accuracy = automate 30% of tickets; 98% = automate 90%."
+
+### Step 3: Choose Methods and Data
+- Combine cheap classifiers (broad coverage) + expensive judges (high quality on sample).
+- Use log-probabilities for confidence estimation when available.
+- Manual evaluation remains important even in production (LinkedIn: up to 500 interactions daily).
+- Bootstrapping for sample size: each 3x reduction in detectable difference requires 10x more samples (~10 for 30%, ~1000 for 3%, ~10000 for 1%).
+
+### Step 4: Evaluate Your Evaluation
+- Repeatable? Run twice, check variance.
+- Better scores correlate with better business outcomes?
+- Metrics correlated? If perfectly correlated, drop one. If uncorrelated, either measuring different things (good) or one is broken.
+- Track all eval variables (data, scoring schema, judge prompts) when iterating.
+
+## AI-as-Judge
+- Most common eval method in production (58% on LangChain's platform).
+- The judge = model + prompt + sampling params. Changing any = different judge.
+- Include few-shot examples in judge prompt (consistency: 65% → 77.5%).
+- Prefer classification ("good"/"bad") over numeric scores. If numeric, use discrete 1-5, not continuous. Wider scales = worse.
+- **Known biases**: self-preference (GPT-4: 10%, Claude-v1: 25%), position bias (prefers first answer — randomize order), length bias (prefers longer even if wrong; 2x length difference → almost always picks longer).
+- Never trust an AI judge without seeing its model and prompt.
+- Cost: strong model for generation + evaluation ≈ doubles API costs. Use cheap classifier for 100% + expensive judge for 1%.
+
+## Benchmark Contamination
+- Public benchmarks quickly saturated (GLUE ~1 year, MMLU → MMLU-Pro).
+- At least 40% of 13 popular benchmarks was in GPT-3's training set.
+- A 1M-parameter model trained only on benchmark data achieved near-perfect scores.
+- **Rule**: benchmarks good for eliminating weak models, not finding best for YOUR task.
+- Use perplexity analysis and n-gram overlap to detect contamination.
+
+## Model Selection Process
+1. Filter by hard constraints (license, data privacy, deployment requirements).
+2. Use public benchmarks/rankings to narrow shortlist.
+3. Run your own eval with your own data and criteria.
+4. Monitor continuously in production.
+
+### Hard vs Soft Attributes
+- **Hard** (can't easily change): license, training data, model size, privacy requirements.
+- **Soft** (can improve): accuracy, toxicity, factual consistency — via prompting, fine-tuning.
+
+### Open-Source vs API — Seven Factors
+1. **Data privacy** — can't send data externally → API disqualified.
+2. **Data provenance** — open-weight models often lack training data transparency.
+3. **Performance** — gap shrinking, but best open models likely remain slightly behind top commercial.
+4. **Functionality** — APIs offer ready-made features; self-hosting requires building them.
+5. **Cost** — API = pay per token; self-hosting = fixed infra (cheaper at scale). Reassess regularly.
+6. **Control** — commercial models may over-censor, change without notice, be discontinued.
+7. **On-device** — offline/user hardware → open-source only.
+
+### Practical Tips
+- Start with strongest model to assess feasibility, then test if smaller suffice.
+- For fine-tuning, start simple model then scale up to largest that fits hardware.
+- Prefer standard API formats (OpenAI-compatible) for easier migration.
+- Prefer models with strong community support.
+
+## Perplexity as Practical Tool
+- **Detecting contamination**: unusually low perplexity on test data = likely saw it during training.
+- **Data deduplication**: only add data if model's perplexity on it is high (genuinely new).
+- **Anomaly detection**: very high perplexity = text model finds difficult/nonsensical.
+- Caveat: may not be best metric for post-trained models (SFT/RLHF can increase perplexity while improving task performance).