Applied AI PM Evaluation Framework

Analyze Applied AI PM resumes against the 6-pillar framework - Get instant, consistent evaluations using OpenAI GPT-5, Anthropic Claude Sonnet 4.5, or Google Gemini 2.5 Pro.

Built for hiring teams evaluating AI PM candidates, recruiters standardizing their process, and especially for people from non-traditional backgrounds looking to transition into AI Product Management roles.

🎯 Why This Exists

"In disaster relief, you don't map safe routes and hide them. You share them."

This conversation between Aakash Gupta and Jaclyn Konzelmann (Google's Director of AI Product) truly inspired this project. Jaclyn's AI PM evaluation framework was so clear and rigorous that it immediately became obvious: this shouldn't be a private checklist.

This tool exists to give everyone—especially people from non-traditional backgrounds—an actionable guide on:

What to build to demonstrate AI PM capabilities
Where to focus your learning and projects
How to position yourself for AI PM roles
How hiring teams actually evaluate candidates

For Job Seekers: Use this to self-assess, identify gaps, and build projects that matter. For Hiring Teams: Use this to standardize evaluations and find builders, not just coordinators.

Built by @abe238 | Full framework at aipmframework.com

✨ Features

🎯 6-Pillar Framework - Evaluates Technical Skills, Product Thinking, AI/ML Knowledge, Communication, Strategic Thinking, and Execution
🤖 Latest AI Models - GPT-5, Claude Sonnet 4.5, Gemini 2.5 Pro (with cheaper alternatives)
🔄 Multiple Model Options - Choose from 8 different models across 3 providers
📄 Multiple File Formats - Supports PDF, DOC, and DOCX resume files
📊 Detailed Scoring - 0-10 scores per pillar with evidence and level assessment
📝 Beautiful Reports - Generates markdown and HTML outputs with clean design
⚡ Fast Analysis - Get results in 30 seconds vs 10+ minutes manual review
💰 Cost-Effective - $0.10-0.60 per resume depending on model choice
🎨 Professional Design - HTML reports styled with Stripe/Tailwind-inspired CSS

📦 Quick Start

Prerequisites

Python 3.7+ installed on your computer
Pandoc (optional, for .doc/.docx support) - Install with:
- macOS: brew install pandoc
- Linux: sudo apt-get install pandoc
- Windows: Download from pandoc.org
API key from ONE of these providers:
- OpenAI (GPT-5, GPT-5 Mini, GPT-4o)
- Anthropic (Claude Sonnet 4.5, Haiku 4.5, Opus 4.1)
- Google (Gemini 2.5 Pro, Gemini 2.5 Flash)

Installation & Setup

1. Download or clone this repository:

git clone https://github.com/abe238/aipm-resume-analyzer.git
cd aipm-resume-analyzer

2. Install dependencies:

macOS/Linux:

./scripts/install.sh

Windows:

pip install -r requirements.txt

3. ⚠️ REQUIRED: Create your .env file with API key

IMPORTANT: The .env file is hidden by default. Follow these steps carefully:

Option A: Copy from example (recommended)

# Copy the example file to create your .env
cp examples/example.env .env

# Then edit .env and add your API key
nano .env  # or use any text editor

Option B: Let the tool create it

# Run the analyzer once - it will create .env automatically
./analyze resume.pdf

# You'll see instructions showing where the .env file was created
# Then edit .env and add your API key

📁 Where is the .env file?

Exact location: aipm-resume-analyzer/.env (in the same folder as the analyze script)
Full path example: /Users/yourname/aipm-resume-analyzer/.env

🔍 Can't see the .env file?

The .env file starts with a dot (.), making it hidden by default:

macOS Finder: Press Cmd + Shift + . to show hidden files
macOS Terminal: Use ls -la instead of ls
Windows Explorer: Go to View → Show → Hidden items
Windows Command Prompt: Use dir /a instead of dir
Linux Terminal: Use ls -la instead of ls

✏️ Editing the .env file:

Open .env in any text editor and add your API key:

# For OpenAI (example)
OPENAI_API_KEY=sk-proj-your-actual-key-here

# OR for Anthropic
ANTHROPIC_API_KEY=sk-ant-your-actual-key-here

# OR for Google
GOOGLE_API_KEY=your-actual-key-here

# Default provider
DEFAULT_PROVIDER=openai

See examples/example.env for a complete template with all options.

4. You're ready! Run your first analysis:

🔑 API Setup (First Time Only)

Step 1: Get an API Key

Choose ONE provider and get your API key:

Option A: OpenAI (GPT-5) - Most Advanced Reasoning

Go to https://platform.openai.com/api-keys
Click "+ Create new secret key"
Name it "AI PM Resume Analyzer"
Copy the key (starts with sk-...)
Models: GPT-5 (default), GPT-5 Mini, GPT-4o
Cost: $0.15-0.50 per resume

Option B: Anthropic (Claude Sonnet 4.5) - Best for Complex Analysis

Go to https://console.anthropic.com/settings/keys
Click "Create Key"
Name it "Resume Analyzer"
Copy the key (starts with sk-ant-...)
Models: Claude Sonnet 4.5 (default), Claude Haiku 4.5, Claude Opus 4.1
Cost: $0.20-0.60 per resume

Option C: Google (Gemini 2.5 Pro) - Advanced Thinking

Go to https://aistudio.google.com/app/apikey
Click "Create API Key"
Select or create a project
Copy the key
Models: Gemini 2.5 Pro (default), Gemini 2.5 Flash
Cost: $0.10-0.30 per resume (often free tier available)

Step 2: Run the Analyzer (It Will Create .env File)

./bin/analyze resume.pdf

The tool will create a .env file and show you where to add your API key.

Step 3: Add Your API Key

Open the .env file (created automatically) and add your key:

# For OpenAI
OPENAI_API_KEY=sk-your-key-here

# OR for Anthropic
ANTHROPIC_API_KEY=sk-ant-your-key-here

# OR for Google
GOOGLE_API_KEY=your-key-here

# Default provider (optional)
DEFAULT_PROVIDER=openai

Save the file. Done! You're ready to analyze resumes.

🚀 Usage

Supported file formats: .pdf, .doc, .docx

Basic Analysis

# Analyze resume (uses default provider from .env)
./bin/analyze resume.pdf
# Or with Word documents:
./bin/analyze resume.docx
./bin/analyze resume.doc

Output files will be created in ./output/:

Candidate_Name_20241025_143022.md (Markdown report)
Candidate_Name_20241025_143022.html (HTML report)
Candidate_Name_20241025_143022.json (Raw JSON data)

Choose Specific Provider

# Use Claude
./bin/analyze resume.pdf --provider anthropic

# Use Gemini
./bin/analyze resume.pdf --provider google

# Use OpenAI (default)
./bin/analyze resume.pdf --provider openai

Choose Specific Model

Each provider has multiple models with different speed/cost tradeoffs:

# List all available models
./bin/analyze --list-models

# OpenAI models
./bin/analyze resume.pdf --provider openai --model gpt-5          # Best reasoning (default)
./bin/analyze resume.pdf --provider openai --model gpt-5-mini     # Faster, cheaper
./bin/analyze resume.pdf --provider openai --model gpt-4o         # Budget option

# Anthropic models
./bin/analyze resume.pdf --provider anthropic --model claude-sonnet-4-5-20250929  # Best (default)
./bin/analyze resume.pdf --provider anthropic --model claude-haiku-4-5            # Fast & cheap
./bin/analyze resume.pdf --provider anthropic --model claude-opus-4-1             # Most capable

# Google models
./bin/analyze resume.pdf --provider google --model gemini-2.5-pro    # Advanced (default)
./bin/analyze resume.pdf --provider google --model gemini-2.5-flash  # Fast & affordable

Custom Output Location

# Save to custom directory
./bin/analyze resume.pdf --output ./reports/

# Different formats
./bin/analyze resume.pdf --format markdown  # MD only
./bin/analyze resume.pdf --format html      # HTML only
./bin/analyze resume.pdf --format both      # Both (default)

Deep Analysis Mode (Maximum Feedback)

For the most comprehensive analysis, use --deep-analysis to run ALL available providers and get aggregated insights:

# Run analysis with all providers (requires API keys for all)
./bin/analyze resume.pdf --deep-analysis

What Deep Analysis Does:

✅ Runs analysis with all providers you have API keys for (GPT-5, Claude Sonnet 4.5, Gemini 2.5 Pro)
✅ Generates consensus scores showing agreement/disagreement across models
✅ Aggregates all strengths and concerns from each provider
✅ Shows detailed comparison table with min/max/avg scores per pillar
✅ Provides multiple perspectives on the same candidate

When to Use:

Making final hiring decisions
Screening senior/principal candidates
When you want maximum confidence in the evaluation
To identify areas where models agree (high confidence) vs. disagree (needs human judgment)

Cost: $0.45-$1.50 per resume (runs 3 analyses instead of 1)

Example Output:

Consensus Score: 52.3/60

Individual Scores:
- GPT-5: 53/60 - Strong Screen
- Claude Sonnet 4.5: 48/60 - Strong Screen
- Gemini 2.5 Pro: 56/60 - Strong Screen

Pillar Scores (showing agreement/disagreement):
Technical Skills: 8.0/10 (range: 7-9)
Communication: 9.3/10 (range: 9-10)  ← High consensus
AI/ML Knowledge: 7.7/10 (range: 6-9)  ← More variation

📊 Understanding the Output

Scoring System

Each resume gets scored across 6 pillars:

Pillar	Weight	What It Measures
Technical Skills	20%	Engineering background, coding, technical decision-making
Product Thinking	25%	User empathy, prioritization, product strategy, metrics
AI/ML Knowledge	20%	ML fundamentals, AI projects, responsible AI awareness
Communication	10%	Writing quality, stakeholder management, presentations
Strategic Thinking	15%	Vision setting, market analysis, long-term planning
Execution	10%	Shipping track record, project management, results

Score Ranges

Total Score	Decision	Action
8.0 - 10.0	Strong Screen	Top candidate - prioritize interview
6.5 - 7.9	Screen	Solid candidate - invite to interview
5.0 - 6.4	Maybe	Borderline - use additional criteria
Below 5.0	No Screen	Does not meet bar

Sample Output

Markdown Report (.md file):

# AI PM Resume Analysis: John Smith

**Total Score**: 42/60 (70/100 weighted)
**Decision**: **Screen**

## Top 3 Strengths
1. Strong product management background with 5 years experience
2. Excellent communication skills - resume is clear and well-organized
3. Proven execution track record with quantified outcomes

## Top 3 Concerns
1. Limited technical background - no engineering experience
2. No AI/ML projects or education
3. Strategic thinking appears tactical (short-term focused)

[... detailed pillar-by-pillar analysis ...]

HTML Report (.html file):

Beautiful, print-friendly design
Color-coded scores and decision badges
Progress bars for each pillar
Professional styling inspired by Stripe Docs + Tailwind CSS

🛠️ Troubleshooting

"Module not found" Errors

# Reinstall dependencies
pip3 install -r requirements.txt

"API key not found"

Problem: The tool can't find your API key.

Solutions:

Check .env file exists:

# Show all files including hidden ones
ls -la .env

# Should show: -rw-r--r--  1 yourname  staff  XXX Oct 25 XX:XX .env

Verify .env file location:
- Must be in the root directory of the project
- Same folder as the analyze script
- NOT in examples/, bin/, or any subfolder

Check file contents:

# View the .env file
cat .env

# Make sure you see your actual API key, not just empty values

Common mistakes:
- ❌ Extra spaces: OPENAI_API_KEY = sk-... (wrong)
- ✅ No spaces: OPENAI_API_KEY=sk-... (correct)
- ❌ Quotes around key: OPENAI_API_KEY="sk-..." (wrong)
- ✅ No quotes: OPENAI_API_KEY=sk-... (correct)
- ❌ Wrong filename: .env.txt or env (wrong)
- ✅ Exact name: .env (correct)

Start fresh:

# Delete old .env and copy from example
rm .env
cp examples/example.env .env
nano .env  # Add your key

"Resume file not found"

# Use full path to resume
./bin/analyze /full/path/to/resume.pdf

# Or move resume to current directory
cp ~/Downloads/resume.pdf ./
./bin/analyze resume.pdf

"Unsupported file format" Error

Ensure file extension is .pdf, .doc, or .docx
For .doc/.docx files, make sure pandoc is installed (see Prerequisites)
Try re-exporting from the original application if file appears corrupted
Some password-protected files may not work - remove protection first

Python Version Issues

# Check Python version (need 3.7+)
python3 --version

# If too old, install newer Python:
# macOS: brew install python3
# Linux: sudo apt install python3.9

API Errors

"Insufficient credits": Add credits to your AI provider account

OpenAI: https://platform.openai.com/account/billing
Anthropic: https://console.anthropic.com/settings/billing
Google: Usually has generous free tier

"Rate limit exceeded": Wait 60 seconds and try again (or upgrade API plan)

💡 Pro Tips

Cost Optimization

Budget Options (Best for high-volume screening):

Gemini 2.5 Flash: $0.10-0.15/resume - Fast and affordable
Claude Haiku 4.5: $0.15-0.25/resume - Fast with great quality
GPT-5 Mini: $0.20-0.30/resume - Faster GPT-5 variant
GPT-4o: $0.15-0.25/resume - Budget-friendly OpenAI option

Premium Options (Best for final round evaluations):

GPT-5: $0.40-0.50/resume - Best reasoning capabilities
Claude Sonnet 4.5: $0.45-0.60/resume - Best for complex analysis
Gemini 2.5 Pro: $0.25-0.30/resume - Advanced thinking model
Claude Opus 4.1: $0.50-0.70/resume - Most capable (highest cost)

Recommendation: Use Gemini Flash or Claude Haiku for initial screening (100+ resumes), then use GPT-5 or Claude Sonnet for top candidates in final rounds.

Batch Processing

# Analyze multiple resumes
for resume in resumes/*.pdf; do
  ./bin/analyze "$resume" --output ./batch_reports/
done

Integration with ATS

# Generate JSON output for ATS integration
./bin/analyze resume.pdf --format markdown
# Then parse the .json file programmatically

Compare Candidates

Open multiple HTML reports in browser tabs for side-by-side comparison.

📖 Examples

See the examples/ folder for:

example.env - Complete template for configuring your API keys (copy this to create your .env file)

🏗️ Project Structure

aipm-resume-analyzer/
├── bin/
│   └── analyze                    # Main analyzer script
├── scripts/
│   └── install.sh                 # Installation script
├── templates/
│   └── output_generator.py        # Report generation
├── examples/
│   └── example.env                # ⭐ Template for your .env file (copy this!)
├── output/                        # Generated reports (created on first run)
├── requirements.txt               # Python dependencies
├── analyze                        # Convenience wrapper script
├── .env                          # ⚠️ YOUR API KEYS (create from example.env)
├── .gitignore
├── LICENSE
└── README.md

Key files:

examples/example.env - Template showing all API configuration options
.env - Your actual API keys (you must create this from the example)

🔒 Privacy & Security

How your data is handled:

⚠️ Resume data is sent to your chosen AI provider (OpenAI, Anthropic, or Google) for analysis
✅ No data stored by this tool - we don't have servers, only your local machine
✅ You control which provider receives your data via API selection
✅ API keys stored locally in .env file on your machine
📋 Review provider policies: Each AI provider has their own data retention and privacy policies

Best practices:

Never commit .env file to git (it's in .gitignore)
Don't share your API keys
Rotate keys if accidentally exposed
Review your chosen AI provider's data policies before analyzing sensitive resumes
Keep this tool updated for security patches

❓ FAQ

Q: Can I use this for non-AI PM roles? A: The framework is AI PM-specific, but you can fork and customize the prompts for other roles.

Q: How accurate is the AI scoring? A: Treat it as a screening tool, not a final decision. Always validate with human review.

Q: Can I customize the evaluation criteria? A: Yes! Edit bin/analyze to modify the scoring prompts and framework.

Q: Which AI provider is most accurate? A: Claude Sonnet 4.5 and GPT-5 give the most nuanced analysis. Use --list-models to see all options. Gemini 2.5 Flash is fastest/cheapest for high-volume screening.

Q: What file formats are supported? A: PDF, DOC, and DOCX files are supported. For .doc/.docx files, pandoc must be installed (see Prerequisites).

Q: Can I analyze my own resume? A: Absolutely! Great for self-assessment before applying to AI PM roles.

🤝 Contributing

Contributions welcome! This tool helps standardize AI PM hiring and provides a roadmap for people transitioning into AI PM roles.

To Contribute

Fork the repository
Create a feature branch: git checkout -b feature-name
Make your changes
Test with sample resumes
Submit a pull request

Ideas for Contributions

Additional output formats (PDF export)
Batch processing UI
Integration with popular ATS systems
Customizable scoring frameworks
Multi-language support

📄 License

MIT License - see LICENSE file for details.

🙏 Acknowledgments

Inspiration:

Jaclyn Konzelmann - Google's Director of AI Product, whose evaluation framework inspired this project
Aakash Gupta - Product Growth expert whose podcast conversation sparked the idea

Built with:

OpenAI GPT-5 - Most advanced reasoning model
Anthropic Claude Sonnet 4.5 - Best-in-class AI assistant
Google Gemini 2.5 Pro - Fast and affordable AI
PyPDF2 - PDF text extraction
Pandoc - Universal document converter (.doc/.docx support)

Design inspired by:

Stripe Docs - Clean documentation design
Tailwind CSS - Modern utility-first CSS
Anthropic Docs - Warm, professional color palette

📚 Related Resources

Framework Website: aipmframework.com - Full evaluation framework and philosophy
Source Code: GitHub Repository
Twitter: @abe238

🚨 Important Note

This tool is a screening aid, not a replacement for human judgment.

✅ Use it to: Standardize initial screening, identify promising candidates, save time, self-assess your readiness
❌ Don't use it to: Make final hiring decisions, replace interviews, avoid talking to candidates

The best hiring process combines AI-assisted screening with thoughtful human evaluation.

Built for hiring managers, recruiters, and candidates—especially those from non-traditional backgrounds—who want standardized, data-driven AI PM evaluations.

⭐ Star this repo if you find it useful!

Need Help?

Issues: https://github.com/abe238/aipm-resume-analyzer/issues
Twitter: @abe238
Website: aipmframework.com

About the Builder

Hola. I'm Abe Diaz. By day, I'm Sr. Manager of Disaster Relief at Amazon, leading a team that uses AI and logistics to deliver 26+ million relief items and response technology across 200+ disasters worldwide. On weekends, I build artisanal developer tools like this :-P

Other Open Source Projects

Project	Description
gg-deploy	Domain → GitHub Pages in 60 seconds
gemini-deep-research	CLI for Gemini's Deep Research Agent
project-kickoff	Scaffold AI production-ready projects

Name		Name	Last commit message	Last commit date
Latest commit History 56 Commits
bin		bin
examples		examples
scripts		scripts
templates		templates
.gitignore		.gitignore
CNAME		CNAME
CONTRIBUTING.md		CONTRIBUTING.md
LICENSE		LICENSE
QUICK_START.md		QUICK_START.md
README.md		README.md
analyze		analyze
index.html		index.html
requirements.txt		requirements.txt
social-share.png		social-share.png

License

abe238/aipm-resume-analyzer

Folders and files

Latest commit

History

Repository files navigation