Skip to content

Fix/update azure documentation #51

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
wants to merge 2 commits into
base: main
Choose a base branch
from
Open

Fix/update azure documentation #51

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
296ed22
Update docs to reference azure instead of azure_ai for provider
rholinshead Nov 18, 2025
d7f6266
Add info on optional dependencies
rholinshead Nov 19, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
9 changes: 4 additions & 5 deletions docs/detailed_guide.mdx
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -25,9 +25,10 @@ Think of `mcp-eval` as your “flight simulator” for tool‑using LLMs. You pl
## Getting Started

1. **Install mcp-eval globally**: `uv tool install mcpevals` (recommended) or `pip install mcpevals`
2. **Initialize your project**: `mcp-eval init` - interactive setup for API keys and configuration
3. **Add your MCP server**: `mcp-eval server add` - configure the server you want to test
4. **Run tests**: `mcp-eval run tests/` - execute your test suite
2. **If needed, install provider dependencies**: `uv add mcpevals "mcp-agent[anthropic,azure,openai]"` or `pip install mcpevals "mcp-agent[anthropic,azure,openai]"`
3. **Initialize your project**: `mcp-eval init` - interactive setup for API keys and configuration
4. **Add your MCP server**: `mcp-eval server add` - configure the server you want to test
5. **Run tests**: `mcp-eval run tests/` - execute your test suite

<Note>
**Test servers written in any language:** Your MCP server can be written in Python, TypeScript, Go, Rust, Java, or any other language. `mcp-eval` connects to it via the MCP protocol, making testing completely language-agnostic.
Expand Down Expand Up @@ -120,5 +121,3 @@ Use `mcp-eval doctor`, `validate`, and `issue` for diagnosis. See [Troubleshooti
- Keep servers in mcp‑agent config; use `mcpeval.yaml` for eval knobs

{/* TODO: Add a closing diagram summarizing the `mcp-eval` lifecycle (init → write tests → run → metrics → reports). */}


7 changes: 6 additions & 1 deletion docs/index.mdx
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -46,6 +46,9 @@ uv tool install mcpevals
# Add mcp-eval dependency to your project
uv add mcpevals

# If needed, install with optional provider dependencies
uv add mcpevals "mcp-agent[anthropic,azure,openai]"

# Initialize your project (interactive setup)
mcp-eval init

Expand All @@ -66,6 +69,9 @@ uv run pytest -q tests
# Install mcp-eval
pip install mcpevals

# If needed, install with optional provider dependencies
pip install mcpevals "mcp-agent[anthropic,azure,openai]"

# Initialize your project
mcp-eval init

Expand Down Expand Up @@ -304,4 +310,3 @@ async def test_fetch_with_pytest():
Get help and share experiences
</Card>
</Columns>

6 changes: 5 additions & 1 deletion docs/quickstart.mdx
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -75,12 +75,16 @@ Let's make sure you have everything ready:

```bash
uv add mcpevals
# Or, with optional provider dependencies
uv add mcpevals "mcp-agent[anthropic,azure,openai]"
```

Alternatively:

```bash
pip install mcpevals
# Or, with optional provider dependencies
pip install mcpevals "mcp-agent[anthropic,azure,openai]"
```

Now set up your API key for the best experience:
Expand Down Expand Up @@ -532,4 +536,4 @@ Ready to become a `mcp-eval` expert? Here's your learning path:

---

**Congratulations!** 🎉 You've successfully set up `mcp-eval` and run your first tests. You're now ready to ensure your MCP servers and agents work flawlessly. Happy testing!
**Congratulations!** 🎉 You've successfully set up `mcp-eval` and run your first tests. You're now ready to ensure your MCP servers and agents work flawlessly. Happy testing!
4 changes: 2 additions & 2 deletions docs/subagents/mcp-eval-config-expert.mdx
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -18,7 +18,7 @@ name: "My MCP Test Suite"
description: "Comprehensive testing for my MCP servers"

# Provider configuration (global defaults)
provider: anthropic # anthropic|openai|google|bedrock|vertex_ai|azure_ai
provider: anthropic # anthropic|openai|google|bedrock|vertex_ai|azure
Copy link
Collaborator

@StreetLamb StreetLamb Nov 20, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Same here

model: claude-3-5-sonnet-20241022 # Or gpt-4-turbo-preview, gemini-pro, etc.

# MCP Server Configuration (REQUIRED for mcp-agent)
Expand Down Expand Up @@ -511,4 +511,4 @@ logger:
file_path: "./ci-logs/mcp-eval.log"
```

Remember: Good configuration is the foundation of reliable testing!
Remember: Good configuration is the foundation of reliable testing!
4 changes: 2 additions & 2 deletions src/mcp_eval/data/subagents/mcp-eval-config-expert.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -18,7 +18,7 @@ name: "My MCP Test Suite"
description: "Comprehensive testing for my MCP servers"

# Provider configuration (global defaults)
provider: anthropic # anthropic|openai|google|bedrock|vertex_ai|azure_ai
provider: anthropic # anthropic|openai|google|bedrock|vertex_ai|azure
Copy link
Collaborator

@StreetLamb StreetLamb Nov 20, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think vertex_ai is a hallucination as well

model: claude-3-5-sonnet-20241022 # Or gpt-4-turbo-preview, gemini-pro, etc.

# MCP Server Configuration (REQUIRED for mcp-agent)
Expand Down Expand Up @@ -511,4 +511,4 @@ logger:
file_path: "./ci-logs/mcp-eval.log"
```

Remember: Good configuration is the foundation of reliable testing!
Remember: Good configuration is the foundation of reliable testing!