feat: add session test data generator script - #42
Conversation
Generates dummy LLM Analytics session events via the capture API — no LLM calls needed. Creates three test scenarios: 1. Session with 120 traces (pagination / "load more" testing) 2. Session with nested spans (latency double-counting bugs) 3. Short control session (baseline comparison)
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: c9812094e9
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| sys.exit(1) | ||
|
|
||
|
|
||
| def capture_event(event: str, properties: dict, timestamp: str | None = None) -> None: |
There was a problem hiding this comment.
The repository README declares Python 3.8+ support, but this signature uses PEP 604 union syntax (str | None), which raises a SyntaxError on Python 3.8/3.9 and prevents the script from running in documented environments. Use Optional[str] (or equivalent compatibility handling) so the generator works across supported Python versions.
Useful? React with 👍 / 👎.
| "$ai_total_cost_usd": str(cost), | ||
| } | ||
| if has_error and i == num_generations - 1: | ||
| props["$ai_is_error"] = "true" |
There was a problem hiding this comment.
This writes $ai_is_error as the string "true" instead of a boolean. Because property typing is based on the ingested JSON value, error filters/aggregations that expect a boolean can miss these events, so the generated “error” scenario may look like a non-error trace in analytics views.
Useful? React with 👍 / 👎.
| "$ai_span_id": parent_span_id, | ||
| "$ai_parent_id": trace_id, | ||
| "$ai_span_name": "agent-chain", | ||
| "$ai_latency": str(parent_latency), |
There was a problem hiding this comment.
Latency is serialized with str(...), which ingests $ai_latency as a string property rather than a numeric one. This undermines the script’s stated purpose of validating latency behavior, since numeric latency rollups/sorting can become incorrect or require extra casting; emit float values directly.
Useful? React with 👍 / 👎.
1 participant
Summary
python/scripts/generate_session_test_data.py— generates dummy LLM Analytics session events directly via the capture API (no LLM calls or OpenAI key needed)Test plan
source python/venv/bin/activate && python python/scripts/generate_session_test_data.pyagainst local dev