feat(data-warehouse): implement the aws config import source

[Gilbert09]

Problem

• AWS Config customers cannot sync AWS Config data into the PostHog data warehouse. The source was a scaffolded stub, hidden from the catalog.

Changes

• Users can now connect AWS Config with an IAM access key and sync these tables:

Table	Sync
resources	Full refresh
config_rules	Full refresh
rule_compliance	Full refresh
conformance_packs	Full refresh

• Requests are SigV4-signed over the tracked HTTP session, following the existing aws_organizations source.
• Pagination is resumable, so a sync picks up after the last written page.
• API version pinned: 2014-11-12.
• The source ships as alpha with unreleasedSource removed.
• Mechanical: the generated config and the SOURCES.md row.

How did you test this code?

• New parameterized unit tests cover request signing and targets, pagination including the last page, incremental and full refresh request shapes, and error mapping.
• Not checked: live calls against a real AWS account. mypy runs in CI only.

Test rationale: the source is new, so no existing test covers its transport.

Release status

• No feature flag controls this change
• This change is behind a feature flag and is not available to users
• This change makes a previously flagged feature available to everyone

Automatic notifications

• Publish to changelog?

Docs update

• A posthog.com page for this source is still needed.

[trunk-io]

⚠️ Pull request failed tests in a previous identical test run and is waiting for other pull requests to finish testing. This SHA was used for testing. See more details here.

[parameterai]

Risk: No findings

The delta since the last review is purely mechanical: Iterator → Generator in the get_rows return annotation and cast(...) wrappers plus one # type: ignore[attr-defined] in tests. No runtime behavior changed, so the previously reviewed AWS Config source (signing, pagination, resume, error handling) is unaffected. No new security-relevant surface is introduced.

_{Sentinel reviewed 2aa4d57 · Review settings}

[github-actions]

Hey @Gilbert09! 👋

It looks like your git author email on this PR isn't your @posthog.com address (owerstom@gmail.com). Since you're on the PostHog team, it's worth pointing your local git author email at your @posthog.com address. Why it matters:

• Consistent work identity in git history — internal tooling that attributes commits to team members keys off your @posthog.com address.
• Keeps team contributions easy to tell apart from external community ones when scanning history.

You can fix it for this repo with:

git config user.email "you@posthog.com"

Or set it globally with git config --global user.email "you@posthog.com". No need to redo this PR — just a nudge for next time. 🙂

[github-actions]

🤖 CI report

⚠️ Trunk lane — backend Python lane

This PR is assigned to the backend Python lane. It runs backend Python tests and may merge in parallel with PRs in other lanes.

⚠️ Duplication (Python) — 5 new duplicated blocks (worst 137 tokens)

New Python code duplication introduced by this branch. Fails at 70+ tokens in app code, or 150+ tokens when both copies live in test files. Advisory while the gate proves itself: extract a shared helper instead of copying.

First copy	Second copy	Lines	Tokens
products/warehouse_sources/backend/temporal/data_imports/sources/acculynx/source.py:18	products/warehouse_sources/backend/temporal/data_imports/sources/aws_config/source.py:22	14	137
products/warehouse_sources/backend/temporal/data_imports/sources/aws_budgets/source.py:140	products/warehouse_sources/backend/temporal/data_imports/sources/aws_config/source.py:110	27	100
products/warehouse_sources/backend/temporal/data_imports/sources/aws_config/aws_config.py:103	products/warehouse_sources/backend/temporal/data_imports/sources/aws_glue_data_catalog/aws_glue_data_catalog.py:111	13	90
products/warehouse_sources/backend/temporal/data_imports/sources/aws_config/aws_config.py:105	products/warehouse_sources/backend/temporal/data_imports/sources/aws_waf/aws_waf.py:112	12	74
products/warehouse_sources/backend/temporal/data_imports/sources/aws_cloudtrail/source.py:142	products/warehouse_sources/backend/temporal/data_imports/sources/aws_config/source.py:117	21	73

✅ Duplication (TypeScript) — clean

New TypeScript code duplication introduced by this branch. Fails at 70+ tokens in app code, or 150+ tokens when both copies live in test files. Advisory while the gate proves itself: extract a shared helper instead of copying.

⚠️ Backend coverage — 98.0% of changed backend lines covered — 5 uncovered

🧪 Backend test coverage

Patch coverage — changed backend lines (products + core): ████████████████████ 98.0% (401 / 406)

File	Patch	Uncovered changed lines
products/warehouse_sources/backend/temporal/data_imports/sources/aws_config/source.py	88.5%	76, 79, 87
products/warehouse_sources/backend/temporal/data_imports/sources/aws_config/aws_config.py	98.7%	65–66

🤖 Agents: add a test only if an uncovered line exposes a realistic regression that existing tests miss. Otherwise explain why no new test is needed under "How did you test this code?". Gap list: the patch-coverage artifact on this run (gh run download 39491575517581 -n patch-coverage), or the coverage-data block at the end of this comment.

Per-product line coverage (touched products)

Product	Coverage	Lines
demo	███████████░░░░░░░░░ 53.4%	1,445 / 2,707
batch_exports	████████████████░░░░ 81.3%	21,584 / 26,561
cdp	██████████████████░░ 88.3%	4,559 / 5,164
mcp_analytics	██████████████████░░ 89.2%	5,038 / 5,651
product_tours	██████████████████░░ 89.3%	1,340 / 1,500
dashboards	██████████████████░░ 89.6%	6,924 / 7,727
notebooks	██████████████████░░ 90.2%	15,514 / 17,207
signals	██████████████████░░ 90.4%	60,161 / 66,532
cohorts	██████████████████░░ 90.5%	8,534 / 9,434
data_warehouse	██████████████████░░ 90.6%	14,375 / 15,860
streamlit_apps	██████████████████░░ 90.8%	2,684 / 2,956
managed_warehouse	██████████████████░░ 91.0%	10,252 / 11,263
data_modeling	██████████████████░░ 91.2%	10,562 / 11,584
tasks	██████████████████░░ 91.3%	79,682 / 87,258
exports	██████████████████░░ 91.7%	9,684 / 10,566
business_knowledge	██████████████████░░ 92.0%	8,472 / 9,208
engineering_analytics	██████████████████░░ 92.2%	11,497 / 12,475
today	██████████████████░░ 92.3%	999 / 1,082
early_access_features	███████████████████░ 92.6%	1,339 / 1,446
conversations	███████████████████░ 92.6%	29,225 / 31,559
stamphog	███████████████████░ 92.8%	8,109 / 8,742
canvas	███████████████████░ 92.9%	7,155 / 7,703
approvals	███████████████████░ 93.0%	3,974 / 4,271
mcp_registry	███████████████████░ 93.1%	1,670 / 1,794
notifications	███████████████████░ 93.2%	1,144 / 1,228
error_tracking	███████████████████░ 93.3%	16,389 / 17,573
surveys	███████████████████░ 93.4%	6,644 / 7,113
autoresearch	███████████████████░ 93.4%	8,837 / 9,457
slack_app	███████████████████░ 93.7%	14,611 / 15,600
context_layer	███████████████████░ 93.8%	3,415 / 3,639
web_analytics	███████████████████░ 93.9%	23,680 / 25,229
billing_alerts	███████████████████░ 94.1%	2,094 / 2,226
mcp_store	███████████████████░ 94.3%	8,959 / 9,501
alerts	███████████████████░ 94.7%	9,319 / 9,844
wizard	███████████████████░ 94.7%	6,150 / 6,496
workflows	███████████████████░ 94.7%	15,187 / 16,034
ai_observability	███████████████████░ 94.7%	26,037 / 27,485
reminders	███████████████████░ 94.8%	760 / 802
review_hog	███████████████████░ 95.0%	11,750 / 12,362
annotations	███████████████████░ 95.1%	817 / 859
endpoints	███████████████████░ 95.1%	9,234 / 9,706
customer_analytics	███████████████████░ 95.2%	26,116 / 27,428
marketing_analytics	███████████████████░ 95.3%	19,450 / 20,413
posthog_ai	███████████████████░ 95.4%	2,530 / 2,653
actions	███████████████████░ 95.5%	756 / 792
logs	███████████████████░ 95.5%	15,468 / 16,200
experiments	███████████████████░ 95.5%	33,000 / 34,548
data_catalog	███████████████████░ 95.5%	4,401 / 4,606
tracing	███████████████████░ 95.6%	3,536 / 3,699
replay_vision	███████████████████░ 95.6%	29,389 / 30,727
growth	███████████████████░ 95.7%	11,381 / 11,888
skills	███████████████████░ 95.8%	6,972 / 7,274
messaging	███████████████████░ 95.9%	3,834 / 3,999
product_analytics	███████████████████░ 96.0%	28,521 / 29,696
revenue_analytics	███████████████████░ 96.4%	1,889 / 1,959
user_interviews	███████████████████░ 96.5%	2,870 / 2,974
feature_flags	███████████████████░ 96.6%	27,119 / 28,060
access_control	███████████████████░ 96.7%	7,739 / 8,007
warehouse_sources	███████████████████░ 97.3%	473,150 / 486,234
data_quality	████████████████████ 97.5%	7,701 / 7,895
metrics	████████████████████ 98.0%	4,252 / 4,338
analytics_platform	████████████████████ 98.3%	2,784 / 2,833
pulse	████████████████████ 98.5%	2,046 / 2,078
live_debugger	████████████████████ 99.2%	626 / 631

Report-only. Patch coverage = changed backend lines covered vs origin/master. Sorted lowest first.
Known gaps: lines covered only by Temporal tests show as uncovered; core line numbers may drift if master changed the same file.

[coderabbitai]

Navigate logical layers of code changes, visualize relationships, and explore their blast radius.

🧰 Additional context used

📚 Code guidelines (5)

.agents/security.md — configured
docs/published/handbook/engineering/type-system.md — configured
.agents/skills/adopting-generated-api-types/SKILL.md — configured
.claude/commands/conventions.md — configured
.agents/skills/writing-code-comments/SKILL.md — configured

📝 Walkthrough

Walkthrough

The change implements AWS Config ingestion for four datasets. It adds endpoint and credential configuration, SigV4-signed requests, error handling, row normalization, and resumable pagination. The source now provides schema discovery, credential validation, and setup metadata. Tests cover requests, pagination, resume behavior, and error handling. The source inventory lists AWS Config as implemented.

Priority: ⬇️ Low

Merge Risk: 🟡 Moderate · up to 05f87

If an AWS Config resume token expires, earlier pages can be added to the warehouse again, duplicating records. Fix the recovery path before merging to protect the accuracy of synced data.

Architecture Summary

Architecture risk: 🔵 Low · up to 05f87

The change affects 1 system.

Changed systems: products

Architecture concerns
No architecture-level concerns identified.

Review details

Systems and components

• observed — products (service) was modified; 8 changed files map to changed impact.

Before / after behavior

• observed — Modified behavior in products/warehouse_sources/backend/temporal/data_imports/sources/aws_config/canonical_descriptions.py: Adds descriptions and AWS API documentation URLs for four AWS Config datasets, along with descriptions of their resource, rule, compliance, and conformance-pack columns.
• observed — Modified behavior in products/warehouse_sources/backend/temporal/data_imports/sources/aws_config/settings.py: Defines the AWS Config API version and target prefix, plus the resource query selecting account, region, identity, configuration, timestamps, and tags.
• observed — Modified behavior in products/warehouse_sources/backend/temporal/data_imports/sources/aws_config/settings.py: Adds the frozen AwsConfigEndpoint configuration type, with required operation, result key, primary key, and description fields, and optional page size and expression.
• observed — Modified behavior in products/warehouse_sources/backend/temporal/data_imports/sources/aws_config/settings.py: Defines four endpoint configurations: resource queries use SelectResourceConfig, page size 100, and a composite account/region/type/resource key; the other entries configure Config rules, rule compliance, and conformance packs, with a page size of 20 for conformance packs. Derives the endpoint-name tuple and description mapping from this configuration.

🚥 Pre-merge checks | ✅ 1

✅ Passed checks (1 passed)

Check name	Status	Explanation
Description check	✅ Passed	The description covers the problem, user-visible changes, supported tables, testing scope and rationale, release status, notifications, and documentation status. It does not include the optional Agent…

✨ Finishing Touches

🧪 Generate unit tests (beta)

• Commit to this branch
• Create a new PR

• Autopilot · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Autopilot is currently an internal CodeRabbit preview.

---

_{Comment @coderabbitai help to get the list of available commands.}

[trunk-io]

   

View Full Report ↗︎ ⋅ Docs

A new stamphog review started for this PR — the fresh verdict replaces this approval.

[Gilbert09]

CI is green on 2aa4d576, there are no unresolved review threads, and GitHub reports the branch merges cleanly. The remaining blocked state requires approval from Team Warehouse Sources.

_{🦉 via talyn.dev}

[Gilbert09]

/trunk merge

A new stamphog review started for this PR — the fresh verdict replaces this approval.

[stamphog]

Approved.

This is a self-contained new warehouse source written by an owning-team author with strong familiarity, and it has thorough tests. Credential handling looks sound: the region is regex-validated before it goes into the host, redirects are disabled, secrets are redacted, and the secret fields are marked secret.

• Author wrote 88% of the modified lines and has 95 merged PRs in these paths (familiarity STRONG).

Gate mechanics and policy version

Gate		Result
prerequisites	✓	all clear
deny-list	✓	no deny categories matched
size	✓	510L, 5F substantive, 903L/8F incl. docs/generated/snapshots — within ceiling
tier	✓	T1-agent / T1d-complex (903L, 8F, single-area, feat)
stamphog 2.3.1		.stamphog/policy.yml @ 05f877e · reviewed head 05f877e

[coderabbitai]

Caution

Some comments are outside the diff and can’t be posted inline due to GitHub limitations.

⚠️ Outside diff range comments (1)

🟠 Major · Abort the resumed attempt after clearing an expired token. · aws_config.py:172-193

products/warehouse_sources/backend/temporal/data_imports/sources/aws_config/aws_config.py:172-193
🗄️ Data Integrity & Integration | 🟠 Major | ⚡ Quick win

Abort the resumed attempt after clearing an expired token.

This branch currently restarts inside the same pipeline. V2 and V3 have already captured their resume flags, so the replayed full-refresh rows append to the existing table.

Clear the checkpoint and re-raise the error. The import workflow retries this activity up to three times. The next attempt sees no resume state, overwrites the first page, and then appends later pages without skipping source data.

Suggested fix

                if restarting and error.code == "InvalidNextTokenException":
+                    manager.clear_state()
-                    next_token = None
-                    restarting = False
-                    continue
+                    raise

---

ℹ️ Review info

⚙️ Run configuration

Configuration used: Repository: PostHog/posthog/.coderabbit.yaml

Review profile: QUIET

Plan: Enterprise

Run ID: 61deaae0-f152-4204-a603-8c30e0edfc9b

📥 Commits

Reviewing files that changed from the base of the PR and between 65c81ae and 05f877e.

📒 Files selected for processing (1)

• products/warehouse_sources/backend/temporal/data_imports/sources/SOURCES.md

Included review availability: This review used your included allowance. Your plan provides up to 12 included reviews per hour; 2 remain after this review.

[Gilbert09]

/trunk merge