feat(data-warehouse): implement recall_ai import source

[Gilbert09]

Problem

Teams that record meetings with Recall.ai have no way to get that data into the PostHog data warehouse. The connector existed only as a hidden scaffold (unreleasedSource=True, no fields, no sync logic).

Why: requested implementation of the scaffolded recall_ai source end to end, so the connector becomes usable.

Changes

• Users can now connect Recall.ai (released as alpha) and sync seven tables: bots, recordings, transcripts, participant_events, meeting_metadata, calendars, and calendar_events.
• The connect form asks for an API key and a region (us-east-1, us-west-2, eu-central-1, ap-northeast-1). Keys are region-scoped, and the region allowlist pins outbound traffic to *.recall.ai.
• The source rides the shared rest_source framework as a ResumableSource: next-URL cursor pagination with a resume checkpoint saved before each page's rows are yielded.
• recordings, transcripts, participant_events, and meeting_metadata sync incrementally on created_at, and calendar_events on updated_at, through the API's server-side filters. The first incremental sync goes out unfiltered so the watermark seeds from real rows.
• bots and calendars are full refresh only. The bot list filters only on join_at, a date-granularity scheduled join time that is not creation-ordered, so a watermark would skip bots. The calendar list filters only on created_at, which misses status changes.
• Calendar rows drop oauth_client_secret and oauth_refresh_token before reaching the warehouse, because the API echoes them in list responses.
• sort_mode is desc: the API documents no ordering and its cursor pagination takes no sort param, and desc commits the incremental watermark only at sync end, which is correct for any actual arrival order.
• Canonical table and column descriptions feed the AI enrichment and the public docs table list (lists_tables_without_credentials=True).
• Mechanical: SOURCES.md moves recall_ai into the implemented table, and generated_configs/recallai.py is regenerated from the new fields.

Note

Endpoint paths, params, auth, pagination envelope, and response shapes were verified against the live API (unauthenticated probes) and the OpenAPI spec embedded in docs.recall.ai. No Recall.ai workspace credentials were available, so the incremental filters were not smoke-tested with a future-date cutoff. They are documented filters on every list endpoint, and merge dedupe on id keeps any overlap safe; a code comment marks the unverified ordering assumption.

How did you test this code?

• products/warehouse_sources/backend/temporal/data_imports/sources/recall_ai/tests/ (32 tests, run locally with pytest): request shaping catches a filter sent without a watermark or with a non-ISO value, pagination/resume catches a broken next_url checkpoint key, scrubbing catches OAuth secrets leaking into rows, and region validation catches a probe fired at an unknown host.
• Registry-wide invariant tests (test_source_versions.py, test_source_categories.py) pass with the new source registered.
• ruff check and ruff format clean on the changed files.
• Not run: hogli build:openapi (this sandbox has no database; the RecallAI enum is already in the generated TS and per-source fields are runtime data, so the schema does not change), live syncs against a real Recall.ai workspace, and no migrations are needed (no model change).

Release status

• No feature flag controls this change

releaseStatus=ALPHA labels the source as new in the wizard; it does not gate it.

Automatic notifications

• Publish to changelog?

Docs update

The posthog.com doc (contents/docs/cdp/sources/recall-ai.md) is written but needs a separate PR in the posthog.com repo; docsUrl already points at https://posthog.com/docs/cdp/sources/recall-ai. audit_source_docs was not run because no posthog.com checkout was available here.

Doc content for the posthog.com PR

---
title: Linking Recall.ai as a source
sidebar: Docs
showTitle: true
availability: { free: full, selfServe: full, enterprise: full }
sourceId: RecallAI
beta: true
---

import AlphaRelease from "../_snippets/alpha-release.mdx"
import SourceSetupIntro from "../_snippets/source-setup-intro.mdx"
import SyncModes from "../_snippets/sync-modes.mdx"
import TroubleshootingLink from "../_snippets/dw-troubleshooting-link.mdx"

<AlphaRelease />

The Recall.ai connector syncs your meeting bots, recordings, transcript metadata, participant events, meeting metadata, and connected calendar data into PostHog, so you can analyze meeting activity alongside your product data.

## Prerequisites

- A [Recall.ai](https://www.recall.ai/) workspace
- A Recall.ai API key

## Adding a data source

<SourceSetupIntro />

You need two things to connect:

1. **API key**: create one in your [Recall.ai dashboard](https://www.recall.ai/login) under **API keys**.
2. **Region**: the region your workspace runs in (`us-east-1`, `us-west-2`, `eu-central-1`, or `ap-northeast-1`). You can find it in your Recall.ai dashboard URL. API keys only work in the region they were created in.

## Sync modes

<SyncModes />

The `bots` and `calendars` tables support full refresh only, because the Recall.ai API has no reliable server-side filter for changes to them. The other tables support incremental syncs on `created_at`, and `calendar_events` on `updated_at`.

## Configuration

<SourceParameters />

## Supported tables

<SourceTables />

A few notes on the synced data:

- Transcript, participant event, and meeting metadata rows carry temporary download URLs for their content, not the content itself. The URLs expire, so fetch fresh ones from the Recall.ai API when you need the underlying transcript or event log.
- Recall.ai clears `meeting_url` on bots a few days after the bot has joined a call.
- OAuth client secrets and refresh tokens on calendar rows are removed before the data reaches PostHog.

## Troubleshooting

**Couldn't connect to Recall.ai**: API keys are region-scoped. Check that the key is valid and that the region you selected matches the region shown in your Recall.ai dashboard URL.

<TroubleshootingLink />

🤖 Agent context

Autonomy: Human-driven (agent-assisted)

Agent: Claude Code (PostHog Desktop task), Claude Fable 5

• Skills invoked: /implementing-warehouse-sources, /documenting-warehouse-sources, /writing-dataclasses, /writing-tests, /writing-code-comments, /writing-user-facing-copy, /writing-pr-descriptions. CodeRabbit local pass skipped (cloud task run).
• No duplicate: gh pr list --state open --search "recall" found no PR touching this source.
• API behavior was established from the vendor's own OpenAPI spec extracted from docs.recall.ai plus unauthenticated probes of every list route; the icon (recall_ai.png) was already committed.
• Regenerating source configs also rewrote googleads/slack/stripe generated configs with environment-dependent output (missing OAuth env vars in the sandbox); those were reverted and only recallai.py is included.
• All sample data in tests is invented; nothing session-private is committed.

---

Created with PostHog Desktop

[trunk-io]

😎 Merged successfully - details.

[Copilot]

Copilot was unable to review this pull request because the user who requested the review has reached their quota limit.

[github-actions]

🤖 CI report

⚠️ Trunk lane — backend Python lane

This PR is assigned to the backend Python lane. It runs backend Python tests and may merge in parallel with PRs in other lanes.

⚠️ Duplication (Python) — 5 new duplicated blocks (worst 171 tokens)

New Python code duplication introduced by this branch. Fails at 70+ tokens in app code, or 150+ tokens when both copies live in test files. Advisory while the gate proves itself: extract a shared helper instead of copying.

First copy	Second copy	Lines	Tokens
products/warehouse_sources/backend/temporal/data_imports/sources/hyros/hyros.py:1	products/warehouse_sources/backend/temporal/data_imports/sources/recall_ai/recall_ai.py:1	21	171
products/warehouse_sources/backend/temporal/data_imports/sources/concord/source.py:1	products/warehouse_sources/backend/temporal/data_imports/sources/recall_ai/source.py:1	21	168
products/warehouse_sources/backend/temporal/data_imports/sources/recall_ai/recall_ai.py:129	products/warehouse_sources/backend/temporal/data_imports/sources/reddit_ads/reddit_ads.py:367	12	83
products/warehouse_sources/backend/temporal/data_imports/sources/apify_dataset/apify_dataset.py:53	products/warehouse_sources/backend/temporal/data_imports/sources/recall_ai/recall_ai.py:38	12	80
products/warehouse_sources/backend/temporal/data_imports/sources/cliniko/cliniko.py:138	products/warehouse_sources/backend/temporal/data_imports/sources/recall_ai/recall_ai.py:129	14	79

✅ Duplication (TypeScript) — clean

New TypeScript code duplication introduced by this branch. Fails at 70+ tokens in app code, or 150+ tokens when both copies live in test files. Advisory while the gate proves itself: extract a shared helper instead of copying.

⚠️ Comment density — 6% of added code lines are comments (42 of 703)

This section warns when comments are more than 3% of the code lines a PR adds, and alerts above 6%. Before agent-assisted PRs, the typical share was about 2%. Only full-line comments count. Docstrings, generated files, snapshots, migrations, and workflow files are left out.

Comments that restate the code, record how the change came about, or narrate the next line add noise for the next reader. Keep the comments that explain a reason the code cannot show, and remove the rest. See .agents/skills/writing-code-comments/SKILL.md for the house rules.

Files with the most added comment lines:

File	Comment lines	Added lines
products/warehouse_sources/backend/temporal/data_imports/sources/recall_ai/recall_ai.py	19	153
products/warehouse_sources/backend/temporal/data_imports/sources/recall_ai/settings.py	13	81
products/warehouse_sources/backend/temporal/data_imports/sources/recall_ai/tests/test_recall_ai.py	6	195
products/warehouse_sources/backend/temporal/data_imports/sources/recall_ai/source.py	3	125
products/warehouse_sources/backend/temporal/data_imports/sources/recall_ai/tests/test_recall_ai_source.py	1	44

This check does not block merging. It updates on every push and clears when the share drops.

⚠️ Backend coverage — 98.0% of changed backend lines covered — 3 uncovered

🧪 Backend test coverage

Patch coverage — changed backend lines (products + core): ████████████████████ 98.0% (214 / 217)

File	Patch	Uncovered changed lines
products/warehouse_sources/backend/temporal/data_imports/sources/recall_ai/source.py	93.3%	98, 106
products/warehouse_sources/backend/temporal/data_imports/sources/recall_ai/recall_ai.py	98.2%	170

🤖 Agents: add a test covering the lines above, or note why under "How did you test this code?". Machine-readable gap list: the patch-coverage artifact on this run (gh run download 546494791956561 -n patch-coverage), or the coverage-data block at the end of this comment.

Per-product line coverage (touched products)

Product	Coverage	Lines
demo	████████████░░░░░░░░ 57.8%	1,545 / 2,673
batch_exports	████████████████░░░░ 81.2%	21,452 / 26,424
cdp	██████████████████░░ 88.2%	4,548 / 5,155
mcp_analytics	██████████████████░░ 88.9%	4,910 / 5,523
product_tours	██████████████████░░ 89.3%	1,331 / 1,491
dashboards	██████████████████░░ 89.5%	6,839 / 7,641
signals	██████████████████░░ 89.9%	54,456 / 60,606
data_warehouse	██████████████████░░ 89.9%	13,912 / 15,470
notebooks	██████████████████░░ 90.2%	15,287 / 16,945
cohorts	██████████████████░░ 90.4%	8,420 / 9,316
streamlit_apps	██████████████████░░ 90.7%	2,625 / 2,895
managed_warehouse	██████████████████░░ 90.9%	10,215 / 11,234
tasks	██████████████████░░ 91.1%	73,947 / 81,146
data_modeling	██████████████████░░ 91.5%	10,525 / 11,498
business_knowledge	██████████████████░░ 91.6%	6,899 / 7,528
engineering_analytics	██████████████████░░ 91.7%	11,002 / 11,999
exports	██████████████████░░ 91.8%	9,685 / 10,555
conversations	███████████████████░ 92.5%	28,726 / 31,047
early_access_features	███████████████████░ 92.6%	1,341 / 1,448
canvas	███████████████████░ 92.8%	6,877 / 7,409
approvals	███████████████████░ 93.0%	3,919 / 4,214
mcp_registry	███████████████████░ 93.1%	1,670 / 1,794
error_tracking	███████████████████░ 93.1%	15,783 / 16,950
notifications	███████████████████░ 93.2%	1,145 / 1,229
slack_app	███████████████████░ 93.2%	13,677 / 14,674
stamphog	███████████████████░ 93.2%	7,885 / 8,456
surveys	███████████████████░ 93.3%	6,571 / 7,040
context_layer	███████████████████░ 93.8%	3,373 / 3,595
web_analytics	███████████████████░ 93.9%	21,653 / 23,051
alerts	███████████████████░ 94.0%	8,541 / 9,082
billing_alerts	███████████████████░ 94.1%	2,094 / 2,226
mcp_store	███████████████████░ 94.4%	8,940 / 9,472
ai_observability	███████████████████░ 94.4%	22,139 / 23,454
wizard	███████████████████░ 94.7%	6,151 / 6,496
reminders	███████████████████░ 94.8%	760 / 802
workflows	███████████████████░ 94.8%	13,615 / 14,360
review_hog	███████████████████░ 94.9%	11,490 / 12,109
annotations	███████████████████░ 95.1%	817 / 859
customer_analytics	███████████████████░ 95.1%	24,756 / 26,028
endpoints	███████████████████░ 95.1%	9,211 / 9,681
marketing_analytics	███████████████████░ 95.3%	19,216 / 20,161
posthog_ai	███████████████████░ 95.4%	2,489 / 2,610
tracing	███████████████████░ 95.4%	3,483 / 3,650
growth	███████████████████░ 95.4%	9,812 / 10,282
logs	███████████████████░ 95.4%	15,290 / 16,022
experiments	███████████████████░ 95.5%	32,956 / 34,526
actions	███████████████████░ 95.5%	756 / 792
data_catalog	███████████████████░ 95.5%	4,401 / 4,606
skills	███████████████████░ 95.8%	6,972 / 7,274
replay_vision	███████████████████░ 96.0%	26,757 / 27,885
product_analytics	███████████████████░ 96.2%	28,495 / 29,617
autoresearch	███████████████████░ 96.2%	8,115 / 8,434
revenue_analytics	███████████████████░ 96.4%	1,876 / 1,946
access_control	███████████████████░ 96.4%	7,122 / 7,386
user_interviews	███████████████████░ 96.5%	2,859 / 2,963
feature_flags	███████████████████░ 96.5%	25,499 / 26,416
warehouse_sources	███████████████████░ 97.2%	448,749 / 461,565
data_quality	████████████████████ 97.7%	7,592 / 7,774
metrics	████████████████████ 98.1%	4,085 / 4,166
analytics_platform	████████████████████ 98.3%	2,783 / 2,832
pulse	████████████████████ 98.5%	2,043 / 2,075
live_debugger	████████████████████ 99.2%	626 / 631

Report-only. Patch coverage = changed backend lines covered vs origin/master. Sorted lowest first.
Known gaps: lines covered only by Temporal tests show as uncovered; core line numbers may drift if master changed the same file.

[coderabbitai]

Navigate logical layers of code changes, visualize relationships, and explore their blast radius.

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info

⚙️ Run configuration

Configuration used: Repository: PostHog/posthog/.coderabbit.yaml

Review profile: QUIET

Plan: Enterprise

Run ID: 6f90ab36-064a-4682-b253-d917b9735de8

📥 Commits

Reviewing files that changed from the base of the PR and between b7e6f93 and 0d59103.

📒 Files selected for processing (4)

• products/warehouse_sources/backend/temporal/data_imports/sources/SOURCES.md
• products/warehouse_sources/backend/temporal/data_imports/sources/recall_ai/recall_ai.py
• products/warehouse_sources/backend/temporal/data_imports/sources/recall_ai/tests/test_recall_ai.py
• products/warehouse_sources/backend/temporal/data_imports/sources/recall_ai/tests/test_recall_ai_source.py

Included review availability: This review used your included allowance. Your plan provides up to 12 included reviews per hour; 11 remain after this review.

---

📝 Walkthrough

Walkthrough

The changes add a Recall.ai warehouse source with seven configured endpoints, regional API-key authentication, incremental filtering, pagination, and resumable state. The source exposes endpoint schemas and canonical descriptions, removes configured sensitive fields from results, and validates credentials. Tests cover request parameters, pagination and resume behavior, field scrubbing, watermark formatting, region validation, and credential checks.

Priority: ➖ Normal

Merge Risk: ⚪ Minimal · up to 0d591

The connector is mergeable after normal checks; the previously identified token-forwarding path is protected.

Security Architecture Review

Security architecture risk: 🟡 Moderate · up to 0d591

The connector introduces a new path for importing meeting data. Its regional request restrictions and calendar-secret removal reduce the most apparent risks, but recovery and completion behavior need more evidence before the design can be assessed as low risk.

Retained concerns
No architecture-level concerns identified.

Security review details

Security Blast Radius

• inferred — Successful imports carry a customer-provided Recall.ai API key to that customer’s selected regional API and place returned meeting records in the warehouse. The inspected path shows no additional service privilege grant.

Trust Boundaries and Controls

• observed — Recall.ai fixes the base URL to four regional HTTPS hosts and disables redirects. The shared client checks the prepared pagination or resume URL’s host, scheme, and port before sending a credential-bearing request, then rejects redirect responses on this path.

Resilience and Maintainability Implications

• observed — The resume manager keys state by team and job and stages cursors for later commit. It provides terminal clearing and optional endpoint namespaces, but the inspected Recall.ai source uses the default namespace; caller-level job isolation, concurrent-attempt handling, and completion cleanup remain unverified.

Hardening Proposals

• proposed — Confirm that vendor-generated next-page URLs cannot carry secrets before persisting or logging them; otherwise sanitize query values in checkpoint diagnostics. This is a precaution, not an observed credential leak.

🚥 Pre-merge checks | ✅ 1

✅ Passed checks (1 passed)

Check name	Status	Explanation
Description check	✅ Passed	The pull request description follows the required template. It explains the problem, user-visible changes, testing and limitations, release status, documentation status, and agent context. Optional is…

✨ Finishing Touches

📝 Generate docstrings

• Commit to this branch
• Create a new PR

🧪 Generate unit tests (beta)

• Commit to this branch
• Create a new PR

---

_{Comment @coderabbitai help to get the list of available commands.}

[coderabbitai]

Actionable comments posted: 1

---

ℹ️ Review info

⚙️ Run configuration

Configuration used: Repository: PostHog/posthog/.coderabbit.yaml

Review profile: QUIET

Plan: Enterprise

Run ID: 61078bae-ad54-4ef8-bac8-89ac79b13493

📥 Commits

Reviewing files that changed from the base of the PR and between ff051eb and b7e6f93.

📒 Files selected for processing (8)

• products/warehouse_sources/backend/temporal/data_imports/sources/SOURCES.md
• products/warehouse_sources/backend/temporal/data_imports/sources/generated_configs/recallai.py
• products/warehouse_sources/backend/temporal/data_imports/sources/recall_ai/canonical_descriptions.py
• products/warehouse_sources/backend/temporal/data_imports/sources/recall_ai/recall_ai.py
• products/warehouse_sources/backend/temporal/data_imports/sources/recall_ai/settings.py
• products/warehouse_sources/backend/temporal/data_imports/sources/recall_ai/source.py
• products/warehouse_sources/backend/temporal/data_imports/sources/recall_ai/tests/test_recall_ai.py
• products/warehouse_sources/backend/temporal/data_imports/sources/recall_ai/tests/test_recall_ai_source.py

Included review availability: This review used your included allowance. Your plan provides up to 12 included reviews per hour; 3 remain after this review.

[veria-ai]

PR overview

All previously flagged issues have been addressed. No open security concerns remain on this pull request.

Security review

No open security issues remain on this pull request.

Fixed/addressed: 2 · PR risk: 0/10

[trunk-io]

   

View Full Report ↗︎ ⋅ Docs

A new stamphog review started for this PR — the fresh verdict replaces this approval.

A new stamphog review started for this PR — the fresh verdict replaces this approval.

A new stamphog review started for this PR — the fresh verdict replaces this approval.

A new stamphog review started for this PR — the fresh verdict replaces this approval.

[stamphog]

Approved.

Follows the codebase's established third-party HTTP source pattern with careful secret handling (OAuth token scrubbing, redirect/host pinning, credential redaction), backed by 32 new unit tests; author is on the owning team with STRONG familiarity (88% of touched lines, 96 merged PRs in this area), and a security-review bot found no open concerns.

• Author wrote 88% of the modified lines and has 96 merged PRs in these paths (familiarity STRONG).

Gate mechanics and policy version

Gate		Result
prerequisites	✓	all clear
deny-list	✓	no deny categories matched
size	✓	518L, 5F substantive, 804L/8F incl. docs/generated/snapshots — within ceiling
tier	✓	T1-agent / T1d-complex (804L, 8F, single-area, feat)
stamphog 2.2.0		.stamphog/policy.yml @ 0d59103 · reviewed head 0d59103

[Gilbert09]

Summary of changes made to get this PR to a clean, mergeable state:

• Security (host pinning): recall_ai_source's client config now sets allowed_hosts: [] and allow_redirects: False, so a forged/off-host next pagination link or a restored resume checkpoint can't carry the Recall.ai API key off the selected region. Defensively added allow_redirects=False to the credential probe too.
• Security (sample capture): the calendars endpoint (the only one with scrub_fields set) now sets capture: False on its client, so raw HTTP samples never store the OAuth client secret/refresh token that _scrub_fields removes from rows.
• CI fix: two mypy failures introduced by this PR's own tests — cast SourceResponse.items() to Iterable[Any] before iterating (matching the existing aftership/autumn convention), and added # type: ignore[arg-type] on the intentionally-invalid region literal used to test rejection (matching docuseal).
• All 3 inline review threads (1 CodeRabbit, 2 veria-ai) are addressed and resolved.
• Branch kept up to date with master (several server-side updates as master moved during CI waits); no unrelated files leaked into the diff.
• CI is green on the current head; one transient Depot CI infra cancellation (unrelated to this PR's files) was cleared by a subsequent branch update, which re-ran CI cleanly.

_{🦉 via talyn.dev}

[Gilbert09]

/trunk merge

[deployment-status-posthog]

Deploy status

Environment	Status	Deployed At	Workflow
dev	✅ Deployed	2026-09-27 15:55 UTC	Run
prod-us	✅ Deployed	2026-09-27 16:05 UTC	Run
prod-eu	✅ Deployed	2026-09-27 16:05 UTC	Run