Skip to content
Closed
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
78 commits
Select commit Hold shift + click to select a range
093308d
feat(warehouse-sources): start new postgres cdc sources on buffered i…
danielcarletti Sep 22, 2026
fb7f1ec
Merge remote-tracking branch 'origin/master' into claude/cdc-new-sour…
danielcarletti Sep 22, 2026
ca6259c
feat(warehouse-sources): recreated cdc slots start on buffered ingress
danielcarletti Sep 22, 2026
e4eac23
feat(warehouse-sources): capture snapshotting cdc tables to the buffer
danielcarletti Sep 23, 2026
dc0c776
fix(warehouse-sources): gate snapshot buffering and purge truncates b…
danielcarletti Sep 23, 2026
d9f7914
fix(warehouse-sources): purge every cdc table's buffer on repair
danielcarletti Sep 23, 2026
6c469d8
Merge remote-tracking branch 'origin/claude/cdc-new-sources-buffered'…
danielcarletti Sep 23, 2026
68e9c60
fix(warehouse-sources): make buffered cdc prerequisites hold without …
danielcarletti Sep 23, 2026
2915bc1
Merge remote-tracking branch 'origin/claude/cdc-new-sources-buffered'…
danielcarletti Sep 23, 2026
c138742
Merge remote-tracking branch 'origin/master' into claude/cdc-buffered…
danielcarletti Sep 23, 2026
d4a09ea
Merge remote-tracking branch 'origin/master' into claude/cdc-buffered…
danielcarletti Sep 23, 2026
aaf447e
fix(warehouse-sources): route each cdc snapshot through one lane and …
danielcarletti Sep 23, 2026
1c267bf
fix(warehouse-sources): purge the cdc buffer before marking a reset s…
danielcarletti Sep 24, 2026
455b7c1
fix(warehouse-sources): hold the schema lock across the cdc snapshot …
danielcarletti Sep 24, 2026
b1c13fb
feat(warehouse-sources): remove the legacy cdc lane
danielcarletti Sep 24, 2026
0db5e72
fix(warehouse-sources): keep a resynced streaming cdc table in the bu…
danielcarletti Sep 24, 2026
840997b
fix(warehouse-sources): close the remaining gaps in cdc buffered snap…
danielcarletti Sep 24, 2026
6a078f7
Merge remote-tracking branch 'origin/master' into claude/cdc-buffered…
danielcarletti Sep 24, 2026
877c4ec
chore(warehouse-sources): describe the cdc snapshot marker gaps and r…
danielcarletti Sep 24, 2026
2b86d2c
Merge branch 'claude/cdc-buffered-snapshot-handoff' into claude/cdc-r…
danielcarletti Sep 24, 2026
c4e0995
fix(warehouse-sources): wait for a legacy snapshot to stop before its…
danielcarletti Sep 24, 2026
a23c964
Merge remote-tracking branch 'origin/master' into claude/cdc-remove-l…
danielcarletti Sep 24, 2026
3ac7553
fix(warehouse-sources): name the cdc sync frequency limit as the ui does
danielcarletti Sep 24, 2026
b805b90
feat(hogql): add a useNewEventsSchema query modifier
aspicer Sep 24, 2026
f9a588d
chore: update OpenAPI generated types
tests-posthog[bot] Sep 24, 2026
fdc5e43
fix(warehouse-sources): keep a legacy source's buffer unread until ca…
danielcarletti Sep 25, 2026
a6b6c38
Merge remote-tracking branch 'origin/master' into claude/cdc-remove-l…
danielcarletti Sep 25, 2026
2bc40af
fix(warehouse-sources): keep capture retries and held restarts from s…
danielcarletti Sep 25, 2026
430b461
Merge remote-tracking branch 'origin/master' into claude/cdc-remove-l…
danielcarletti Sep 25, 2026
f665c77
fix(warehouse-sources): treat a capture failure outside an activity a…
danielcarletti Sep 25, 2026
e522735
Merge remote-tracking branch 'origin/master' into claude/cdc-remove-l…
danielcarletti Sep 25, 2026
c8d914d
Merge remote-tracking branch 'origin/master' into claude/cdc-remove-l…
danielcarletti Sep 25, 2026
67e1cc8
chore(warehouse-sources): tell apart request and capture resets in th…
danielcarletti Sep 25, 2026
433202f
fix(hogql): read moved event properties from temporary_properties
aspicer Sep 25, 2026
1267bd4
fix(replay-vision): stop search results crashing when a recording exp…
posthog[bot] Sep 26, 2026
0656288
fix(hogql): use the source funnel's modifiers in correlation queries
aspicer Sep 28, 2026
46ed044
chore(hogql): document the project fallback for useNewEventsSchema
aspicer Sep 28, 2026
4e960c0
fix(hogql): keep moved property siblings readable under restrictions
aspicer Sep 28, 2026
1cf897a
Merge remote-tracking branch 'origin/master' into aspicer/hogql-nativ…
aspicer Sep 28, 2026
40d296f
fix(hogql): give the actors options runner an explicit signature
aspicer Sep 28, 2026
a13bfc2
Merge remote-tracking branch 'origin/master' into aspicer/new-events-…
aspicer Sep 28, 2026
8e6dc05
chore(hogql): move the wrapper modifier test into product analytics
aspicer Sep 28, 2026
c93cf67
Merge origin/master into claude/cdc-remove-legacy-lane
danielcarletti Sep 28, 2026
fb4904a
Merge origin/master into claude/cdc-remove-legacy-lane
danielcarletti Sep 28, 2026
0d5aae9
Merge origin/master into claude/cdc-remove-legacy-lane
danielcarletti Sep 28, 2026
bf668e3
Merge origin/master into claude/cdc-remove-legacy-lane
danielcarletti Sep 28, 2026
bdb7d79
Merge origin/master into claude/cdc-remove-legacy-lane
danielcarletti Sep 28, 2026
e510bb0
Merge remote-tracking branch 'origin/master' into aspicer/new-events-…
aspicer Sep 29, 2026
2d6d49b
fix(hogql): keep the useNewEventsSchema project value staff-only
aspicer Sep 29, 2026
65414f1
Merge remote-tracking branch 'origin/master' into aspicer/hogql-nativ…
aspicer Sep 29, 2026
dfa6621
fix(hogql): read moved keys through JSON_VALUE and reject computed keys
aspicer Sep 29, 2026
6f929c0
fix(warehouse-sources): keep converted legacy cdc tables current and …
danielcarletti Sep 29, 2026
a6b5711
Merge origin/master into claude/cdc-remove-legacy-lane
danielcarletti Sep 29, 2026
4da0e6f
fix(mcp): expire cached personal API key scopes after 2 minutes
posthog[bot] Sep 24, 2026
35cda2b
chore(mcp): drop unreachable setup from the api key refresh test
posthog[bot] Sep 24, 2026
396c87b
fix(mcp): refresh scopes only for personal API keys
JakeRuth Sep 29, 2026
1ee3b01
fix(workflows): pass the consent mode default as a value, not a member
webjunkie Sep 29, 2026
69f9da7
fix(growth): label the newest enrichment fetches first
MattBro Sep 29, 2026
6f26437
Merge remote-tracking branch 'origin/master' into posthog-self-drivin…
clr182 Sep 29, 2026
5e1b6a9
fix(web-analytics): export web vitals tiles in the csv download
jordanm-posthog Sep 29, 2026
f7ac988
fix(growth): share the windowed fetch query and pin walk order in tests
MattBro Sep 29, 2026
481030a
fix(web-analytics): type web vitals export test insight props
jordanm-posthog Sep 29, 2026
80afa8b
Merge remote-tracking branch 'refs/remotes/origin/master' into claude…
danielcarletti Sep 29, 2026
defbdd2
chore(growth): fold the lookback test into the walk-order cases
MattBro Sep 29, 2026
e52866d
fix(signals): filter inbox pr reports by team before joining
posthog[bot] Sep 29, 2026
116f901
chore(signals): trim the equals-any lookup
posthog[bot] Sep 29, 2026
81d4786
Merging 396c87b5c4f84b09a7b280fe164d11f562ed18f3 into trunk-temp/pr-1…
trunk-io[bot] Sep 29, 2026
124f0ae
Merging 2d6d49b42bbf591512aa55b33a6d85b84bc59cf3 into trunk-temp/pr-1…
trunk-io[bot] Sep 29, 2026
e94ba53
Merging 80afa8beaaf72a1939a8478a08278d6851aeb5ce into trunk-temp/pr-1…
trunk-io[bot] Sep 29, 2026
8d38680
Merging 6f26437a51168961302a8b181ebb8ce486b64f04 into trunk-temp/pr-1…
trunk-io[bot] Sep 29, 2026
0439f76
Merging 8d38680f4cb91fdbab2d63b75442af4ae0d88a34 into trunk-temp/pr-1…
trunk-io[bot] Sep 29, 2026
c7cc5f7
Merging 481030a208d520f13ea9ebd31263b4f1d7e34649 into trunk-temp/pr-1…
trunk-io[bot] Sep 29, 2026
820e862
Merging dfa6621d49f51ed9a00076fe9ab428821fb7e1ce into trunk-temp/pr-1…
trunk-io[bot] Sep 29, 2026
2a29b54
Merging defbdd26c66287e2c1a6db619a289d4661567070 into trunk-temp/pr-1…
trunk-io[bot] Sep 29, 2026
13f5f34
Merging 116f90168fbb020ea4a6f7a2e433c26a0dad4341 into trunk-temp/pr-1…
trunk-io[bot] Sep 29, 2026
447c5e0
Merging 13f5f34da6c61f2d774f53e13408d007313cc507 into trunk-temp/pr-1…
trunk-io[bot] Sep 29, 2026
92b87c1
Merging 1ee3b01ddea1c9289cb0214d6b93fecd1370b6a4 into trunk-temp/pr-1…
trunk-io[bot] Sep 29, 2026
a0bbfcc
chore: update OpenAPI generated types
tests-posthog[bot] Sep 29, 2026
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions .github/new-events-schema-targets.txt
Original file line number Diff line number Diff line change
Expand Up @@ -40,6 +40,7 @@ products/web_analytics/backend/hogql_queries

# Code outside HogQL that builds events SQL itself and branches on the schema
posthog/dags/tests/test_deletes.py
posthog/models/person/test/test_point_in_time_properties_integration.py
posthog/models/team/test_production_event_activation.py
posthog/models/test/test_deletion_targets.py
posthog/tasks/test/test_ai_observability_usage_report.py
Expand Down
282 changes: 74 additions & 208 deletions docs/internal/cdc-buffered-ingress-runbook.md

Large diffs are not rendered by default.

4 changes: 4 additions & 0 deletions docs/internal/clickhouse-utility-udfs.md
Original file line number Diff line number Diff line change
Expand Up @@ -73,6 +73,10 @@ Both functions use the same executable. The temporary entry point uses `--tempor
Documents exceeding the shared depth limit produce `{}` in the temporary output; the permanent cleaner quarantines the original document.

Native events retain `temporary_properties` for 60 days after insertion, including historical events; TTL merges clear the column asynchronously.
On native events, HogQL reads a property in this allowlist from `temporary_properties`: property access such as `properties.$set.email`, filters, `JSONHas`, `JSONLength`, `JSONType`, the `JSONExtract*` functions with the property as their first key, and `JSON_VALUE` with the property as the first member of its path.
These functions reject a first key computed per row, because such a key can name a moved property.
Whole-document reads of `properties`, such as `SELECT properties`, do not include these properties, because rebuilding them would serialize `temporary_properties` on every row read.
`is_temporary_event_property` in `posthog/clickhouse/events_json.py` mirrors the allowlist, so update both together.
Fresh installations use the updated schema definitions. Existing tables require a manual schema rollout and feature-flag query compatibility before native reads are enabled.

Native-event queries derive `$active_feature_flags` from the `$feature_flags` map, excluding empty and `false` values and restricted flags. `$false` counts as active, and reads of `$feature/<key>`, `$feature_flags.<key>` and the `$feature_flags` map return it as `false`. Whole-document reads of `properties`, and batch exports of the whole `$feature_flags` map, return it as stored; single-flag export fields and filters map it back. Array order follows the stored map rather than the original SDK evaluation order. No separate active-flags column is required.
Expand Down
8 changes: 6 additions & 2 deletions frontend/src/queries/nodes/WebVitals/WebVitals.tsx
Original file line number Diff line number Diff line change
Expand Up @@ -10,6 +10,7 @@ import { addProductIntentForCrossSell } from 'lib/utils/product-intents'
import { urls } from 'scenes/urls'
import { webAnalyticsLogic } from 'scenes/web-analytics/webAnalyticsLogic'

import { insightVizDataNodeKey } from '~/queries/nodes/InsightViz/insightVizKeys'
import { Query } from '~/queries/Query/Query'
import {
AnyResponseType,
Expand All @@ -32,8 +33,11 @@ export function WebVitals(props: {
context: QueryContext
attachTo?: LogicWrapper | BuiltLogic
}): JSX.Element | null {
const { onData, loadPriority, dataNodeCollectionId } = props.context.insightProps ?? {}
const [key] = useState(() => `WebVitals.${uniqueNode++}`)
const { insightProps } = props.context
const { onData, loadPriority, dataNodeCollectionId } = insightProps ?? {}
const [key] = useState(() =>
insightProps?.dashboardItemId ? insightVizDataNodeKey(insightProps) : `WebVitals.${uniqueNode++}`
)

const logic = dataNodeLogic({
query: props.query,
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -8,6 +8,7 @@ import { LemonSkeleton } from 'lib/lemon-ui/LemonSkeleton'
import { useAttachedLogic } from 'lib/logic/scenes/useAttachedLogic'
import { webAnalyticsLogic } from 'scenes/web-analytics/webAnalyticsLogic'

import { insightVizDataNodeKey } from '~/queries/nodes/InsightViz/insightVizKeys'
import {
AnyResponseType,
WebAnalyticsPreComputeStrategy,
Expand All @@ -34,8 +35,11 @@ export function WebVitalsPathBreakdown(props: {
context: QueryContext
attachTo?: LogicWrapper | BuiltLogic
}): JSX.Element | null {
const { onData, loadPriority, dataNodeCollectionId } = props.context.insightProps ?? {}
const [key] = useState(() => `WebVitalsPathBreakdown.${uniqueNode++}`)
const { insightProps } = props.context
const { onData, loadPriority, dataNodeCollectionId } = insightProps ?? {}
const [key] = useState(() =>
insightProps?.dashboardItemId ? insightVizDataNodeKey(insightProps) : `WebVitalsPathBreakdown.${uniqueNode++}`
)

const logic = dataNodeLogic({
query: props.query,
Expand Down
4 changes: 4 additions & 0 deletions frontend/src/queries/schema.json

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

2 changes: 2 additions & 0 deletions frontend/src/queries/schema/schema-general.ts
Original file line number Diff line number Diff line change
Expand Up @@ -533,6 +533,8 @@ export interface HogQLQueryModifiers {
materializedColumnsOptimizationMode?: 'disabled' | 'optimized'
propertyGroupsMode?: 'enabled' | 'disabled' | 'optimized'
useMaterializedViews?: boolean
/** Read events from the native JSON events table (`true`) or the legacy events table (`false`). When unset, the project's stored value applies, then the `CLICKHOUSE_HOGQL_USE_NEW_EVENTS_SCHEMA` instance settings. This is an internal rollout switch. PostHog staff set the project value in Django admin and the project settings API ignores it. */
useNewEventsSchema?: boolean
customChannelTypeRules?: CustomChannelRule[]
customBotDefinitions?: CustomBotRule[]
/** Do not treat a missing user agent as automation on cookieless events. Positive bot signals and custom project rules still apply. Resolved server-side; not intended to be set by clients. */
Expand Down
142 changes: 140 additions & 2 deletions frontend/src/scenes/web-analytics/WebAnalyticsExport.test.tsx
Original file line number Diff line number Diff line change
@@ -1,6 +1,68 @@
import { DataTableNode, NodeKind, WebStatsBreakdown, WebStatsTableQueryResponse } from '~/queries/schema/schema-general'
import { dataNodeLogic } from '~/queries/nodes/DataNode/dataNodeLogic'
import { insightVizDataNodeKey } from '~/queries/nodes/InsightViz/insightVizKeys'
import {
DataTableNode,
NodeKind,
WebStatsBreakdown,
WebStatsTableQueryResponse,
WebVitalsPathBreakdownQuery,
WebVitalsPathBreakdownQueryResponse,
WebVitalsQuery,
WebVitalsQueryResponse,
} from '~/queries/schema/schema-general'
import { initKeaTests } from '~/test/init'
import { InsightLogicProps, PropertyMathType } from '~/types'

import { WebAnalyticsTableAdapter, buildCsvFilenames } from './webAnalyticsExportUtils'
import { TileId, WebAnalyticsTile } from './common'
import {
WebAnalyticsTableAdapter,
buildCsvFilenames,
collectAllTilesTableData,
getExportAdapter,
} from './webAnalyticsExportUtils'

const webVitalsQuery: WebVitalsQuery = {
kind: NodeKind.WebVitalsQuery,
properties: [],
source: {
kind: NodeKind.TrendsQuery,
series: [],
},
}

const webVitalsResponse: WebVitalsQueryResponse = {
results: [
{
action: { custom_name: 'LCP', math: PropertyMathType.P90 },
days: ['2026-09-01', '2026-09-02'],
data: [2400, 2600],
},
{
action: { custom_name: 'CLS', math: PropertyMathType.P90 },
days: ['2026-09-01', '2026-09-02'],
data: [0.04, 0.05],
},
],
}

const pathBreakdownQuery: WebVitalsPathBreakdownQuery = {
kind: NodeKind.WebVitalsPathBreakdownQuery,
dateRange: { date_from: '-7d' },
properties: [],
percentile: PropertyMathType.P90,
metric: 'FCP',
thresholds: [1800, 3000],
}

const pathBreakdownResponse: WebVitalsPathBreakdownQueryResponse = {
results: [
{
good: [{ path: '/pricing', value: 900 }],
needs_improvements: [],
poor: [{ path: '/', value: 13320 }],
},
],
}

describe('WebAnalyticsExport adapters', () => {
describe('buildCsvFilenames', () => {
Expand Down Expand Up @@ -121,4 +183,80 @@ describe('WebAnalyticsExport adapters', () => {
])
})
})

describe('web vitals adapters', () => {
it.each([
{
name: 'timeseries',
response: webVitalsResponse,
query: webVitalsQuery,
expected: [
['Date', 'LCP p90 (ms)', 'CLS p90'],
['2026-09-01', '2400', '0.04'],
['2026-09-02', '2600', '0.05'],
],
},
{
name: 'timeseries without results',
response: { results: [] },
query: webVitalsQuery,
expected: [],
},
{
name: 'path breakdown',
response: pathBreakdownResponse,
query: pathBreakdownQuery,
expected: [
['Band', 'Path', 'FCP p90 (ms)'],
['Great', '/pricing', '900'],
['Poor', '/', '13320'],
],
},
{
name: 'path breakdown without paths',
response: { results: [{ good: [], needs_improvements: [], poor: [] }] },
query: pathBreakdownQuery,
expected: [],
},
])('converts $name', ({ response, query, expected }) => {
expect(getExportAdapter(response, query)?.toTableData()).toEqual(expected)
})

it('collects the web vitals tiles from their mounted data node logics', () => {
initKeaTests()
const webVitalsInsightProps: InsightLogicProps = { dashboardItemId: 'new-web-vitals-tile' }
const pathBreakdownInsightProps: InsightLogicProps = { dashboardItemId: 'new-web-vitals-path-tile' }
const tiles: WebAnalyticsTile[] = [
{
kind: 'query',
tileId: TileId.WEB_VITALS,
layout: {},
query: webVitalsQuery,
insightProps: webVitalsInsightProps,
},
{
kind: 'query',
tileId: TileId.WEB_VITALS_PATH_BREAKDOWN,
layout: {},
query: pathBreakdownQuery,
insightProps: pathBreakdownInsightProps,
},
]
dataNodeLogic({
key: insightVizDataNodeKey(webVitalsInsightProps),
query: webVitalsQuery,
cachedResults: webVitalsResponse,
}).mount()
dataNodeLogic({
key: insightVizDataNodeKey(pathBreakdownInsightProps),
query: pathBreakdownQuery,
cachedResults: pathBreakdownResponse,
}).mount()

expect(collectAllTilesTableData(tiles).map(({ title, tableData }) => [title, tableData.length])).toEqual([
['Web vitals', 3],
['Web vitals path breakdown: FCP', 3],
])
})
})
})
2 changes: 2 additions & 0 deletions frontend/src/scenes/web-analytics/WebAnalyticsMenu.tsx
Original file line number Diff line number Diff line change
Expand Up @@ -35,8 +35,9 @@
TileId.FRUSTRATING_PAGES,
]

export const WebAnalyticsMenu = (): JSX.Element => {

Check warning on line 38 in frontend/src/scenes/web-analytics/WebAnalyticsMenu.tsx

View workflow job for this annotation

GitHub Actions / Frontend formatting

lint:complexity

`WebAnalyticsMenu` has cyclomatic complexity 12 (warn >10)
const {
exportAllDisabledReason,
hasSavedFocusMode,
hiddenTiles,
isFocusModeActive,
Expand All @@ -63,6 +64,7 @@
<ButtonPrimitive
menuItem
data-attr="web-analytics-export-all-csv"
disabledReasons={exportAllDisabledReason ? { [exportAllDisabledReason]: true } : undefined}
onClick={() => {
if (exportAllTilesAsCsvZip(tiles)) {
exportTriggered()
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -89,8 +89,9 @@
return <WebAnalyticsSceneMenuBarInner />
}

function WebAnalyticsSceneMenuBarInner(): JSX.Element {

Check warning on line 92 in frontend/src/scenes/web-analytics/WebAnalyticsSceneMenuBar.tsx

View workflow job for this annotation

GitHub Actions / Frontend formatting

lint:complexity

`WebAnalyticsSceneMenuBarInner` has cyclomatic complexity 13 (warn >10)
const {
exportAllDisabledReason,
hasSavedFocusMode,
hiddenTiles,
isFocusModeActive,
Expand Down Expand Up @@ -166,6 +167,8 @@
</SceneMenuBarItem>
)}
<SceneMenuBarItem
disabled={!!exportAllDisabledReason}
tooltip={exportAllDisabledReason ?? undefined}
onClick={() => {
if (exportAllTilesAsCsvZip(tiles)) {
exportTriggered()
Expand Down
Loading
Loading