Optimize audit log filter-attribute query (6 scans → 1)

[priyamkarn]

Summary

Optimizes queryUniqueFilterAttributes in:

server/routers/auditLogs/queryRequestAuditLog.ts

Previously, the function executed six separate SELECT DISTINCT queries in parallel to populate the request audit log filter options:

• Actors
• Locations
• Hosts
• Paths
• Resources
• Site resources

This resolves the existing:

// TODO: SOMEONE PLEASE OPTIMIZE THIS!!!!!

Problem

All six queries used the same orgId and time-range filters.

However, the table only has indexes on:

• timestamp
• (orgId, timestamp)

There are no indexes on actor, location, host, path, resourceId, or siteResourceId.

As a result, each DISTINCT query had to independently scan the same matched row set. Every request to this endpoint therefore performed six database scans and six round trips over effectively the same underlying dataset.

Fix

Replaced the six separate queries with a single UNION ALL query.

The query contains six subqueries—one per filter field—with each subquery:

• Applying the same organization and time-range filters
• Performing its own GROUP BY <field>
• Applying its own LIMIT
• Adding a field discriminator column so the combined result can be separated in application code

resourceId and siteResourceId are cast to text inside their respective subqueries so that all six UNION ALL branches return compatible column types. They are parsed back into numbers after the query returns.

This keeps distinct-value computation inside the database rather than pulling every matched audit-log row into application memory.

That is important because audit logs typically have a high row count but low-to-moderate cardinality for individual filter fields. For example, a time range may contain millions of requests but only a small number of distinct paths. Deduplicating in application memory would require transferring the entire matched dataset across the wire just to derive a limited number of filter values.

The new approach reduces the number of database round trips from six to one while keeping every field's result set bounded by DISTINCT_LIMIT, matching the original per-query limits.

Credit to @oschwartz10612 for suggesting this approach during review. An earlier version of the PR used a single scan followed by in-memory Set deduplication, which reduced database scans but introduced an unbounded network and application-memory footprint for large time ranges. This version avoids that trade-off.

Behavior

Existing behavior is preserved:

• Each field remains capped at DISTINCT_LIMIT entries—currently 500—matching the previous per-query limit.
• The resourceId and siteResourceId mutual-exclusivity rule is unchanged.
• A siteResourceId is only included when resourceId is null, matching the original isNull(resourceId) condition.
• Resource and site-resource name lookups against the primary database are unchanged.
• Sorting is unchanged.
• The final API response shape is unchanged.

Cleanup

Adjusted the isNull import usage to support the siteResourceId subquery's null condition.

Testing

• Verified the generated SQL for both SQLite and PostgreSQL using Drizzle's .toSQL().
• Confirmed that both dialects produce valid, parameterized queries.
• Ran the query logic against a seeded in-memory SQLite database containing 20,000 rows, including rows outside the target organization and time range.
• Cross-checked all six result sets against a brute-force reference implementation.
• The optimized query matched the reference output exactly across all six fields.

I was unable to run the full test suite in my local environment, so a CI run would be appreciated before merging.

Scope / Risk

• Limited to the request audit log filter-options query.
• No API or response-shape changes.
• No database schema changes.
• No index changes.
• No migrations required.

[priyamkarn]

@oschwartz10612 @miloschwartz can you please review and merge this

[oschwartz10612]

Haha thanks for taking a look at that comment. I am thinking longer term, the actual fix for the underlying TODO is probably adding an index (or narrowing what needs to be indexed) rather than moving aggregation into the app tier. Or a single query using array_agg(DISTINCT ...) (or similar) per column, or a UNION ALL of six small LIMIT-ed subqueries, so distinctness is still computed in Postgres and only small result sets cross the wire - genuinely 1 round trip without the memory blowup

[priyamkarn]

@oschwartz10612 can you please review now