[DRAFT] fix: Use CTE in Enrollment analytics queries [DHIS-16705]

[luciano-fiandesio]

WIP

Changes

This PR addresses the generation of SQL queries for Enrollment analytics

Problem

Currently, the Enrollment queries are structured so that sub-queries are used to fetch events values from the analytics_event_* tables.
For instance:

select
    enrollment,
    ...
    ax."ou",
    (select
         "fyjPqlHE7Dn"
     from
	 analytics_event_M3xtLkYBlKI
     where
	 analytics_event_M3xtLkYBlKI.eventstatus != 'SCHEDULE'
	 and analytics_event_M3xtLkYBlKI.enrollment = ax.enrollment
	 and ps = 'CWaAcQYKVpq'
limit 1 ) as "CWaAcQYKVpq[-1].fyjPqlHE7Dn",
from
     analytics_enrollment_m3xtlkyblki as ax
...

The above query works in Postgres but does not work in Doris, because correlation with outer layers of the parent query is not supported.

Mitigation

The current approach is trying to refactor the Enrollment query so that the subqueries (both as select and as where conditions) are "moved" into CTE (Common Table Expressions).
Common Table Expressions are temporary result sets in SQL, defined within a WITH clause, that simplify complex queries by improving readability and modularizing logic. They are particularly useful for recursive queries and can be referenced multiple times within the main query.

The above query can be rewritten like so:

with ps_cwaacqykvpq_fyjpqlhe7dn as (
select
	distinct on
	(enrollment) enrollment,
	"fyjPqlHE7Dn" as value
from
	analytics_event_M3xtLkYBlKI
where
	eventstatus != 'SCHEDULE'
	and ps = 'CWaAcQYKVpq'
	
order by
	enrollment,
	occurreddate desc,
	created desc)
select
	ax.enrollment,
	...
	ps_cwaacqykvpq_fyjpqlhe7dn.value as "CWaAcQYKVpq.fyjPqlHE7Dn"
from
	analytics_enrollment_m3xtlkyblki as ax
left join ps_cwaacqykvpq_fyjpqlhe7dn on
	ax.enrollment = ps_cwaacqykvpq_fyjpqlhe7dn.enrollment
where
	(((enrollmentdate >= '2021-01-01'
	and enrollmentdate < '2022-01-01')))
	and (ax."uidlevel1" = 'ImspTQPwCqd' )
...

The above query structure is compatible with Doris and it makes the execution of the query faster.
As a comparison:

Original query with sub-select

Refactored query with CTEs

Testing strategy

I am using the e2e project and aim at having 100% green tests on both Postgres and Doris.

To fix the problem, we need to validate the offset parameter before performing arithmetic operations on it. Specifically, we should ensure that the offset value is within a safe range to prevent overflow or underflow. We can achieve this by adding a guard clause that checks the value of offset and handles extreme values appropriately.

The best way to fix the problem without changing existing functionality is to add a validation step in the computeRowNumberOffset method. We will check if the offset value is within the range of Integer.MIN_VALUE to Integer.MAX_VALUE and handle cases where it is not.

[larshelge]

Rename to CteContext. Use Pascal-case for acronyms in class names in Java.

https://www.oreilly.com/library/view/java-8-pocket/9781491901083/ch01.html

[larshelge]

Rename to addCte.

[luciano-fiandesio]

Fixed, renamed also other methods CTE -> Cte on the same class

[larshelge]

Let's use lower-case for SQL keywords in Java, i.e. with.

[larshelge]

Lower-case for SQL keywords in Java, e.g. in.

[larshelge]

Use !isEmpty() over .size() > 0 for readability.

[luciano-fiandesio]

Removed function, as it looks like it wasn't used

[sonarqubecloud]

Quality Gate failed

Failed conditions
23 New issues
23 New Code Smells (required ≤ 0)

See analysis details on SonarQube Cloud

Catch issues before they fail your Quality Gate with our IDE extension SonarQube for IDE