New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Feat: support filters and transformation #125

Open

pieterlukasse wants to merge 32 commits into master from feat/support_filters_and_transformation

Contributor

pieterlukasse commented Jan 10, 2025

Link to JIRA ticket if there is one: https://ctds-planx.atlassian.net/browse/VADC-1584

New Features

new support for filters and transformation arguments in histogram endpoint

Deployment changes

cohort-middleware might need an extra permission to create temp tables

pieterlukasse added 5 commits

December 31, 2024 14:29


          fix: fix the naming for concept lists

cb8d570

...from ids to definitions (defs)
and add some parsing for the new filter entries
in request body


          feat: working histogram endpoint POC for using tmp table for transfor…

e2fbc49

…mations


          feat: added first transformation

1090e84


          feat: added z-score transformation

d5935dd


          feat: update tempttablecache to include cleanup of oldest / least acc…

6e15ec8

…essed item

github-advanced-security bot found potential problems

View reviewed changes

models/helper.go Fixed Show resolved Hide resolved

pieterlukasse changed the title ~~Feat/support filters and transformation~~ Feat: support filters and transformation

pieterlukasse added 2 commits

January 10, 2025 20:58


          fix: fix code error

151b59f


          fix: fix tests and compilation errors

02f6f00

github-actions bot commented Jan 10, 2025

filepath	$$\textcolor{#23d18b}{\tt{passed}}$$	SUBTOTAL
$$\textcolor{#23d18b}{\tt{tests/test\_homepage.py}}$$	$$\textcolor{#23d18b}{\tt{1}}$$	$$\textcolor{#23d18b}{\tt{1}}$$
$$\textcolor{#23d18b}{\tt{TOTAL}}$$	$$\textcolor{#23d18b}{\tt{1}}$$	$$\textcolor{#23d18b}{\tt{1}}$$

Please find the detailed integration test report here

Please find the ci env pod logs here

pieterlukasse added 3 commits

January 14, 2025 20:04


          fix: fix to avoid double parsing of request body, which fails

e074037


          fix: fix parsing and failing tests

3cb31dc


          tmp: temporarily disable strict validation until frontend is fixed

903ec0b

github-actions bot commented Jan 14, 2025

filepath	$$\textcolor{#23d18b}{\tt{passed}}$$	SUBTOTAL
$$\textcolor{#23d18b}{\tt{tests/test\_homepage.py}}$$	$$\textcolor{#23d18b}{\tt{1}}$$	$$\textcolor{#23d18b}{\tt{1}}$$
$$\textcolor{#23d18b}{\tt{TOTAL}}$$	$$\textcolor{#23d18b}{\tt{1}}$$	$$\textcolor{#23d18b}{\tt{1}}$$

Please find the detailed integration test report here

Please find the ci env pod logs here


          fix: remove code duplication and fix filter_0 issue

680dad9

github-advanced-security bot found potential problems

View reviewed changes

models/helper.go Dismissed Show dismissed Hide dismissed

github-actions bot commented Jan 16, 2025

Please find the ci env pod logs here

pieterlukasse added 2 commits

January 16, 2025 17:04


          fix: improve test code

d2ce4cf


          fix: using ORM methods instead of raw query

835006e

github-actions bot commented Jan 16, 2025

filepath	$$\textcolor{#23d18b}{\tt{passed}}$$	SUBTOTAL
$$\textcolor{#23d18b}{\tt{tests/test\_homepage.py}}$$	$$\textcolor{#23d18b}{\tt{1}}$$	$$\textcolor{#23d18b}{\tt{1}}$$
$$\textcolor{#23d18b}{\tt{TOTAL}}$$	$$\textcolor{#23d18b}{\tt{1}}$$	$$\textcolor{#23d18b}{\tt{1}}$$

Please find the detailed integration test report here

Please find the ci env pod logs here


          feat: add some debug log statements

32dd028

github-actions bot commented Jan 16, 2025

filepath	$$\textcolor{#23d18b}{\tt{passed}}$$	SUBTOTAL
$$\textcolor{#23d18b}{\tt{tests/test\_homepage.py}}$$	$$\textcolor{#23d18b}{\tt{1}}$$	$$\textcolor{#23d18b}{\tt{1}}$$
$$\textcolor{#23d18b}{\tt{TOTAL}}$$	$$\textcolor{#23d18b}{\tt{1}}$$	$$\textcolor{#23d18b}{\tt{1}}$$

Please find the detailed integration test report here

Please find the ci env pod logs here

pieterlukasse added 3 commits

January 16, 2025 20:27


          tmp: try different tosql method

7c4d45a


          tmp: remove interpolate

f3951c2


          feat: simplify toSQL

f790702

github-actions bot commented Jan 16, 2025

filepath	$$\textcolor{#23d18b}{\tt{passed}}$$	SUBTOTAL
$$\textcolor{#23d18b}{\tt{tests/test\_homepage.py}}$$	$$\textcolor{#23d18b}{\tt{1}}$$	$$\textcolor{#23d18b}{\tt{1}}$$
$$\textcolor{#23d18b}{\tt{TOTAL}}$$	$$\textcolor{#23d18b}{\tt{1}}$$	$$\textcolor{#23d18b}{\tt{1}}$$

Please find the detailed integration test report here

Please find the ci env pod logs here


          feat: add support for sqlserver

802e335

github-actions bot commented Jan 20, 2025

filepath	$$\textcolor{#23d18b}{\tt{passed}}$$	SUBTOTAL
$$\textcolor{#23d18b}{\tt{tests/test\_homepage.py}}$$	$$\textcolor{#23d18b}{\tt{1}}$$	$$\textcolor{#23d18b}{\tt{1}}$$
$$\textcolor{#23d18b}{\tt{TOTAL}}$$	$$\textcolor{#23d18b}{\tt{1}}$$	$$\textcolor{#23d18b}{\tt{1}}$$

Please find the detailed integration test report here

Please find the ci env pod logs here

pieterlukasse added 2 commits

January 20, 2025 21:18


          fix: fix tests and fix temp table name

b8c2fea


          fix: avoid negative or 0's for log transformation; fix sqlserver syntax

e0fd0d4

github-actions bot commented Jan 20, 2025

filepath	$$\textcolor{#23d18b}{\tt{passed}}$$	SUBTOTAL
$$\textcolor{#23d18b}{\tt{tests/test\_homepage.py}}$$	$$\textcolor{#23d18b}{\tt{1}}$$	$$\textcolor{#23d18b}{\tt{1}}$$
$$\textcolor{#23d18b}{\tt{TOTAL}}$$	$$\textcolor{#23d18b}{\tt{1}}$$	$$\textcolor{#23d18b}{\tt{1}}$$

Please find the detailed integration test report here

Please find the ci env pod logs here


          fix: fix log and stdev queries

bc5394e

github-actions bot commented Jan 21, 2025

filepath	$$\textcolor{#23d18b}{\tt{passed}}$$	SUBTOTAL
$$\textcolor{#23d18b}{\tt{tests/test\_homepage.py}}$$	$$\textcolor{#23d18b}{\tt{1}}$$	$$\textcolor{#23d18b}{\tt{1}}$$
$$\textcolor{#23d18b}{\tt{TOTAL}}$$	$$\textcolor{#23d18b}{\tt{1}}$$	$$\textcolor{#23d18b}{\tt{1}}$$

Please find the detailed integration test report here

Please find the ci env pod logs here


          fix: fix temp table alias

3d11a93

github-actions bot commented Jan 21, 2025

filepath	$$\textcolor{#23d18b}{\tt{passed}}$$	SUBTOTAL
$$\textcolor{#23d18b}{\tt{tests/test\_homepage.py}}$$	$$\textcolor{#23d18b}{\tt{1}}$$	$$\textcolor{#23d18b}{\tt{1}}$$
$$\textcolor{#23d18b}{\tt{TOTAL}}$$	$$\textcolor{#23d18b}{\tt{1}}$$	$$\textcolor{#23d18b}{\tt{1}}$$

Please find the detailed integration test report here

Please find the ci env pod logs here


          test: using global temp table

fc14ccd

github-actions bot commented Jan 21, 2025

filepath	$$\textcolor{#23d18b}{\tt{passed}}$$	SUBTOTAL
$$\textcolor{#23d18b}{\tt{tests/test\_homepage.py}}$$	$$\textcolor{#23d18b}{\tt{1}}$$	$$\textcolor{#23d18b}{\tt{1}}$$
$$\textcolor{#23d18b}{\tt{TOTAL}}$$	$$\textcolor{#23d18b}{\tt{1}}$$	$$\textcolor{#23d18b}{\tt{1}}$$

Please find the detailed integration test report here

Please find the ci env pod logs here

pieterlukasse added 2 commits

January 22, 2025 11:21


          test: try cloning the session

efd202e


          feat: remove confusing code; test without dryrun

6b02d5c

github-actions bot commented Jan 22, 2025

Please find the ci env pod logs here


          feat: wrap all in single transaction

674a0fc

github-actions bot commented Jan 22, 2025

filepath	$$\textcolor{#23d18b}{\tt{passed}}$$	SUBTOTAL
$$\textcolor{#23d18b}{\tt{tests/test\_homepage.py}}$$	$$\textcolor{#23d18b}{\tt{1}}$$	$$\textcolor{#23d18b}{\tt{1}}$$
$$\textcolor{#23d18b}{\tt{TOTAL}}$$	$$\textcolor{#23d18b}{\tt{1}}$$	$$\textcolor{#23d18b}{\tt{1}}$$

Please find the detailed integration test report here

Please find the ci env pod logs here


          feat: adjust temp table usage to check for table existence first

0763d91

github-actions bot commented Jan 23, 2025

Please find the ci env pod logs here


          fix: fix scenario with multiple filters

96d3cd8

github-actions bot commented Jan 23, 2025

filepath	$$\textcolor{#23d18b}{\tt{passed}}$$	SUBTOTAL
$$\textcolor{#23d18b}{\tt{tests/test\_homepage.py}}$$	$$\textcolor{#23d18b}{\tt{1}}$$	$$\textcolor{#23d18b}{\tt{1}}$$
$$\textcolor{#23d18b}{\tt{TOTAL}}$$	$$\textcolor{#23d18b}{\tt{1}}$$	$$\textcolor{#23d18b}{\tt{1}}$$

Please find the detailed integration test report here

Please find the ci env pod logs here


          fix: reset the error state for the table exists check

98baf7b

github-actions bot commented Jan 23, 2025

Please find the ci env pod logs here


          feat: remove histogramConceptId from histogram endpoint

8a5dbe8

...and expect it as part of the variables instead

github-actions bot commented Jan 24, 2025

filepath	$$\textcolor{#23d18b}{\tt{passed}}$$	SUBTOTAL
$$\textcolor{#23d18b}{\tt{tests/test\_homepage.py}}$$	$$\textcolor{#23d18b}{\tt{1}}$$	$$\textcolor{#23d18b}{\tt{1}}$$
$$\textcolor{#23d18b}{\tt{TOTAL}}$$	$$\textcolor{#23d18b}{\tt{1}}$$	$$\textcolor{#23d18b}{\tt{1}}$$

Please find the detailed integration test report here

Please find the ci env pod logs here


          feat: remove histogram concept id from histogram endpoint

1473df6

... and read from concept variables list instead

github-actions bot commented Jan 24, 2025

filepath	$$\textcolor{#23d18b}{\tt{passed}}$$	SUBTOTAL
$$\textcolor{#23d18b}{\tt{tests/test\_homepage.py}}$$	$$\textcolor{#23d18b}{\tt{1}}$$	$$\textcolor{#23d18b}{\tt{1}}$$
$$\textcolor{#23d18b}{\tt{TOTAL}}$$	$$\textcolor{#23d18b}{\tt{1}}$$	$$\textcolor{#23d18b}{\tt{1}}$$

Please find the detailed integration test report here

Please find the ci env pod logs here

pieterlukasse commented

View reviewed changes

models/helper.go Outdated Show resolved Hide resolved


          feat: cleanup / review todos

bd144e7

pieterlukasse requested a review from tianj7

January 29, 2025 20:12

pieterlukasse marked this pull request as ready for review

January 29, 2025 20:12

github-actions bot commented Jan 29, 2025

Please find the ci env pod logs here

pieterlukasse mentioned this pull request

Feat: add transformation dropdown to histogram uc-cdis/data-portal#1666

Open

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet