Skip to content

Add support for Dremio (update for merging PR 41) #44

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
erezsh merged 8 commits into from
Oct 23, 2025
Merged

Add support for Dremio (update for merging PR 41) #44

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
8 commits
Select commit Hold shift + click to select a range
03c1d59
Added support for Dremio.
scottkwalker Aug 28, 2025
51ba35a
Removed unnecessary func.
scottkwalker Sep 26, 2025
f63e25a
Add function nan_to_none
scottkwalker Sep 26, 2025
ec500c5
move import to top
scottkwalker Sep 26, 2025
aa1eec9
test correct timezone
scottkwalker Sep 26, 2025
a5656eb
test correct timezone with precision 3
scottkwalker Sep 26, 2025
2e8f5f5
replace set_timezone_to_utc with a pass
scottkwalker Sep 29, 2025
6c4b0ee
Update poetry.lock; Keep DuckDB < 1.2
erezsh Oct 23, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
18 changes: 11 additions & 7 deletions .github/workflows/ci.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,11 +1,12 @@
name: CI-COVER-VERSIONS

on:
# push:
# paths:
# - '**.py'
# - '.github/workflows/**'
# - '!dev/**'
# push:
# paths:
# - '**.py'
# - '.github/workflows/**'
# - '!dev/**'
# - '**.lock'
pull_request:
branches: [ master ]
workflow_dispatch:
Expand Down Expand Up @@ -34,7 +35,8 @@ jobs:
python-version: ${{ matrix.python-version }}

- name: Build the stack
run: docker compose up -d mysql postgres trino clickhouse vertica
# The vertica docker image is 404
run: docker compose up -d mysql postgres trino clickhouse dremio #vertica

- name: Install Poetry
run: pip install poetry
Expand All @@ -47,9 +49,11 @@ jobs:
# PRESTO_URI: 'presto://[email protected]/postgresql/public'
TRINO_URI: 'trino://[email protected]:8081/postgresql/public'
CLICKHOUSE_URI: 'clickhouse://clickhouse:Password1@localhost:9000/clickhouse'
VERTICA_URI: 'vertica://vertica:Password1@localhost:5433/vertica'
# The vertica docker image is 404
#VERTICA_URI: 'vertica://vertica:Password1@localhost:5433/vertica'
SNOWFLAKE_URI: '${{ secrets.SNOWFLAKE_URI }}'
REDSHIFT_URI: '${{ secrets.REDSHIFT_URI }}'
DREMIO_URI: 'dremio://dremio:dremio123@localhost:32010/$scratch?UseEncryption=false'
run: |
chmod +x tests/waiting_for_stack_up.sh
./tests/waiting_for_stack_up.sh && TEST_ACROSS_ALL_DBS=0 poetry run unittest-parallel -j 16
10 changes: 7 additions & 3 deletions .github/workflows/ci_full.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,11 +1,12 @@
name: CI-COVER-DATABASES

on:
# push:
# push:
# paths:
# - '**.py'
# - '.github/workflows/**'
# - '!dev/**'
# - '**.lock'
pull_request:
branches: [ master ]
workflow_dispatch:
Expand All @@ -30,7 +31,8 @@ jobs:
python-version: ${{ matrix.python-version }}

- name: Build the stack
run: docker compose up -d mysql postgres trino clickhouse vertica
# The vertica docker image is 404
run: docker compose up -d mysql postgres trino clickhouse dremio #vertica

- name: Install Poetry
run: pip install poetry
Expand All @@ -43,9 +45,11 @@ jobs:
# PRESTO_URI: 'presto://[email protected]/postgresql/public'
TRINO_URI: 'trino://[email protected]:8081/postgresql/public'
CLICKHOUSE_URI: 'clickhouse://clickhouse:Password1@localhost:9000/clickhouse'
VERTICA_URI: 'vertica://vertica:Password1@localhost:5433/vertica'
# The vertica docker image is 404
#VERTICA_URI: 'vertica://vertica:Password1@localhost:5433/vertica'
SNOWFLAKE_URI: '${{ secrets.SNOWFLAKE_URI }}'
REDSHIFT_URI: '${{ secrets.REDSHIFT_URI }}'
DREMIO_URI: 'dremio://dremio:dremio123@localhost:32010/$scratch?UseEncryption=false'
run: |
chmod +x tests/waiting_for_stack_up.sh
./tests/waiting_for_stack_up.sh && poetry run unittest-parallel -j 16
1 change: 1 addition & 0 deletions README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -42,6 +42,7 @@ _Databases we fully support_:
- Vertica
- DuckDB >=0.6
- SQLite (coming soon)
- Dremio

💻 **Built-in SQL client**: Connect to any of the supported databases with just one line.

Expand Down
59 changes: 40 additions & 19 deletions docker-compose.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -99,31 +99,52 @@ services:
networks:
- local

vertica:
container_name: dd-vertica
image: vertica/vertica-ce:12.0.0-0
restart: always
volumes:
- vertica-data:/data:delegated
ports:
- '5433:5433'
- '5444:5444'
expose:
- '5433'
- '5444'
env_file:
- dev/dev.env
tty: true
networks:
- local

# The vertica docker image is 404
# vertica:
# container_name: dd-vertica
# image: vertica/vertica-ce:12.0.0-0
# restart: always
# volumes:
# - vertica-data:/data:delegated
# ports:
# - '5433:5433'
# - '5444:5444'
# expose:
# - '5433'
# - '5444'
# env_file:
# - dev/dev.env
# tty: true
# networks:
# - local

dremio:
container_name: dd-dremio
image: dremio/dremio-oss:latest
restart: always
ports:
- '9047:9047'
- '31010:31010'
- '32010:32010'
expose:
- '9047'
- '31010'
- '32010'
networks:
- local
tty: true
env_file:
- dev/dev.env
volumes:
- dremio-data:/var/lib/dremio:delegated

volumes:
postgresql-data:
mysql-data:
clickhouse-data:
vertica-data:
# The vertica docker image is 404
#vertica-data:
dremio-data:

networks:
local:
Expand Down
2 changes: 2 additions & 0 deletions docs/install.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -26,6 +26,8 @@ To ensure that the database drivers are compatible with sqeleton, we recommend i

- `pip install 'sqeleton[vertica]'`

- `pip install 'sqeleton[dremio]'`

- For BigQuery, see: https://pypi.org/project/google-cloud-bigquery/

_Some drivers have dependencies that cannot be installed using `pip` and still need to be installed manually._
Expand Down
41 changes: 21 additions & 20 deletions docs/supported-databases.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,25 +1,26 @@
# List of supported databases

| Database | Status | Connection string |
|---------------|-------------------------------------------------------------------------------------------------------------------------------------|--------|
| PostgreSQL >=10 | 💚 | `postgresql://<user>:<password>@<host>:5432/<database>` |
| MySQL | 💚 | `mysql://<user>:<password>@<hostname>:5432/<database>` |
| Snowflake | 💚 | `"snowflake://<user>[:<password>]@<account>/<database>/<SCHEMA>?warehouse=<WAREHOUSE>&role=<role>[&authenticator=externalbrowser]"` |
| BigQuery | 💚 | `bigquery://<project>/<dataset>` |
| Redshift | 💚 | `redshift://<username>:<password>@<hostname>:5439/<database>` |
| Oracle | 💛 | `oracle://<username>:<password>@<hostname>/database` |
| Presto | 💛 | `presto://<username>:<password>@<hostname>:8080/<database>` |
| Databricks | 💛 | `databricks://<http_path>:<access_token>@<server_hostname>/<catalog>/<schema>` |
| Trino | 💛 | `trino://<username>:<password>@<hostname>:8080/<database>` |
| Clickhouse | 💛 | `clickhouse://<username>:<password>@<hostname>:9000/<database>` |
| Vertica | 💛 | `vertica://<username>:<password>@<hostname>:5433/<database>` |
| DuckDB | 💛 | |
| ElasticSearch | 📝 | |
| Planetscale | 📝 | |
| Pinot | 📝 | |
| Druid | 📝 | |
| Kafka | 📝 | |
| SQLite | 📝 | |
| Database | Status | Connection string |
|-----------------|-------------------------------------------------------------------------------------------------------------------------------------|---------------------------------------------------------------------------------------------------------------------------------------------------------------------|
| PostgreSQL >=10 | 💚 | `postgresql://<user>:<password>@<host>:5432/<database>` |
| MySQL | 💚 | `mysql://<user>:<password>@<hostname>:5432/<database>` |
| Snowflake | 💚 | `"snowflake://<user>[:<password>]@<account>/<database>/<SCHEMA>?warehouse=<WAREHOUSE>&role=<role>[&authenticator=externalbrowser]"` |
| BigQuery | 💚 | `bigquery://<project>/<dataset>` |
| Redshift | 💚 | `redshift://<username>:<password>@<hostname>:5439/<database>` |
| Oracle | 💛 | `oracle://<username>:<password>@<hostname>/database` |
| Presto | 💛 | `presto://<username>:<password>@<hostname>:8080/<database>` |
| Databricks | 💛 | `databricks://<http_path>:<access_token>@<server_hostname>/<catalog>/<schema>` |
| Trino | 💛 | `trino://<username>:<password>@<hostname>:8080/<database>` |
| Clickhouse | 💛 | `clickhouse://<username>:<password>@<hostname>:9000/<database>` |
| Vertica | 💛 | `vertica://<username>:<password>@<hostname>:5433/<database>` |
| DuckDB | 💛 | |
| Dremio | 💛 | `dremio://<user>[:<password>]@<hostname>:32010/<space>?Token=<Token>&UseEncryption=<UseEncryption>&DisableCertificateVerification=<DisableCertificateVerification>` |
| ElasticSearch | 📝 | |
| Planetscale | 📝 | |
| Pinot | 📝 | |
| Druid | 📝 | |
| Kafka | 📝 | |
| SQLite | 📝 | |

* 💚: Implemented and thoroughly tested.
* 💛: Implemented, but not thoroughly tested yet.
Expand Down
Loading