Skip to content

Commit 5f57083

Browse files
erezsherezsh
authored
Merge pull request #44 from erezsh/dremio
Add support for Dremio (update for merging PR 41)
2 parents 164f62d + 6c4b0ee commit 5f57083

14 files changed

Lines changed: 2711 additions & 239 deletions

File tree

‎.github/workflows/ci.yml‎

Lines changed: 11 additions & 7 deletions
Original file line numberDiff line numberDiff line change
@@ -1,11 +1,12 @@
11
name: CI-COVER-VERSIONS
22

33
on:
4-
# push:
5-
# paths:
6-
# - '**.py'
7-
# - '.github/workflows/**'
8-
# - '!dev/**'
4+
# push:
5+
# paths:
6+
# - '**.py'
7+
# - '.github/workflows/**'
8+
# - '!dev/**'
9+
# - '**.lock'
910
pull_request:
1011
branches: [ master ]
1112
workflow_dispatch:
@@ -34,7 +35,8 @@ jobs:
3435
python-version: ${{ matrix.python-version }}
3536

3637
- name: Build the stack
37-
run: docker compose up -d mysql postgres trino clickhouse vertica
38+
# The vertica docker image is 404
39+
run: docker compose up -d mysql postgres trino clickhouse dremio #vertica
3840

3941
- name: Install Poetry
4042
run: pip install poetry
@@ -47,9 +49,11 @@ jobs:
4749
# PRESTO_URI: 'presto://presto@127.0.0.1/postgresql/public'
4850
TRINO_URI: 'trino://postgres@127.0.0.1:8081/postgresql/public'
4951
CLICKHOUSE_URI: 'clickhouse://clickhouse:Password1@localhost:9000/clickhouse'
50-
VERTICA_URI: 'vertica://vertica:Password1@localhost:5433/vertica'
52+
# The vertica docker image is 404
53+
#VERTICA_URI: 'vertica://vertica:Password1@localhost:5433/vertica'
5154
SNOWFLAKE_URI: '${{ secrets.SNOWFLAKE_URI }}'
5255
REDSHIFT_URI: '${{ secrets.REDSHIFT_URI }}'
56+
DREMIO_URI: 'dremio://dremio:dremio123@localhost:32010/$scratch?UseEncryption=false'
5357
run: |
5458
chmod +x tests/waiting_for_stack_up.sh
5559
./tests/waiting_for_stack_up.sh && TEST_ACROSS_ALL_DBS=0 poetry run unittest-parallel -j 16

‎.github/workflows/ci_full.yml‎

Lines changed: 7 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -1,11 +1,12 @@
11
name: CI-COVER-DATABASES
22

33
on:
4-
# push:
4+
# push:
55
# paths:
66
# - '**.py'
77
# - '.github/workflows/**'
88
# - '!dev/**'
9+
# - '**.lock'
910
pull_request:
1011
branches: [ master ]
1112
workflow_dispatch:
@@ -30,7 +31,8 @@ jobs:
3031
python-version: ${{ matrix.python-version }}
3132

3233
- name: Build the stack
33-
run: docker compose up -d mysql postgres trino clickhouse vertica
34+
# The vertica docker image is 404
35+
run: docker compose up -d mysql postgres trino clickhouse dremio #vertica
3436

3537
- name: Install Poetry
3638
run: pip install poetry
@@ -43,9 +45,11 @@ jobs:
4345
# PRESTO_URI: 'presto://presto@127.0.0.1/postgresql/public'
4446
TRINO_URI: 'trino://postgres@127.0.0.1:8081/postgresql/public'
4547
CLICKHOUSE_URI: 'clickhouse://clickhouse:Password1@localhost:9000/clickhouse'
46-
VERTICA_URI: 'vertica://vertica:Password1@localhost:5433/vertica'
48+
# The vertica docker image is 404
49+
#VERTICA_URI: 'vertica://vertica:Password1@localhost:5433/vertica'
4750
SNOWFLAKE_URI: '${{ secrets.SNOWFLAKE_URI }}'
4851
REDSHIFT_URI: '${{ secrets.REDSHIFT_URI }}'
52+
DREMIO_URI: 'dremio://dremio:dremio123@localhost:32010/$scratch?UseEncryption=false'
4953
run: |
5054
chmod +x tests/waiting_for_stack_up.sh
5155
./tests/waiting_for_stack_up.sh && poetry run unittest-parallel -j 16

‎README.md‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -42,6 +42,7 @@ _Databases we fully support_:
4242
- Vertica
4343
- DuckDB >=0.6
4444
- SQLite (coming soon)
45+
- Dremio
4546

4647
💻 **Built-in SQL client**: Connect to any of the supported databases with just one line.
4748

‎docker-compose.yml‎

Lines changed: 40 additions & 19 deletions
Original file line numberDiff line numberDiff line change
@@ -99,31 +99,52 @@ services:
9999
networks:
100100
- local
101101

102-
vertica:
103-
container_name: dd-vertica
104-
image: vertica/vertica-ce:12.0.0-0
105-
restart: always
106-
volumes:
107-
- vertica-data:/data:delegated
108-
ports:
109-
- '5433:5433'
110-
- '5444:5444'
111-
expose:
112-
- '5433'
113-
- '5444'
114-
env_file:
115-
- dev/dev.env
116-
tty: true
117-
networks:
118-
- local
119-
102+
# The vertica docker image is 404
103+
# vertica:
104+
# container_name: dd-vertica
105+
# image: vertica/vertica-ce:12.0.0-0
106+
# restart: always
107+
# volumes:
108+
# - vertica-data:/data:delegated
109+
# ports:
110+
# - '5433:5433'
111+
# - '5444:5444'
112+
# expose:
113+
# - '5433'
114+
# - '5444'
115+
# env_file:
116+
# - dev/dev.env
117+
# tty: true
118+
# networks:
119+
# - local
120120

121+
dremio:
122+
container_name: dd-dremio
123+
image: dremio/dremio-oss:latest
124+
restart: always
125+
ports:
126+
- '9047:9047'
127+
- '31010:31010'
128+
- '32010:32010'
129+
expose:
130+
- '9047'
131+
- '31010'
132+
- '32010'
133+
networks:
134+
- local
135+
tty: true
136+
env_file:
137+
- dev/dev.env
138+
volumes:
139+
- dremio-data:/var/lib/dremio:delegated
121140

122141
volumes:
123142
postgresql-data:
124143
mysql-data:
125144
clickhouse-data:
126-
vertica-data:
145+
# The vertica docker image is 404
146+
#vertica-data:
147+
dremio-data:
127148

128149
networks:
129150
local:

‎docs/install.md‎

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -26,6 +26,8 @@ To ensure that the database drivers are compatible with sqeleton, we recommend i
2626

2727
- `pip install 'sqeleton[vertica]'`
2828

29+
- `pip install 'sqeleton[dremio]'`
30+
2931
- For BigQuery, see: https://pypi.org/project/google-cloud-bigquery/
3032

3133
_Some drivers have dependencies that cannot be installed using `pip` and still need to be installed manually._

‎docs/supported-databases.md‎

Lines changed: 21 additions & 20 deletions
Original file line numberDiff line numberDiff line change
@@ -1,25 +1,26 @@
11
# List of supported databases
22

3-
| Database | Status | Connection string |
4-
|---------------|-------------------------------------------------------------------------------------------------------------------------------------|--------|
5-
| PostgreSQL >=10 | 💚 | `postgresql://<user>:<password>@<host>:5432/<database>` |
6-
| MySQL | 💚 | `mysql://<user>:<password>@<hostname>:5432/<database>` |
7-
| Snowflake | 💚 | `"snowflake://<user>[:<password>]@<account>/<database>/<SCHEMA>?warehouse=<WAREHOUSE>&role=<role>[&authenticator=externalbrowser]"` |
8-
| BigQuery | 💚 | `bigquery://<project>/<dataset>` |
9-
| Redshift | 💚 | `redshift://<username>:<password>@<hostname>:5439/<database>` |
10-
| Oracle | 💛 | `oracle://<username>:<password>@<hostname>/database` |
11-
| Presto | 💛 | `presto://<username>:<password>@<hostname>:8080/<database>` |
12-
| Databricks | 💛 | `databricks://<http_path>:<access_token>@<server_hostname>/<catalog>/<schema>` |
13-
| Trino | 💛 | `trino://<username>:<password>@<hostname>:8080/<database>` |
14-
| Clickhouse | 💛 | `clickhouse://<username>:<password>@<hostname>:9000/<database>` |
15-
| Vertica | 💛 | `vertica://<username>:<password>@<hostname>:5433/<database>` |
16-
| DuckDB | 💛 | |
17-
| ElasticSearch | 📝 | |
18-
| Planetscale | 📝 | |
19-
| Pinot | 📝 | |
20-
| Druid | 📝 | |
21-
| Kafka | 📝 | |
22-
| SQLite | 📝 | |
3+
| Database | Status | Connection string |
4+
|-----------------|-------------------------------------------------------------------------------------------------------------------------------------|---------------------------------------------------------------------------------------------------------------------------------------------------------------------|
5+
| PostgreSQL >=10 | 💚 | `postgresql://<user>:<password>@<host>:5432/<database>` |
6+
| MySQL | 💚 | `mysql://<user>:<password>@<hostname>:5432/<database>` |
7+
| Snowflake | 💚 | `"snowflake://<user>[:<password>]@<account>/<database>/<SCHEMA>?warehouse=<WAREHOUSE>&role=<role>[&authenticator=externalbrowser]"` |
8+
| BigQuery | 💚 | `bigquery://<project>/<dataset>` |
9+
| Redshift | 💚 | `redshift://<username>:<password>@<hostname>:5439/<database>` |
10+
| Oracle | 💛 | `oracle://<username>:<password>@<hostname>/database` |
11+
| Presto | 💛 | `presto://<username>:<password>@<hostname>:8080/<database>` |
12+
| Databricks | 💛 | `databricks://<http_path>:<access_token>@<server_hostname>/<catalog>/<schema>` |
13+
| Trino | 💛 | `trino://<username>:<password>@<hostname>:8080/<database>` |
14+
| Clickhouse | 💛 | `clickhouse://<username>:<password>@<hostname>:9000/<database>` |
15+
| Vertica | 💛 | `vertica://<username>:<password>@<hostname>:5433/<database>` |
16+
| DuckDB | 💛 | |
17+
| Dremio | 💛 | `dremio://<user>[:<password>]@<hostname>:32010/<space>?Token=<Token>&UseEncryption=<UseEncryption>&DisableCertificateVerification=<DisableCertificateVerification>` |
18+
| ElasticSearch | 📝 | |
19+
| Planetscale | 📝 | |
20+
| Pinot | 📝 | |
21+
| Druid | 📝 | |
22+
| Kafka | 📝 | |
23+
| SQLite | 📝 | |
2324

2425
* 💚: Implemented and thoroughly tested.
2526
* 💛: Implemented, but not thoroughly tested yet.

0 commit comments

Comments
 (0)