Skip to content

SEA: e2e testing #639

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Draft
wants to merge 63 commits into
base: main
Choose a base branch
from
Draft
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
63 commits
Select commit Hold shift + click to select a range
5bf5d4c
Separate Session related functionality from Connection class (#571)
varun-edachali-dbx May 28, 2025
400a8bd
Introduce Backend Interface (DatabricksClient) (#573)
varun-edachali-dbx May 30, 2025
3c78ed7
Implement ResultSet Abstraction (backend interfaces for fetch phase) …
varun-edachali-dbx Jun 3, 2025
9625229
Introduce Sea HTTP Client and test script (#583)
varun-edachali-dbx Jun 4, 2025
0887bc1
Introduce `SeaDatabricksClient` (Session Implementation) (#582)
varun-edachali-dbx Jun 9, 2025
6d63df0
Normalise Execution Response (clean backend interfaces) (#587)
varun-edachali-dbx Jun 11, 2025
ba8d9fd
Introduce models for `SeaDatabricksClient` (#595)
varun-edachali-dbx Jun 12, 2025
bb3f15a
Introduce preliminary SEA Result Set (#588)
varun-edachali-dbx Jun 12, 2025
19f1fae
Merge branch 'main' into sea-migration
varun-edachali-dbx Jun 17, 2025
6c5ba6d
remove invalid ExecuteResponse import
varun-edachali-dbx Jun 17, 2025
5e5147b
Separate Session related functionality from Connection class (#571)
varun-edachali-dbx May 28, 2025
57370b3
Introduce Backend Interface (DatabricksClient) (#573)
varun-edachali-dbx May 30, 2025
75752bf
Implement ResultSet Abstraction (backend interfaces for fetch phase) …
varun-edachali-dbx Jun 3, 2025
450b80d
remove un-necessary initialisation assertions
varun-edachali-dbx Jun 18, 2025
a926f02
remove un-necessary line break s
varun-edachali-dbx Jun 18, 2025
55ad001
more un-necessary line breaks
varun-edachali-dbx Jun 18, 2025
fa15730
constrain diff of test_closing_connection_closes_commands
varun-edachali-dbx Jun 18, 2025
019c7fb
reduce diff of test_closing_connection_closes_commands
varun-edachali-dbx Jun 18, 2025
726abe7
use pytest-like assertions for test_closing_connection_closes_commands
varun-edachali-dbx Jun 18, 2025
bf6d41c
ensure command_id is not None
varun-edachali-dbx Jun 18, 2025
5afa733
line breaks after multi-line pyfocs
varun-edachali-dbx Jun 18, 2025
e3dfd36
ensure non null operationHandle for commandId creation
varun-edachali-dbx Jun 18, 2025
63360b3
use command_id methods instead of explicit guid_to_hex_id conversion
varun-edachali-dbx Jun 18, 2025
13ffb8d
remove un-necessary artifacts in test_session, add back assertion
varun-edachali-dbx Jun 18, 2025
a74d279
Implement SeaDatabricksClient (Complete Execution Spec) (#590)
varun-edachali-dbx Jun 18, 2025
d759050
add from __future__ import annotations to remove string literals arou…
varun-edachali-dbx Jun 19, 2025
1e21434
move docstring of DatabricksClient within class
varun-edachali-dbx Jun 24, 2025
cd4015b
move ThriftResultSet import to top of file
varun-edachali-dbx Jun 24, 2025
ed8b610
make backend/utils __init__ file empty
varun-edachali-dbx Jun 24, 2025
94d951e
use from __future__ import annotations to remove string literals arou…
varun-edachali-dbx Jun 24, 2025
c20058e
use lazy logging
varun-edachali-dbx Jun 24, 2025
fe3acb1
replace getters with property tag
varun-edachali-dbx Jun 24, 2025
9fb6a76
Merge branch 'main' into backend-refactors
varun-edachali-dbx Jun 24, 2025
61dfc4d
set active_command_id to None, not active_op_handle
varun-edachali-dbx Jun 24, 2025
64fb9b2
align test_session with pytest instead of unittest
varun-edachali-dbx Jun 24, 2025
cbf63f9
Merge branch 'main' into sea-migration
varun-edachali-dbx Jun 26, 2025
59b4825
remove duplicate test, correct active_command_id attribute
varun-edachali-dbx Jun 26, 2025
e380654
SeaDatabricksClient: Add Metadata Commands (#593)
varun-edachali-dbx Jun 26, 2025
677a7b0
SEA volume operations fix: assign `manifest.is_volume_operation` to `…
varun-edachali-dbx Jun 26, 2025
45585d4
Introduce manual SEA test scripts for Exec Phase (#589)
varun-edachali-dbx Jun 27, 2025
70c7dc8
Complete Fetch Phase (for `INLINE` disposition and `JSON_ARRAY` forma…
varun-edachali-dbx Jul 2, 2025
abf9aab
Merge branch 'main' into sea-migration
varun-edachali-dbx Jul 3, 2025
9b4b606
Merge branch 'main' into backend-refactors
varun-edachali-dbx Jul 3, 2025
4f11ff0
Introduce `row_limit` param (#607)
varun-edachali-dbx Jul 7, 2025
45f5c26
Merge branch 'main' into backend-refactors
varun-edachali-dbx Jul 10, 2025
2c9368a
formatting (black)
varun-edachali-dbx Jul 10, 2025
9b1b1f5
remove repetition from Session.__init__
varun-edachali-dbx Jul 10, 2025
77e23d3
Merge branch 'backend-refactors' into sea-migration
varun-edachali-dbx Jul 11, 2025
3bd3aef
fix merge artifacts
varun-edachali-dbx Jul 11, 2025
6d4701f
correct patch paths
varun-edachali-dbx Jul 11, 2025
dc1cb6d
fix type issues
varun-edachali-dbx Jul 14, 2025
5d04cd0
Merge branch 'main' into sea-migration
varun-edachali-dbx Jul 15, 2025
922c448
explicitly close result queue
varun-edachali-dbx Jul 15, 2025
7d10756
add more sea e2e testing
varun-edachali-dbx Jul 16, 2025
4e6ee34
test_get_catalogs for SEA
varun-edachali-dbx Jul 16, 2025
6aa2991
add missing pytest import
varun-edachali-dbx Jul 16, 2025
ea4beaf
test_get_tables
varun-edachali-dbx Jul 16, 2025
1a0575a
Complete Fetch Phase (`EXTERNAL_LINKS` disposition and `ARROW` format…
varun-edachali-dbx Jul 16, 2025
c07beb1
SEA Session Configuration Fix: Explicitly convert values to `str` (#…
varun-edachali-dbx Jul 16, 2025
640cc82
SEA: add support for `Hybrid` disposition (#631)
varun-edachali-dbx Jul 17, 2025
8fbca9d
SEA: Reduce network calls for synchronous commands (#633)
varun-edachali-dbx Jul 19, 2025
d9f59a8
Merge branch 'sea-migration' into sea-e2e-tests
varun-edachali-dbx Jul 19, 2025
3ae7d04
Merge branch 'main' into sea-e2e-tests
varun-edachali-dbx Jul 31, 2025
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 3 additions & 0 deletions src/databricks/sql/types.py
Original file line number Diff line number Diff line change
Expand Up @@ -187,6 +187,7 @@ def __contains__(self, item: Any) -> bool:
# let object acts like class
def __call__(self, *args: Any) -> "Row":
"""create new Row object"""

if len(args) > len(self):
raise ValueError(
"Can not create Row with fields %s, expected %d values "
Expand Down Expand Up @@ -229,13 +230,15 @@ def __reduce__(
self,
) -> Union[str, Tuple[Any, ...]]:
"""Returns a tuple so Python knows how to pickle Row."""

if hasattr(self, "__fields__"):
return (_create_row, (self.__fields__, tuple(self)))
else:
return tuple.__reduce__(self)

def __repr__(self) -> str:
"""Printable representation of Row used in Python REPL."""

if hasattr(self, "__fields__"):
return "Row(%s)" % ", ".join(
"%s=%r" % (k, v) for k, v in zip(self.__fields__, tuple(self))
Expand Down
1 change: 0 additions & 1 deletion src/databricks/sql/utils.py
Original file line number Diff line number Diff line change
Expand Up @@ -740,7 +740,6 @@ def convert_decimals_in_arrow_table(table, description) -> "pyarrow.Table":


def convert_to_assigned_datatypes_in_column_table(column_table, description):

converted_column_table = []
for i, col in enumerate(column_table):
if description[i][1] == "decimal":
Expand Down
28 changes: 24 additions & 4 deletions tests/e2e/common/decimal_tests.py
Original file line number Diff line number Diff line change
Expand Up @@ -38,22 +38,42 @@ class DecimalTestsMixin:
),
]

@pytest.mark.parametrize(
"backend_params",
[
{},
{
"use_sea": True,
},
],
)
@pytest.mark.parametrize(
"decimal, expected_value, expected_type", decimal_and_expected_results
)
def test_decimals(self, decimal, expected_value, expected_type):
with self.cursor({}) as cursor:
def test_decimals(self, decimal, expected_value, expected_type, backend_params):
with self.cursor(backend_params) as cursor:
query = "SELECT CAST ({})".format(decimal)
cursor.execute(query)
table = cursor.fetchmany_arrow(1)
assert table.field(0).type == expected_type
assert table.to_pydict().popitem()[1][0] == expected_value

@pytest.mark.parametrize(
"backend_params",
[
{},
{
"use_sea": True,
},
],
)
@pytest.mark.parametrize(
"decimals, expected_values, expected_type", multi_decimals_and_expected_results
)
def test_multi_decimals(self, decimals, expected_values, expected_type):
with self.cursor({}) as cursor:
def test_multi_decimals(
self, decimals, expected_values, expected_type, backend_params
):
with self.cursor(backend_params) as cursor:
union_str = " UNION ".join(
["(SELECT CAST ({}))".format(dec) for dec in decimals]
)
Expand Down
61 changes: 53 additions & 8 deletions tests/e2e/common/retry_test_mixins.py
Original file line number Diff line number Diff line change
Expand Up @@ -17,17 +17,37 @@


class Client429ResponseMixin:
def test_client_should_retry_automatically_when_getting_429(self):
with self.cursor() as cursor:
@pytest.mark.parametrize(
"backend_params",
[
{},
{
"use_sea": True,
},
],
)
def test_client_should_retry_automatically_when_getting_429(self, backend_params):
with self.cursor(backend_params) as cursor:
for _ in range(10):
cursor.execute("SELECT 1")
rows = cursor.fetchall()
self.assertEqual(len(rows), 1)
self.assertEqual(rows[0][0], 1)

def test_client_should_not_retry_429_if_RateLimitRetry_is_0(self):
@pytest.mark.parametrize(
"backend_params",
[
{},
{
"use_sea": True,
},
],
)
def test_client_should_not_retry_429_if_RateLimitRetry_is_0(self, backend_params):
with pytest.raises(self.error_type) as cm:
with self.cursor(self.conf_to_disable_rate_limit_retries) as cursor:
with self.cursor(
{**self.conf_to_disable_rate_limit_retries, **backend_params}
) as cursor:
for _ in range(10):
cursor.execute("SELECT 1")
rows = cursor.fetchall()
Expand All @@ -46,14 +66,39 @@ def test_client_should_not_retry_429_if_RateLimitRetry_is_0(self):


class Client503ResponseMixin:
def test_wait_cluster_startup(self):
with self.cursor() as cursor:
@pytest.mark.parametrize(
"backend_params",
[
{},
{
"use_sea": True,
},
],
)
def test_wait_cluster_startup(self, backend_params):
with self.cursor(backend_params) as cursor:
cursor.execute("SELECT 1")
cursor.fetchall()

def _test_retry_disabled_with_message(self, error_msg_substring, exception_type):
@pytest.mark.parametrize(
"backend_params",
[
{},
{
"use_sea": True,
},
],
)
def _test_retry_disabled_with_message(
self, error_msg_substring, exception_type, backend_params
):
with pytest.raises(exception_type) as cm:
with self.connection(self.conf_to_disable_temporarily_unavailable_retries):
with self.connection(
{
**self.conf_to_disable_temporarily_unavailable_retries,
**backend_params,
}
):
pass
assert error_msg_substring in str(cm.exception)

Expand Down
Loading
Loading