Skip to content

Commit

Permalink
remove comments on duckdb settings for multimodal in FilterTransform.…
Browse files Browse the repository at this point in the history
…init().

Signed-off-by: David Wood <[email protected]>
  • Loading branch information
daw3rd committed Dec 2, 2024
1 parent 6123a6e commit 0eb6a29
Showing 1 changed file with 0 additions and 4 deletions.
4 changes: 0 additions & 4 deletions transforms/universal/filter/python/src/filter_transform.py
Original file line number Diff line number Diff line change
Expand Up @@ -67,10 +67,6 @@ def __init__(self, config: dict):
self.logical_operator = config.get(filter_logical_operator_key, filter_logical_operator_default)
self.columns_to_drop = config.get(filter_columns_to_drop_key, filter_columns_to_drop_default)

# Temporarily here to test if this can allow use to process files that are required to be read by polars for mm
# If this works, we should add as a configurable or always enable (not sure of the downside of enabling this).
# duckdb.execute("SET arrow_large_buffer_size = true")

def transform(self, table: pa.Table, file_name: str = None) -> tuple[list[pa.Table], dict]:
"""
This implementation filters the input table using a SQL statement and
Expand Down

0 comments on commit 0eb6a29

Please sign in to comment.