Skip to content

Commit 5628293

Browse files
vustefgbrgr
andauthored
FFI and Julia Bindings for incremental scan changes (#6)
FFI and Julia Bindings for incremental scan changes that were introduced with RelationalAI/iceberg-rust#3. I refactored the file structures a bit, for both Rust and Julia code. Rust code reuses some common parts through macros, for Julia I didn't bother to do that (mostly afraid of macros and ccall interaction being a rabbit hole with little benefit). Note that I had to use struct instead of const for `ScanRef`, since now with additional type, we have method overloads, which if we use `ScanRef` const aliases actually use same type, and then become overwrites instead of overloads. There's also a new test data, and new test that exercises positional delete and inserts. --------- Co-authored-by: Gerald Berger <[email protected]>
1 parent 4926a9d commit 5628293

File tree

64 files changed

+2185
-1444
lines changed

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

64 files changed

+2185
-1444
lines changed

.github/workflows/test.yml

Lines changed: 4 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -61,10 +61,13 @@ jobs:
6161
-e MINIO_ROOT_USER=root \
6262
-e MINIO_ROOT_PASSWORD=password \
6363
-e MINIO_DOMAIN=minio \
64-
-v ${{ github.workspace }}/assets/tpch:/input minio/minio:latest server /data/
64+
-v ${{ github.workspace }}/assets/tpch:/input \
65+
-v ${{ github.workspace }}/assets/incremental:/input_incremental \
66+
minio/minio:latest server /data/
6567
until (docker exec minio mc alias set minio http://localhost:9000 root password) do echo '... waiting ...' && sleep 1; done;
6668
docker exec minio mc mb minio/warehouse
6769
docker exec minio mc cp -r /input/tpch.sf01/ minio/warehouse/tpch.sf01/
70+
docker exec minio mc cp -r /input_incremental/ minio/warehouse/incremental/;
6871
docker run -d --name rest \
6972
--network=iceberg_net \
7073
-p 8181:8181 \

.gitignore

Lines changed: 3 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1,5 +1,7 @@
11
**/.env
22
iceberg_rust_ffi/target
3+
iceberg_rust_ffi/target/**/*
34
iceberg_rust_ffi/integration_test
45
**/*.dylib
5-
**/.claude
6+
**/.claude
7+
**/.DS_Store

CLAUDE.md

Lines changed: 0 additions & 5 deletions

Makefile

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -48,7 +48,7 @@ test: build
4848
exit 1; \
4949
fi
5050
@set -a && . ./.env && set +a && \
51-
export ICEBERG_RUST_LIB=$(TARGET_DIR) && \
51+
export ICEBERG_RUST_LIB=$$(pwd)/$(TARGET_DIR) && \
5252
$(JULIA_THREADS_ENV) julia --project=. -e 'using Pkg; Pkg.test()'
5353

5454
# Start Julia REPL with environment configured (requires .env file)
@@ -58,7 +58,7 @@ repl: build
5858
exit 1; \
5959
fi
6060
@set -a && . ./.env && set +a && \
61-
export ICEBERG_RUST_LIB=$(TARGET_DIR) && \
61+
export ICEBERG_RUST_LIB=$$(pwd)/$(TARGET_DIR) && \
6262
$(JULIA_THREADS_ENV) julia --project=.
6363

6464
# Clean build artifacts
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.

0 commit comments

Comments
 (0)