[cigrainger]

Summary

• Dux.insert_into/3 — new verb for writing pipeline results into any table DuckDB can write to (local, attached Postgres, DuckLake, etc.). Supports create: true for CREATE TABLE AS semantics.
• Distributed DuckLake reads — the coordinator now resolves DuckLake attached sources by querying ducklake_list_files() for the backing Parquet file manifest, then distributes those files across workers. Workers read Parquet directly — no DuckLake extension needed on worker nodes.

insert_into examples

# Write to an attached DuckLake table
Dux.attach(:lake, "ducklake:metadata.db", type: :ducklake, read_only: false)

Dux.from_parquet("s3://bucket/raw/*.parquet")
|> Dux.filter(col("status") == "active")
|> Dux.insert_into("lake.analytics.users")

# Create a new table from a pipeline
Dux.from_csv("/data/events.csv")
|> Dux.insert_into("lake.events", create: true)

Distributed DuckLake read flow

1. Coordinator detects {:attached, ...} source is DuckLake (via pragma_database_list)
2. Queries ducklake_list_files() to get backing Parquet paths
3. Distributes file list across workers via Partitioner
4. Workers scan Parquet directly — standard read_parquet(), no DuckLake attach needed

Test plan

• insert_into with create: true creates table from pipeline
• insert_into appends to existing table
• insert_into works with attached DuckDB database
• Partitioner distributes DuckLake file list across workers as parquet sources
• Single-file DuckLake source assigned to one worker
• All existing tests pass (566 tests, 0 failures)
• Credo clean

🤖 Generated with Claude Code