Is Arrow Flight supported?

[fadmaa]

Hello,
I am considering my options for getting data remotely from a service and loading it into duckdb-wasm in the browser.
The remote service has the data in CSV and will convert it to a format more suitable for Duckdb Wasm... Arrow Flight would probably be ideal but as far as I can tell, it is not supported. Am I right?
Also, when I experimented with Arrow IPC, I found it extremely verbose (a 150MB CSV file resulted in about 2GB in Arrow IPC format)... is that something other people experienced or am I doing something wrong?

Thanks!

[sgilligan]

We send a lot of CSV results through to duckdb-wasm in the browser, and it is always pretty much a similar size or better (however we're not dealing with 150mb files). We wrote our own csv-to-arrow implementation which dictionary encodes and chooses the smallest datatype when possible (eg Int8 vs BigInt64).

[carlopi]

Arrow Flight is not supported.

@sgilligan & @fadmaa: I would give it a try here to duckdb's own storage format, basically having server side a duckdb instance (natively or via python / nodejs clients) parse the CSV and exporting a duckdb database, to then be send to duckdb-wasm and read natively.
With that you get all advantage of typing, duckdb's own compression AND on top you can compress with brotli / snappy / gzip.
Main drawback of duckdb storage format is that's not stable yet, but it's a matter of updating at the same time the duckdb-wasm version with the duckdb version.

Otherwise Parquet or Arrow should also be good to go (potentially also converted by DuckDB!)