Making duckdb-wasm bundle CORS requests

[llimllib]

I'm trying to get duckdb-wasm to load its files from my CDN (a Digital Ocean Spaces CDN), and failing. Here is a regular fetch request for https://cdn.billmill.org/nbastats/dist/duckdb/duckdb-browser.worker.js and the request from duckdb-wasm, side by side:

This shows that CORS is enabled for the file, but it's still not working with duckdb-wasm.

I think that the issue may be that the request from duckdb-wasm is not sending the Origin header along with its request. When I copy as curl from the console there, here's what I get (lightly edited for console display):

$ curl -v 'https://cdn.billmill.org/nbastats/dist/duckdb/duckdb-browser.worker.js' -H 'User-Agent: Mozilla
/5.0 (Macintosh; Intel Mac OS X 10.15; rv:94.0) Gecko/20100101 Firefox/94.0' -H 'Accept: */*' -H 'Accept-L
anguage: en-US,en;q=0.5' --compressed -H 'Referer: https://devd.io:8000/' -o /dev/null
<snip ssl stuff>
> GET /nbastats/dist/duckdb/duckdb-browser.worker.js HTTP/1.1
> Host: cdn.billmill.org
> Accept-Encoding: deflate, gzip
> User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:94.0) Gecko/20100101 Firefox/94.0
> Accept: */*
> Accept-Language: en-US,en;q=0.5
> Referer: https://devd.io:8000/
>
* Mark bundle as not supporting multiuse
< HTTP/1.1 200 OK
< Date: Tue, 23 Nov 2021 14:45:35 GMT
< Connection: Keep-Alive
< Cache-Control: max-age=2331
< Content-Length: 396184
< Content-Type: application/javascript
< Last-Modified: Mon, 22 Nov 2021 21:01:30 GMT
< Accept-Ranges: bytes
< x-rgw-object-type: Normal
< etag: "53da1599b4a084e7c7c5d9c248332fda"
< x-amz-meta-s3cmd-attrs: atime:1637594740/ctime:1637594737/gid:20/gname:staff/md5:53da1599b4a084e7c7c5d9c
248332fda/mode:33188/mtime:1637594737/uid:501/uname:llimllib
< x-amz-request-id: tx0000000000000db523614-00619cf99a-1800930a-nyc3c
< strict-transport-security: max-age=15552000; includeSubDomains; preload
< Vary: Access-Control-Request-Headers,Access-Control-Request-Method,Origin
< X-HW: 1637678735.dop061.ny3.t,1637678735.cds224.ny3.shn,1637678735.dop061.ny3.t,1637678735.cds152.ny3.c
<
{ [16384 bytes data]

You can see that the response does not include an access-control-allow-origin header. If I modify the request to use Origin instead of Referer, the server does respond with that header:

$ curl -v 'https://cdn.billmill.org/nbastats/dist/duckdb/duckdb-browser.worker.js' -H 'User-Agent: Mozilla
/5.0 (Macintosh; Intel Mac OS X 10.15; rv:94.0) Gecko/20100101 Firefox/94.0' -H 'Accept: */*' -H 'Accept-L
anguage: en-US,en;q=0.5' --compressed -H 'Origin: http://devd.io:8000' -o /dev/null
<snip ssl stuff>
> GET /nbastats/dist/duckdb/duckdb-browser.worker.js HTTP/1.1
> Host: cdn.billmill.org
> Accept-Encoding: deflate, gzip
> User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:94.0) Gecko/20100101 Firefox/94.0
> Accept: */*
> Accept-Language: en-US,en;q=0.5
> Origin: http://devd.io:8000
>
* Mark bundle as not supporting multiuse
< HTTP/1.1 200 OK
< Date: Tue, 23 Nov 2021 14:45:54 GMT
< Connection: Keep-Alive
< Accept-Ranges: bytes
< Cache-Control: max-age=3600
< Content-Length: 396184
< Content-Type: application/javascript
< X-HW: 1637678753.dop145.ny3.t,1637678753.cds076.ny3.shn,1637678753.dop145.ny3.t,1637678754.cds034.ny3.p
< x-rgw-object-type: Normal
< etag: "53da1599b4a084e7c7c5d9c248332fda"
< x-amz-meta-s3cmd-attrs: atime:1637594740/ctime:1637594737/gid:20/gname:staff/md5:53da1599b4a084e7c7c5d9c
248332fda/mode:33188/mtime:1637594737/uid:501/uname:llimllib
< x-amz-request-id: tx0000000000000db5d5f76-00619cfea2-1800930a-nyc3c
< access-control-allow-origin: *
< access-control-allow-methods: GET
< access-control-max-age: 0
< strict-transport-security: max-age=15552000; includeSubDomains; preload
< Last-Modified: Mon, 22 Nov 2021 21:01:30 GMT
<
{ [16384 bytes data]

Given that I can't modify the server here, is there anything that can be done on the duckdb side to make the request with an Origin header? I don't know if wasm has any constraints that javascript does not - I tried reading some things online but it wasn't very clear to me at all.

Any other ideas for things I can try are welcome.

[ankoh]

The dilemma here is likely, that the creation of cross-origin workers requires some workaround.
(More information here: https://stackoverflow.com/questions/21913673/execute-web-worker-from-different-origin)

I added a tiny helper for that purpose in @duckdb/[email protected].

import * as duckdb from "@duckdb/duckdb-wasm"
const workerScript = "https://cdn.billmill.org/nbastats/dist/duckdb/duckdb-browser.worker.js";
const worker = await duckdb.createWorker(workerScript);

I would, however, recommend to use the jsdelivr bundles since we have multiple workers to get most out of different client browsers.

const allBundles = duckdb.getJsDelivrBundles();
const bestBundle = await duckdb.selectBundle(allBundles);
const worker = await duckdb.createWorker(bestBundle.mainWorker);