[Inbox] Cool Feature Ideas

[rufuspollock]

• DuckDB WASM in the page if you want it https://duckdb.org/2021/10/29/duckdb-wasm.html - https://observablehq.com/@cmudig/introducing-sql-with-duckdb
• Data APIs datopian/datahub-next#93
• Datasette APIs as a service or embedded in the page

[davidgasquez]

Expanding some ideas/tools for the Data API layer.

RoAPI

This is probably the most straightforward way. Point RoAPI to your slowly changing dataset and have a bunch of API ready. Could be served via Cloud Run/Lambda to reduce costs.

Seafowl

With Seafowl, datasets could be packaged and deployed on the edge (CF Workers, Fly.io, Serverless platform, ...). They've figured out things like caching and CDNs. They have a great end to end example showcasing some interesting features.

Other Interesting Tools

DuckDB WASM

As you point out in the issue, it would be super easy to make a small wrapper around DuckDB that understood the Frictionless specs and allowed everyone to run things like:

select * from my.com/datapackage.json union all other.io/datapackage.yml

Trustfall

Trustfall is a query language for any combination of data sources. If there was a datapackage adapter. You could run queries in Trustfall against these and, at the same time, join with "live" datasets. E.g: Top S&P500 companies (static dataset) by volume traded today (via API).

Datasette Little

With each "project"/"readme", a static datasette little endpoint could be served alongside it. This allows people to run SQL (and many more things) in the datasets. E.g: Using the data from the Finance VIX dataset to run a SQL query.

Graphic Walker

Might be cool to plug a Graphic Walker. I've used it a few times (online demo) to quickly plot things and worked very well. It also generates Vega syntax! This tool is powering RATH.

[rufuspollock]

@davidgasquez great ideas. BTW i've just converted to a discussion (it obviously should be!)

I've especially thought about datasette for quite a few years - and it seems with wasm we could even run in the browser (i've also been wondering if one could somehow get it or something like it running in a cloudflare worker ...)

[davidgasquez]

It definitely works on the browser with WASM! I'd say it also works in CF Workers. Curious, when would that option be preferred?

[rufuspollock]

It definitely works on the browser with WASM! I'd say it also works in CF Workers. Curious, when would that option be preferred?

There are two distinct use cases here:

• Data exploration by a user: this can be datasette in browser pretty much (depending on size of data etc).
• A Data API so that a separate apps can use it (mine or someone else's): in this case i need an actual API running ongoingly and that's some datasette also provides.

For the latter option we need to run datasette somewhere and that requires e.g. cloudflare or google cloud run or whatever.

[davidgasquez]

I see! Makes sense. Thanks for clarifying! I think for the API side, I'd rely on other projects like RoAPI and Seafowl as they provide more dialects and are optimized for analytical queries¹.

¹ Not that will matter much given that probably around 95% of the datasets are less than 10MB. It could reduce a bit of costs though!

[rufuspollock]

@davidgasquez great points - i'm going to start a sub-thread here https://github.com/datopian/datahub-next/discussions/93 about Data API and perhaps you could chime in with your thoughts on RoAPI and Seafowl.