feat(RFC): Adds altair.datasets

[dangotbanned]

Related

• `vega_datasets` source #3150
• feat: allow for creating new_series / from_dict without specifying a backend narwhals-dev/narwhals#876 (comment)

Tracking

Waiting on the next vega-datasets release.
Once there is a stable datapackage.json available - there is quite a lot of tools/datasets that can be simplified/removed.

• 3.0.0 Release vega-datasets#654
• feat: Update for [email protected]

Discovered a bug that makes some handling of expressions a little less efficient.

• [Bug]: Missing handling for Iterator[IntoExpr] narwhals-dev/narwhals#1897

Upstreaming some nw.Schema stuff to narwhals

• [Enh]: nw.(DType|Schema) conversion API narwhals-dev/narwhals#1912
• feat(RFC): Adds altair.datasets #3631 (comment)

Improve user-facing interface

• feat(RFC): Adds altair.datasets #3631 (comment)

Description

Providing a minimal, but up-to-date source for https://github.com/vega/vega-datasets.

This PR takes a different approach to that of https://github.com/altair-viz/vega_datasets, notably:

• No datasets are included in the package
  • Instead, several metadata files form a dense summary of vega-datasets/datapackage.json
  • 3 files provide a reduction (70-150kb -> 15kb) and optimized views for this particular use-case
  • Includes redundancies for missing dependencies
• Strong support for typing
  • Annotations are generated from the metadata itself
  • https://github.com/vega/altair/blob/9e9deeb95668d2c4e7d30311e85a8f9f6acdc88c/altair/datasets/_typing.py
• So far, 4 backends have been implemented, instead of only pandas
  • These provide precise IDE completions, with a lot of help from https://github.com/narwhals-dev/narwhals
• Users can opt-out of caching remote dataset requests
  • With the "polars" backend, the slowest I've had on a cache-hit is 0.1s to load
    • https://cdn.jsdelivr.net/npm/[email protected]/data/flights-200k.json

Examples

These all come from the docstrings of:

• Loader
• Loader.from_backend
• Loader.__call__

from altair.datasets import Loader

load = Loader.from_backend("polars")
>>> load
Loader[polars]

cars = load("cars")

>>> type(cars)
polars.dataframe.frame.DataFrame

load = Loader.from_backend("pandas")
cars = load("cars")

>>> type(cars)
pandas.core.frame.DataFrame

load = Loader.from_backend("pandas[pyarrow]")
cars = load("cars")

>>> type(cars)
pandas.core.frame.DataFrame

>>> cars.dtypes
Name                       string[pyarrow]
Miles_per_Gallon           double[pyarrow]
Cylinders                   int64[pyarrow]
Displacement               double[pyarrow]
Horsepower                  int64[pyarrow]
Weight_in_lbs               int64[pyarrow]
Acceleration               double[pyarrow]
Year                timestamp[ns][pyarrow]
Origin                     string[pyarrow]
dtype: object

load = Loader.from_backend("pandas")
source = load("stocks")

>>> source.columns
Index(['symbol', 'date', 'price'], dtype='object')

load = Loader.from_backend("pyarrow")
source = load("stocks")

>>> source.column_names
['symbol', 'date', 'price']

[mattijn]

I’ve added an item to the tracking list in OP. I raised this suggestion during the review as an important improvement to the user interface. I’d be grateful if it could be addressed before merging and not is overlooked. Almost there!🙌

[dangotbanned]

I’ve added an item to the tracking list in OP. I raised this suggestion during the review as an important improvement to the user interface. I’d be grateful if it could be addressed before merging and not is overlooked. Almost there!🙌

Thanks @mattijn

I actually have that one bookmarked, rest assured I have not forgotten!

This was what I was thinking of doing, while we wait on narwhals-dev/narwhals#1924 and vega/vega-datasets#654:

• converting (`vega_datasets` source #3150) into an issue
• adding some background on the larger changes since my initial RFC
• continue the discussion on
  • feat(RFC): Adds altair.datasets #3631 (comment)
  • https://github.com/vega/altair/pull/3631/files/7b3a89e5b5374eb391b7ae73ace219327069f979#r1848299112

FYI, I have tried responding to (#3631 (comment)) twice but both times I ended up writing waaaaay too much.
I've still got those in-progress responses - but hopefully I'll be able to trim it down into something more digestible

Note

(to self) stashed an experiment w/ this under the name "datasets-wip-211"

[mattijn]

Thanks! Appreciate you having this on your list. Great that we’re aligned.

[dangotbanned]

Thanks! Appreciate you having this on your list. Great that we’re aligned.

No worries @mattijn, sorry for not communicating any of this sooner.
Thanks for taking the time to go through this PR