Unable to recognize polars Enum or DuckDB Enum?

[bzm3r]

I am using "ibis-framework[duckdb,polars]=9.3.0". I create a polars dataframe:

import polars as pl

xs = ["a", "b", "c"]
xs = pl.Series("x", xs, dtype=pl.Enum(xs))
df = pl.DataFrame(xs)
print(df)

Which produces:

shape: (3, 1)
┌──────┐
│ x    │
│ ---  │
│ enum │
╞══════╡
│ a    │
│ b    │
│ c    │
└──────┘

Then, I load this using Ibis:

import ibis

ibis.options.interactive = True
t = ibis.memtable(df, name="t")
print(t)

Which produces:

File ~/.venv/lib/python3.12/site-packages/ibis/expr/api.py:557, in _memtable_from_polars_dataframe(data, name, schema, columns)
    553     assert schema is None, "if `columns` is not `None` then `schema` must be `None`"
    554     schema = sch.Schema(dict(zip(columns, sch.infer(data).values())))
...
     75     )
     76 else:
---> 77     return _from_polars_types[base_type](nullable=nullable)

KeyError: Enum

I do the same thing, but this time using DuckDB:

from pathlib import Path
import duckdb as ddb

path = Path("enum.db")
if path.exists():
    path.unlink()

# create a table, and then let the conneciton close
with ddb.connect("enum.db") as db:
    db.sql(
        "CREATE TYPE xtype AS ENUM ('a', 'b', 'c');"
        "CREATE TABLE dummy (xs xtype);"
        "INSERT INTO dummy VALUES ('a'), ('b'), ('c');"
    )

# reconnect to the table, and confirm that we have an enum-type column
with ddb.connect("enum.db") as db:
    db.table("dummy").show()

┌─────────────────────┐
│         xs          │
│ enum('a', 'b', 'c') │
├─────────────────────┤
│ a                   │
│ b                   │
│ c                   │
└─────────────────────┘

Okay, now loading it using ibis:

import ibis

ibis.options.interactive = True
conn = ibis.connect("duckdb://enum.db")
conn.table("dummy")

┏━━━━━━━━┓
┃ xs     ┃
┡━━━━━━━━┩
│ string │
├────────┤
│ a      │
│ b      │
│ c      │
└────────┘

So we get the VARCHAR type from DuckDB, not the enum type, but at least the table loaded without error.

It seems like ibis knows about Enum...but...yeah, I am not sure what to make of what's going on. What am I misunderstanding?

[gforsyth]

Thanks for the report @bzm3r ! The polars issue looks like a bug, we can fix that up.

The DuckDB behavior is expected -- Ibis treats them as a string column, but that's only in our representation, they still exist as enum types in the backend table.

[gforsyth]

Hi @bzm3r -- the polars bug is fixed on main now.

[bzm3r]

@gforsyth 🤍