Bridge HuggingFace datasets with Apache Iceberg tables — no data copying, just metadata.
Faceberg maps HuggingFace datasets to Apache Iceberg tables. Your catalog metadata lives on HuggingFace Spaces (or Buckets) with an auto-deployed REST API, and any Iceberg-compatible query engine can access the data.
Note
Faceberg is early-stage, alpha-quality software — APIs and CLI flags may still evolve between releases. See CHANGELOG.md for what's landed so far.
pip install facebergRequires Python 3.10+. The REST catalog server (faceberg serve) is included by default.
A HuggingFace token is required for remote catalogs (user/repo, hf://...). Local catalogs (plain paths) don't need one.
The token is automatically discovered from the default cache (~/.cache/huggingface/token), so no environment variable is needed. You can also pass it explicitly via the CLI or Python API, or set HF_TOKEN as a fallback.
Get a token from HuggingFace Settings and log in:
hf auth loginOr export it manually:
export HF_TOKEN=your_huggingface_token# Create a catalog on HuggingFace Hub (deploys a Space with a REST API)
# No HF_TOKEN needed — automatically discovered from ~/.cache/huggingface/token
faceberg user/mycatalog init
# Add datasets — table identifier is inferred (org.repo) unless --table is given
faceberg user/mycatalog add stanfordnlp/imdb
faceberg user/mycatalog add openai/gsm8k --config main
# List what's in the catalog
faceberg user/mycatalog list
# Query with interactive DuckDB shell
faceberg user/mycatalog quackSELECT label, substr(text, 1, 100) as preview
FROM faceberg.stanfordnlp.imdb
LIMIT 10;Every command follows the same shape: faceberg <catalog-uri> <command> [args], where <catalog-uri> is one of:
| URI form | Backend |
|---|---|
org/repo or hf://datasets/org/repo |
HuggingFace dataset repo (Space auto-deployed) |
hf://spaces/org/repo |
HuggingFace Space |
hf://buckets/org/repo |
HuggingFace Bucket (S3-like, no git history) |
./path or /abs/path |
Local filesystem catalog |
| Command | Description |
|---|---|
init [config.yml] |
Create the catalog. Auto-discovers ./faceberg.yml if no path is given; --sync populates tables immediately. |
add <dataset> |
Add a HuggingFace dataset as a table. --table ns.table sets an explicit identifier, --config selects a dataset config. |
sync [table] |
Re-check datasets for new revisions and update Iceberg metadata. Omit table to sync everything; --tree-view shows progress as a tree. |
list |
List all namespaces/tables in the catalog. |
info <table> |
Show a table's schema, partitioning, and metadata location. |
scan <table> |
Read and print sample rows. --limit/-n controls row count (default 5). |
remove <identifier> |
Drop a table (ns.table) or an empty namespace. --yes skips the confirmation prompt. |
serve |
Start an Iceberg REST catalog server. --host, --port (default 8181), --reload, --prefix. |
quack |
Open an interactive DuckDB shell with the catalog pre-attached. --endpoint overrides auto-detection. |
Run faceberg <uri> <command> --help for full flag details and examples on any command.
Catalogs are described by a faceberg.yml config that tracks dataset-to-table mappings:
default:
imdb:
type: dataset
repo: stanfordnlp/imdb
config: plain_text
gsm8k:
type: dataset
repo: openai/gsm8k
config: mainfaceberg init tables.yml bootstraps a catalog from a file like this in one shot; faceberg sync re-reads it and updates any table whose source dataset revision changed.
For development, testing, or CI, point the CLI at a filesystem path instead of a HuggingFace URI — no token needed:
faceberg ./mycatalog init
faceberg ./mycatalog add stanfordnlp/imdb --config plain_text
faceberg ./mycatalog serve --port 8181 # in one terminal
faceberg ./mycatalog quack # in anotherSee Local Catalogs for the on-disk layout and testing patterns.
Catalog metadata can also live in a HuggingFace Bucket (hf://buckets/org/name) instead of a Space — an S3-like storage backend with no git history, useful for catalogs that don't need a hosted REST endpoint:
faceberg hf://buckets/user/mycatalog init
faceberg hf://buckets/user/mycatalog add stanfordnlp/imdbHuggingFace Hub
┌─────────────────────────────────────────────────────────┐
│ │
│ ┌─────────────────────┐ ┌─────────────────────────┐ │
│ │ HF Datasets │ │ HF Spaces (Catalog) │ │
│ │ (Original Parquet) │◄───│ • Iceberg metadata │ │
│ │ │ │ • REST API endpoint │ │
│ │ stanfordnlp/imdb/ │ │ • faceberg.yml │ │
│ │ └── *.parquet │ │ │ │
│ └─────────────────────┘ └───────────┬─────────────┘ │
│ │ │
└─────────────────────────────────────────┼───────────────┘
│ Iceberg REST API
▼
┌─────────────────────────┐
│ Query Engines │
│ DuckDB, Pandas, Spark │
└─────────────────────────┘
No data is copied — only metadata is created. Query with DuckDB, PyIceberg, Spark, or any Iceberg-compatible tool.
import os
from faceberg import catalog
cat = catalog("user/mycatalog", hf_token=os.environ.get("HF_TOKEN"))
table = cat.load_table("stanfordnlp.imdb")
df = table.scan(limit=100).to_pandas()The catalog object exposes the usual Iceberg operations:
| Method | Description |
|---|---|
init(config) |
Initialize catalog storage, optionally with a Config |
config() |
Load the catalog's faceberg.yml configuration |
add_dataset(identifier, repo, config) |
Add a HuggingFace dataset as an Iceberg table |
sync_dataset(identifier) / sync_datasets() |
Sync one or all datasets (update if source changed) |
load_table(identifier) |
Load a table for querying |
list_tables(namespace) / list_namespaces() |
Enumerate tables / namespaces |
drop_table(identifier) / drop_namespace(identifier) |
Remove a table or empty namespace |
table_exists(identifier) |
Check if a table exists |
import pandas as pd
df = pd.read_iceberg(
table_identifier="stanfordnlp.imdb",
catalog_name="faceberg",
catalog_properties={"type": "rest", "uri": "https://user-mycatalog.hf.space"},
columns=["text", "label"],
limit=10,
)See Pandas Integration for the local-catalog variant and more examples.
Your catalog is accessible to anyone via the REST API:
import duckdb
conn = duckdb.connect()
conn.execute("INSTALL iceberg; LOAD iceberg")
conn.execute("""
ATTACH 'https://user-mycatalog.hf.space' AS cat (
TYPE ICEBERG,
ENDPOINT 'https://user-mycatalog.hf.space',
AUTHORIZATION_TYPE 'none'
)
""")
result = conn.execute("SELECT * FROM cat.stanfordnlp.imdb LIMIT 5").fetchdf()To make your catalog private, set the underlying HuggingFace Space/dataset/Bucket to private.
Every deployed catalog Space also serves a web UI at its own URL — no install
needed. Visit https://user-mycatalog.hf.space to browse namespaces and
tables, and run SQL directly against a DuckDB-WASM shell in the page.
- Getting Started — Full quickstart guide
- Local Catalogs — Use local catalogs for development
- Buckets — Store catalog metadata in HF Buckets
- DuckDB Integration — Advanced SQL queries
- Pandas Integration — Load into DataFrames
- Architecture — How Faceberg maps datasets to Iceberg metadata
git clone https://github.com/huggingface/faceberg
cd faceberg
pip install -e '.[dev]'Common tasks are wired up via just (see the justfile):
just test # run the test suite (pytest faceberg/tests/)
just cov # run tests with coverage
just format # format code with ruff
just check # lint + format check
just build # build distribution packagesDocs are written in Quarto (docs/*.qmd) and published to https://faceberg.kszucs.dev/.
Apache 2.0