Add admin facing UI for delta lake dump and spec configuration

[bdpedigo]

This is currently deployed on LTV5

Select an export

Select output options for one or more delta lake partition/zorder/filter schemes

Confirmation

Logging

Progress for long running jobs

Success

Fail on export that already exists, general error reporting

[Copilot]

Pull request overview

Adds an admin-facing Delta Lake export wizard (3-step UI) plus a “running exports” monitoring page, and extends the export workflow/API to support per-spec output settings and richer progress reporting (phase, error, log entries) keyed by a job id.

Changes:

• Introduces new Delta Lake wizard and running-exports pages (templates + Alpine.js store/components).
• Extends deltalake_export workflow with per-spec fields (name, target_file_size_mb, bloom filter FPP), job-scoped Redis progress keys, and capped Redis log entries.
• Updates the materialize API to return a job_id and accept job_id for progress polling; adds a new deltalake blueprint for wizard helper APIs (discover/recalculate/check-exists).

Reviewed changes

Copilot reviewed 17 out of 18 changed files in this pull request and generated 10 comments.

Show a summary per file

File	Description
tests/test_deltalake_export.py	Adds unit coverage for new output spec fields and Redis progress/log payloads.
templates/deltalake/step1.html	Wizard step 1 UI for selecting datastack/version/table and advanced defaults.
templates/deltalake/step2.html	Wizard step 2 UI for editing output specs (partitioning/zorder/bloom/file size).
templates/deltalake/step3.html	Wizard step 3 confirmation + export launch.
templates/deltalake/running_exports.html	Monitoring UI for progress, phase, errors, and log tail.
templates/deltalake_wizard.html	Wizard container and step indicator/progress UI.
templates/base.html	Adds navigation links to the Delta Lake export wizard and running exports page.
static/js/deltalakeWizardStore.js	Alpine store for persisting wizard state and exports list in localStorage.
static/js/deltalakeStep1.js	Step 1 logic for fetching versions/tables, discovery, and exists checks.
static/js/deltalakeStep2.js	Step 2 spec editing + server-side partition recalculation call.
static/js/deltalakeStep3.js	Step 3 export launch (POST) and redirect to monitoring, stores job_id.
static/js/deltalakeRunningExports.js	Polling loop for progress endpoint (with optional job_id).
materializationengine/workflows/deltalake_export.py	Workflow updates: richer spec dataclass, optimize callbacks, Redis progress/logging keyed by job.
materializationengine/blueprints/materialize/api.py	Adds job_id to export enqueue response; supports job_id query param for progress polling.
materializationengine/blueprints/deltalake/api.py	New blueprint: wizard page routes + helper APIs (discover/recalc/check-exists).
materializationengine/blueprints/deltalake/__init__.py	Exposes the new deltalake_bp.
materializationengine/app.py	Registers the new deltalake blueprint; minor formatting/import ordering updates.
.gitignore	Ignores preview_ui.py.

Comments suppressed due to low confidence (1)

materializationengine/workflows/deltalake_export.py:1445

• Unused import: create_engine is imported but never referenced in this function. Removing it avoids lint noise and makes dependencies clearer.

    from dynamicannotationdb.key_utils import build_segmentation_table_name
    from sqlalchemy import create_engine

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[Copilot]

Pull request overview

Copilot reviewed 17 out of 18 changed files in this pull request and generated 9 comments.

[Copilot]

Pull request overview

Copilot reviewed 17 out of 18 changed files in this pull request and generated 6 comments.