Workflow orchestration with Prefect

Getting Started

1. Install dependencies from pyproject.toml and activate the created virtualenv:

uv sync && source .venv/bin/activate

2. (Optional) Install pre-commit:

brew install pre-commit

# From root folder where `.pre-commit-config.yaml` is located, run:
pre-commit install

3. Start the Prefect Server:

Note: The Prefect Orion server is now called Prefect Server

prefect server start

4. Setup Prefect server for the flows:

prefect config set PREFECT_API_URL=http://localhost:4200/api

Set the GOOGLE_APPLICATION_CREDENTIALS env variable:

export GOOGLE_APPLICATION_CREDENTIALS=/path/to/gcs_credentials.json

Then, execute with:

python flows/web_cs_to_gcs.py

Set the environment variables:

export DB_USERNAME=postgres
export DB_PASSWORD=postgres
export DB_HOST=localhost
export DB_PORT=5432
export DB=nyc_taxi

Now, execute it with:

python flows/sqlalchemy_ingest.py