-
Notifications
You must be signed in to change notification settings - Fork 1k
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
operator example runbook & api reference docs
Signed-off-by: Tommy Hughes <[email protected]>
- Loading branch information
1 parent
28d91b6
commit c8d7c48
Showing
15 changed files
with
1,357 additions
and
4 deletions.
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,282 @@ | ||
| { | ||
| "cells": [ | ||
| { | ||
| "cell_type": "markdown", | ||
| "metadata": {}, | ||
| "source": [ | ||
| "# Install Feast on Kind with the Feast Operator\n", | ||
| "## Objective\n", | ||
| "\n", | ||
| "Provide a reference implementation of a runbook to deploy a Feast development environment on a Kubernetes cluster using [Kind](https://kind.sigs.k8s.io/docs/user/quick-start) and the [Feast Operator](../../infra/feast-operator/)." | ||
| ] | ||
| }, | ||
| { | ||
| "cell_type": "markdown", | ||
| "metadata": {}, | ||
| "source": [ | ||
| "## Prerequisites\n", | ||
| "* Kubernetes Cluster - e.g. [Kind](https://kind.sigs.k8s.io/) or OpenShift\n", | ||
| "* [kubectl](https://kubernetes.io/docs/tasks/tools/#kubectl) Kubernetes CLI tool." | ||
| ] | ||
| }, | ||
| { | ||
| "cell_type": "markdown", | ||
| "metadata": {}, | ||
| "source": [ | ||
| "## Install Prerequisites\n", | ||
| "\n", | ||
| "The following commands install and configure all the prerequisites on MacOS environment. You can find the\n", | ||
| "equivalent instructions on the offical documentation pages:\n", | ||
| "* Install Kind and Container runtime (e.g. [Colima](https://github.com/abiosoft/colima)).\n", | ||
| "* Create Kind cluster named `feast`.\n", | ||
| "* Install and setup the `kubectl` context.\n", | ||
| "```bash\n", | ||
| "brew install colima\n", | ||
| "colima start\n", | ||
| "brew install kind\n", | ||
| "kind create cluster --name feast\n", | ||
| "kind start\n", | ||
| "brew install kubectl\n", | ||
| "kubectl config use-context kind-feast\n", | ||
| "```\n", | ||
| "\n", | ||
| "Additionally, we create a `feast` namespace and use it as the default for the `kubectl` CLI:" | ||
| ] | ||
| }, | ||
| { | ||
| "cell_type": "code", | ||
| "execution_count": null, | ||
| "metadata": {}, | ||
| "outputs": [], | ||
| "source": [ | ||
| "!kubectl create ns feast\n", | ||
| "!kubectl config set-context --current --namespace feast" | ||
| ] | ||
| }, | ||
| { | ||
| "cell_type": "markdown", | ||
| "metadata": {}, | ||
| "source": [ | ||
| "Validate the cluster setup:" | ||
| ] | ||
| }, | ||
| { | ||
| "cell_type": "code", | ||
| "execution_count": null, | ||
| "metadata": {}, | ||
| "outputs": [], | ||
| "source": [ | ||
| "!kubectl get ns feast" | ||
| ] | ||
| }, | ||
| { | ||
| "cell_type": "markdown", | ||
| "metadata": {}, | ||
| "source": [ | ||
| "## Deployment Architecture\n", | ||
| "The primary objective of this runbook is to guide the deployment of Feast services on a Kubernetes Kind cluster, using the `postgres` template to set up a basic feature store.\n", | ||
| "\n", | ||
| "In this notebook, we will deploy a distributed topology of Feast services, which includes:\n", | ||
| "\n", | ||
| "* `Registry Server`: Handles metadata storage for feature definitions.\n", | ||
| "* `Online Store Server`: Uses the `Registry Server` to query metadata and is responsible for low-latency serving of features.\n", | ||
| "* `Offline Store Server`: Uses the `Registry Server` to query metadata and provides access to batch data for historical feature retrieval.\n", | ||
| "\n", | ||
| "Each service is backed by a `PostgreSQL` database, which is also deployed within the same Kind cluster." | ||
| ] | ||
| }, | ||
| { | ||
| "cell_type": "markdown", | ||
| "metadata": {}, | ||
| "source": [ | ||
| "## Install PostgreSQL\n", | ||
| "Install the [reference deployment](postgres.yaml) to install and configure a simple PostgreSQL database." | ||
| ] | ||
| }, | ||
| { | ||
| "cell_type": "code", | ||
| "execution_count": null, | ||
| "metadata": {}, | ||
| "outputs": [], | ||
| "source": [ | ||
| "!kubectl apply -f postgres.yaml\n", | ||
| "!kubectl wait --for=condition=available deployment/postgres --timeout=2m" | ||
| ] | ||
| }, | ||
| { | ||
| "cell_type": "code", | ||
| "execution_count": null, | ||
| "metadata": {}, | ||
| "outputs": [], | ||
| "source": [ | ||
| "!kubectl get all" | ||
| ] | ||
| }, | ||
| { | ||
| "cell_type": "markdown", | ||
| "metadata": {}, | ||
| "source": [ | ||
| "### Install the Operator" | ||
| ] | ||
| }, | ||
| { | ||
| "cell_type": "code", | ||
| "execution_count": null, | ||
| "metadata": {}, | ||
| "outputs": [], | ||
| "source": [ | ||
| "!kubectl apply -f ../../infra/feast-operator/dist/install.yaml\n", | ||
| "#!make -C ../../infra/feast-operator install\n", | ||
| "#!make -C ../../infra/feast-operator deploy IMG=quay.io/<org>/feast-operator:<tag>" | ||
| ] | ||
| }, | ||
| { | ||
| "cell_type": "markdown", | ||
| "metadata": {}, | ||
| "source": [ | ||
| "## Install the Feast services via FeatureStore CR\n", | ||
| "We'll use the Operator in this local repository to install the feast services. Install the [reference deployment](feast.yaml) to install and configure Feast." | ||
| ] | ||
| }, | ||
| { | ||
| "cell_type": "code", | ||
| "execution_count": null, | ||
| "metadata": {}, | ||
| "outputs": [], | ||
| "source": [ | ||
| "!kubectl apply -f feast.yaml" | ||
| ] | ||
| }, | ||
| { | ||
| "cell_type": "markdown", | ||
| "metadata": {}, | ||
| "source": [ | ||
| "### Validate deployment\n", | ||
| "Validate application and service status:" | ||
| ] | ||
| }, | ||
| { | ||
| "cell_type": "code", | ||
| "execution_count": null, | ||
| "metadata": {}, | ||
| "outputs": [], | ||
| "source": [ | ||
| "!kubectl get feast\n", | ||
| "!kubectl get all\n", | ||
| "!kubectl wait --for=condition=available deployment/feast-example-registry --timeout=2m\n", | ||
| "!kubectl wait --for=condition=available deployment/feast-example-offline --timeout=2m\n", | ||
| "!kubectl wait --for=condition=available deployment/feast-example-online --timeout=2m" | ||
| ] | ||
| }, | ||
| { | ||
| "cell_type": "markdown", | ||
| "metadata": {}, | ||
| "source": [ | ||
| "Then verify the content of each feast service's local configuration file (it's stored in the `FEATURE_STORE_YAML_BASE64` env variable)." | ||
| ] | ||
| }, | ||
| { | ||
| "cell_type": "code", | ||
| "execution_count": null, | ||
| "metadata": {}, | ||
| "outputs": [], | ||
| "source": [ | ||
| "!kubectl get deploy feast-example-online -o jsonpath='{.spec.template.spec.containers[*].env[?(@.name==\"FEATURE_STORE_YAML_BASE64\")].value}' | base64 -d" | ||
| ] | ||
| }, | ||
| { | ||
| "cell_type": "code", | ||
| "execution_count": null, | ||
| "metadata": {}, | ||
| "outputs": [], | ||
| "source": [ | ||
| "!kubectl get deploy feast-example-offline -o jsonpath='{.spec.template.spec.containers[*].env[?(@.name==\"FEATURE_STORE_YAML_BASE64\")].value}' | base64 -d" | ||
| ] | ||
| }, | ||
| { | ||
| "cell_type": "code", | ||
| "execution_count": null, | ||
| "metadata": {}, | ||
| "outputs": [], | ||
| "source": [ | ||
| "!kubectl get deploy feast-example-registry -o jsonpath='{.spec.template.spec.containers[*].env[?(@.name==\"FEATURE_STORE_YAML_BASE64\")].value}' | base64 -d" | ||
| ] | ||
| }, | ||
| { | ||
| "cell_type": "markdown", | ||
| "metadata": {}, | ||
| "source": [ | ||
| "Verify the content of the feast client configuration file (it's generated by the Operator and stored in a `ConfigMap`)." | ||
| ] | ||
| }, | ||
| { | ||
| "cell_type": "code", | ||
| "execution_count": null, | ||
| "metadata": {}, | ||
| "outputs": [], | ||
| "source": [ | ||
| "!kubectl get cm feast-example-client -o jsonpath='{.data.feature_store\\.yaml}'" | ||
| ] | ||
| }, | ||
| { | ||
| "cell_type": "markdown", | ||
| "metadata": {}, | ||
| "source": [ | ||
| "Verify that the DB includes the expected tables." | ||
| ] | ||
| }, | ||
| { | ||
| "cell_type": "code", | ||
| "execution_count": null, | ||
| "metadata": {}, | ||
| "outputs": [], | ||
| "source": [ | ||
| "!kubectl exec deploy/postgres -- psql -h localhost -U feast feast -c '\\dt'" | ||
| ] | ||
| }, | ||
| { | ||
| "cell_type": "markdown", | ||
| "metadata": {}, | ||
| "source": [ | ||
| "Finally, let's verify the `feast` version in each server" | ||
| ] | ||
| }, | ||
| { | ||
| "cell_type": "code", | ||
| "execution_count": null, | ||
| "metadata": {}, | ||
| "outputs": [], | ||
| "source": [ | ||
| "!kubectl exec deployment/feast-example-registry -- feast version\n", | ||
| "!kubectl exec deployment/feast-example-offline -- feast version\n", | ||
| "!kubectl exec deployment/feast-example-online -- feast version" | ||
| ] | ||
| }, | ||
| { | ||
| "cell_type": "markdown", | ||
| "metadata": {}, | ||
| "source": [] | ||
| } | ||
| ], | ||
| "metadata": { | ||
| "kernelspec": { | ||
| "display_name": "Python 3 (ipykernel)", | ||
| "language": "python", | ||
| "name": "python3" | ||
| }, | ||
| "language_info": { | ||
| "codemirror_mode": { | ||
| "name": "ipython", | ||
| "version": 3 | ||
| }, | ||
| "file_extension": ".py", | ||
| "mimetype": "text/x-python", | ||
| "name": "python", | ||
| "nbconvert_exporter": "python", | ||
| "pygments_lexer": "ipython3", | ||
| "version": "3.12.4" | ||
| } | ||
| }, | ||
| "nbformat": 4, | ||
| "nbformat_minor": 4 | ||
| } |
Oops, something went wrong.