Feast v0.10.1

Fixed bugs:

• Fix time zone issue with get_historical_features #1475 (tsotnet)

Merged pull requests:

• Improve exception handling, logging, and validation #1477 (woop)
• Remove duped pic #1476 (YikSanChan)
• Fix created timestamp related errors for BigQuery source #1474 (jklegar)
• Remove unnecessary MAVEN_CONFIG #1472 (danielsiwiec)
• Fix CLI entities command & add feature-views command #1471 (tsotnet)

Feast v0.10.0

Overview

Feast 0.10 brings major improvements towards making Feast an easy to use and operate feature store. Feast 0.10 ships with a lightweight SDK/CLI that allows teams to stand up a local or production feature store within seconds. This latest release also comes with extensibility in its provider model, allowing teams to easily add new data stores, compute layers, or cloud providers.

👥 Contributors

Thanks to @jklegar, @oavdeev, @tsotnet, @woop, @terryyylim, @randxie, @josegpg, @mrzzy, @pyalex, @tedhtchang, @theofpa, @YikSanChan and @jparthasarthy for the pull requests and contributions!

✨ New Features:

Add template generation to Feast CLI for Google Cloud Platform #1460 (woop)
Add support for retrieving data from sources that don't match providers #1454 (woop)
Add materialize-incremental CLI command #1442 (tsotnet)
Add registry refreshing and caching #1431 (woop)
Add missing FeatureStore methods #1423 (jklegar)
Allow importing of new FeatureStore classes #1422 (woop)
Add Feast init command #1414 (oavdeev)
Add support for parquet ingestion #1410 (oavdeev)
Add materialize_incremental method #1407 (jklegar)
Add support for pull query from BigQuery #1403 (jklegar)
Add support for partial apply to create infra #1402 (oavdeev)
Add online read API to FeatureStore class #1399 (oavdeev)
Add historical retrieval for BigQuery and Parquet #1389 (woop)
Add feature views #1386 (oavdeev)
Implement materialize method #1379 (jklegar)
Read and write path for Datastore and SQLite #1376 (oavdeev)
Download BigQuery table to pyarrow table for python-based ingestion flow #1366 (jklegar)
FeatureStore, FeatureView, Config, and BigQuerySource classes for updated SDK #1364 (jklegar)
Add support for new deploy CLI #1362 (oavdeev)

🔴 Fixed bugs:

Fix time zone access with native python datetimes #1469 (tsotnet)
Small fixes for created_timestamp #1468 (jklegar)
Fix offline store (tz-naive & field_mapping issues) #1466 (tsotnet)
Fix get_online_features return schema #1455 (jklegar)
Fix noisy path warning #1452 (woop)
Fix flaky test_feature_store fixture #1447 (jklegar)
Use timestamp check for token refresh #1444 (terryyylim)
Fix bug in event timestamp removal in local mode #1441 (jklegar)
Fix timezone issue in materialize & materialize_incremental #1439 (tsotnet)
Fix materialization_intervals initialization in FeatureView #1438 (tsotnet)
Fix broken Terraform installation files #1420 (josegpg)
Fix retry handling for GCP datastore #1416 (oavdeev)
Make CLI apply in local mode idempotent #1401 (oavdeev)
Fix a bug in client archive_project method and fix lint in grpc auth #1396 (randxie)

🔨 Merged pull requests:

Change GCP template names to match local template #1470 (jklegar)
Add logging to materialize #1467 (woop)
Validate timestamp column present in entity dataframe #1464 (jklegar)
Fix & clean up Feast CLI commands #1463 (tsotnet)
Flatten configuration structure for online store #1459 (woop)
Optimize write rate in Gcp Firestore #1458 (tsotnet)
Allow apply to take a single Entity or FeatureView #1457 (jklegar)
Validate datetimes in materialize in correct order #1456 (jklegar)
Add test to ensure saving and loading from registry is safe #1453 (woop)
Port telemetry to FeatureStore API #1446 (jklegar)
Add materialize-incremental cli test #1445 (tsotnet)
Support join keys in historical feature retrieval #1440 (jklegar)
Refactor OfflineStore into Provider #1437 (woop)
Fix multi-entity online retrieval #1435 (woop)
Fix feature name consistency between online & historical apis #1434 (tsotnet)
Rename Metadata Store to Registry #1433 (woop)
Add support for Pydantic as configuration loader #1432 (woop)
Add entity join key and fix entity references #1429 (jklegar)
Slightly more sensible test names #1428 (oavdeev)
Make entity description optional and fix empty table_ref #1425 (jklegar)
Add Development Guide for Main Feast Repo Feast Components #1424 (mrzzy)
Fix protobuf building for Python SDK #1418 (woop)
Add project name generator #1417 (woop)
[Python SDK][Auth] Refresh token id w/o gcloud cli #1413 (pyalex)
Firestore ingestion perf improvements + benchmark script #1411 (oavdeev)
Fixed old urls in documentation #1406 (tedhtchang)
Upgrade Gcloud setup dependency #1405 (woop)
Fix documentation building for Feast SDK #1400 (woop)
Bump jinja2 from 2.11.2 to 2.11.3 in /sdk/python #1398 (dependabot[bot])
Improve spark-on-k8s-operator documentation #1397 (jklegar)
Update Python SDK dependencies #1394 (woop)
Move Python proto package into submodule #1393 (woop)
Add nicer validation for repo config #1392 (oavdeev)
Remove Python CI dependencies #1390 (woop)
Move Project field to Table/View spec #1388 (woop)
Remove Mirror CI #1387 (woop)
Add feedback link to install docs page #1375 (jparthasarthy)
Support multiple features per key in firestore format spec #1374 (oavdeev)
Fix hashing algorithm in the firestore spec #1373 (oavdeev)
Support make protos on Mac #1371 (tedhtchang)
Add support for privileged tests #1369 (woop)
Remove base tests folder #1368 (woop)
Add Firestore online format specification #1367 (oavdeev)
Improve documentation for k8s-spark resource template #1363 (theofpa)

Feast v0.9.1

Changelog

v0.9.1 (2021-01-29)

Full Changelog

Implemented enhancements:

• Add telemetry to Python SDK #1289 (jklegar)

Fixed bugs:

• Fix kafka download url #1298 (jklegar)
• disable telemetry in docker-compose test and job_service #1297 (jklegar)

Feast v0.9.0

Full Changelog

Implemented enhancements:

• Enable user to provide spark job template as input for jobservice deployment #1285 (khorshuheng)
• Add feature table name filter to jobs list api #1282 (terryyylim)
• Report observed value for aggregated checks in pre-ingestion feature validation #1278 (pyalex)
• Add docs page for Azure setup #1276 (jklegar)
• Azure example terraform #1274 (jklegar)

Fixed bugs:

• make EMR jar uploader work the same as k8s one #1284 (oavdeev)
• Don't error when azure vars not set #1277 (jklegar)
• Prevent ingestion job config parser from unwanted fieldMapping transformation #1261 (pyalex)
• Features are not being ingested due to max age overflow #1209 (pyalex)
• Feature Table is not being update when only max_age was changed #1208 (pyalex)
• Truncate staging timestamps in entities dataset to ms #1207 (pyalex)
• Bump terraform rds module version #1204 (oavdeev)

Merged pull requests:

• Use date partitioning column in FileSource #1293 (pyalex)
• Add EMR CI/CD entrypoint script #1290 (oavdeev)
• Online serving optimizations #1286 (pyalex)
• Make third party grpc packages recognizable as python module #1283 (khorshuheng)
• Report observed values in feature validation as Gauge #1280 (pyalex)
• Keep same amount of partitions after repartitioning in IngestionJob #1279 (pyalex)
• Add request feature counter metric #1272 (terryyylim)
• Use SEND_INTERRUPT to cancel EMR jobs #1271 (oavdeev)
• Fix historical test flakiness #1270 (jklegar)
• Allow https url for spark ingestion jar #1266 (jklegar)
• Add project name to feature validation metric #1264 (pyalex)
• Use dataproc console url instead of gcs for log uri #1263 (khorshuheng)
• Make nodes priority (for redis cluster) configurable in Serving #1260 (pyalex)
• Enhance job api to return associated feature table and start time #1259 (khorshuheng)
• Reporting metrics from validation UDF #1256 (pyalex)
• Allow use the same timestamp column for both created & even timestamp in Historical Retrieval #1255 (pyalex)
• Apply grpc tracing interceptor on Feast SDK #1243 (khorshuheng)
• Apply grpc tracing interceptor on online serving #1242 (khorshuheng)
• Python UDF in Ingestion being used for feature validation #1234 (pyalex)
• Add spark k8s operator launcher #1225 (oavdeev)
• Add deadletter/read-from-source metrics to batch and stream ingestion #1223 (terryyylim)
• Implement AbstractStagingClient for azure blob storage #1218 (jklegar)
• Configurable materialization destination for view in BigQuerySource #1201 (pyalex)
• Update Feast Core list features method #1176 (terryyylim)
• S3 endpoint configuration #1169 #1172 (mike0sv)
• Increase kafka consumer waiting time in e2e tests #1268 (pyalex)
• E2E tests support for jobservice's control loop #1267 (pyalex)
• Optimize memory footprint for Spark Ingestion Job #1265 (pyalex)
• Fix historical test for azure #1262 (jklegar)
• Change azure https to wasbs and add azure creds to spark #1258 (jklegar)
• Docs, fixes and scripts to run e2e tests in minikube #1254 (oavdeev)
• Fix azure blob storage access in e2e tests #1253 (jklegar)
• Update python version requirements to 3.7 for Dataproc launcher #1251 (pyalex)
• Fix build-ingestion-py-dependencies script #1250 (pyalex)
• Add datadog(statsd) client to python package for IngestionJob #1249 (pyalex)
• Add prow job for azure e2e test #1244 (jklegar)
• Azure e2e test #1241 (jklegar)
• Add Feast Serving histogram metrics #1240 (terryyylim)
• CI should work on python 3.6 #1237 (pyalex)
• Integration test for k8s spark operator support #1236 (oavdeev)
• Add prow config for spark k8s operator integration testing #1235 (oavdeev-tt)
• Upgrading spark to 3.0.1 #1227 (pyalex)
• Support TFRecord as one of the output formats for historical feature retrieval #1222 (khorshuheng)
• Remove stage_dataframe from the launcher interface #1220 (oavdeev)
• Refactor staging client uploader and use it in EMR launcher #1219 (oavdeev)
• Remove unused EMR code #1217 (oavdeev)
• Remove job id from ingested row counter metric #1216 (terryyylim)
• Quickstart link fixed #1213 (szczeles)
• Delete v1 concepts #1194 (terryyylim)
• Dont write defaults to config #1188 (mike0sv)
• Refactor tests which utilizes feature sets #1186 (terryyylim)
• Refactor configurable options and add sphinx docs #1174 (terryyylim)
• Remove unnecessary Google Auth dependency #1170 (woop)

Feast v0.8.4

Bug fixes
Spark job jar URL handling for EMR spark launcher

Feast v0.8.3

Fixed bugs

Spark job jar URL handling for EMR spark launcher

Feast v0.8.2

Implemented enhancements:

• Configurable materialization destination for view in BigQuerySource #1201 (pyalex)

Fixed bugs:

• Fix tag order for release workflow #1205 (terryyylim)
• Fix Feature Table not updated on new feature addition #1197 (khorshuheng)

Merged pull requests:

• Suppress kafka logs in Ingestion Job #1206 (pyalex)
• Add project name to metrics labels in Ingestion Job #1202 (pyalex)

Feast v0.8.1

Implemented enhancements:

• Expires Redis Keys based on Feature Table Max Age #1161 (khorshuheng)
• Jobservice control loop (based on #1140) #1156 (oavdeev)

Fixed bugs:

• Lazy metrics initialization (to correct pick up in executor) #1195 (pyalex)
• Add missing third_party folder #1185 (terryyylim)
• Fix missing name variable instantiation #1166 (terryyylim)

Merged pull requests:

• Bump ssh-agent version #1175 (terryyylim)
• Refactor configurable options and add sphinx docs #1174 (terryyylim)
• Stabilize flaky e2e tests #1173 (pyalex)
• Fix connection resets in CI for Maven #1164 (woop)
• Add dataproc executor resource config #1160 (terryyylim)
• Fix github workflow deprecating env variable #1158 (terryyylim)
• Ensure consistency of github workflow #1157 (terryyylim)

Feast v0.8.0

Implemented enhancements:

• Implement JobService API calls & connect it to SDK #1129 (tsotnet)
• Allow user to specify custom secrets to be mounted on Feast Serving and Feast Core pods #1127 (khorshuheng)
• Allow spark expressions in field mapping during Ingestion #1122 (pyalex)
• Update rest endpoints #1121 (terryyylim)
• Add feature table deletion #1114 (terryyylim)
• Add historical retrieval via job service #1107 (oavdeev)
• Implement list job and get job methods for Dataproc launcher #1106 (khorshuheng)
• Allow entities and features to be updated #1105 (terryyylim)
• Add get_by_id and list_jobs interface to the launcher interface and implement it for EMR #1095 (oavdeev)
• Support redis ssl in feast-serving #1092 (oavdeev)
• Add helm charts for feast jobservice #1081 (tsotnet)
• Terraform cleanup: tags, formatting, better defaults #1080 (oavdeev)
• Update docker-compose for Feast 0.8 #1078 (khorshuheng)
• Configure jupyter env on AWS via terraform #1077 (oavdeev)
• Implement EMR job cancelling #1075 (oavdeev)
• Streaming Ingestion Job supports AVRO format as input #1072 (pyalex)
• Accept Pandas dataframe as input for historical feature retrieval #1071 (khorshuheng)
• Add EMR streaming job launcher #1065 (oavdeev)
• EMR launcher #1061 (oavdeev)
• Add AWS managed kafka config to the example terraform #1058 (oavdeev)
• Feast SDK integration for historical feature retrieval using Spark #1054 (khorshuheng)
• Update GetOnlineFeatures method in sdks #1052 (terryyylim)
• "Start Offline-to-online ingestion" method in Python SDK #1051 (pyalex)
• Adding support for custom grpc dial options in Go SDK #1043 (ankurs)
• CLI command to start/stop/list streaming ingestion job on emr #1040 (oavdeev)
• Update serving service to handle new redis encoding #1038 (terryyylim)
• terraform config for aws #1033 (oavdeev)
• Streaming Ingestion Pipeline with Spark #1027 (pyalex)
• Run offline-to-online ingestion job on EMR #1026 (oavdeev)
• Add redis SSL support to the offline-to-online ingestion job #1025 (oavdeev)
• Dataproc and Standalone Cluster Spark Job launcher #1022 (khorshuheng)
• Pyspark job for feature batch retrieval #1021 (khorshuheng)
• Batch Ingestion Job rewritten on Spark #1020 (pyalex)
• Add Feature Tables API to Core & Python SDK #1019 (mrzzy)
• Introduce Entity as higher-level concept #1014 (terryyylim)

Fixed bugs:

• Fix stencil client serialization issue #1147 (pyalex)
• Deadletter path is being incorrectly joined #1144 (pyalex)
• In Historical Retrieval (SDK) use project from client context #1138 (pyalex)
• Pass project from context to get entities #1137 (pyalex)
• JobService is in crashloop after installing helm chart #1135 (pyalex)
• Fix env var names for jupyter terraform config #1085 (oavdeev)
• Fix java class name validation #1084 (oavdeev)
• Multiple tiny AWS related fixes #1083 (oavdeev)

Merged pull requests:

• Make created_timestamp property optional in KafkaSource #1146 (pyalex)
• In Streaming E2E Test filter kafka consumers by group id prefix #1145 (pyalex)
• Limit concurrency on e2e test runs to 1 #1142 (oavdeev)
• Update prow trigger for AWS #1139 (oavdeev)
• e2e test fixes to make them work on AWS #1132 (oavdeev)
• Add feature table name & job id to deadletter destination #1143 (pyalex)
• Drop hardcoded FEAST_CORE_URL env from JobService helm chart #1136 (pyalex)
• Add Prow to AWS codebuild trigger #1133 (oavdeev)
• Optional IngestionJob parameters passed by Spark Launcher #1130 (pyalex)
• Wait for job to be ready before cancelling #1126 (khorshuheng)
• Ability to run e2e tests in non-default project #1125 (pyalex)
• Ensure job is completed when ingesting to BQ #1123 (terryyylim)
• Add end-to-end Prow Job launcher for AWS tests #1118 (woop)
• Add confluent kafka installation to minimal notebook #1116 (woop)
• Scaffolding for integration tests #1113 (khorshuheng)
• Add serving integration test for updated feature type #1112 (terryyylim)
• In Historical Retrieval from BQ join between source & entities is performed inside BQ #1110 (pyalex)
• Make created timestamp column optional #1108 (khorshuheng)
• Add validation when feature type is changed #1102 (terryyylim)
• E2E flow in prow is working #1101 (pyalex)
• Return e2e back to prow runner #1100 (pyalex)
• Add gh workflow for dockerhub #1098 (terryyylim)
• Make demo notebook work on AWS #1097 (oavdeev)
• Update Kubernetes setup for Feast 0.8 #1096 (khorshuheng)
• Refactored end-to-end tests fully orchestrated by pytest #1094 (pyalex)
• D...

Feast v0.7.1

Full Changelog

Fixed bugs:

• Provide stable jobName in RowMetrics labels #1028 (pyalex)