Model Monitoring in SageWorks

SageWorks supports the creation, deployment, and monitoring of your Models inside your AWS account.

In particular SageWorks includes setup of model monitoring for both models created in SageWorks and even models created outside of SageWorks.

Types of Model Monitoring

Data Quality Monitoring

This type of monitoring checks the quality of the data being fed to the model. The primary focus is on data quality by analyzing a 'baseline' of representative data to create a set of constraints and statistics that represent the expected data profile.

Model Quality Monitoring

To monitor the quality of the model (like accuracy, precision, recall), you would typically capture inference results and ground truth labels. Then, you would create a baseline that includes statistical analysis of these inference results.

Bias Monitoring

For bias monitoring, you would focus on specific aspects of your data that could lead to biased predictions. This might involve capturing additional data and setting up specific types of analysis in your baseline creation.

Feature Attribution Monitoring

This involves monitoring the model's feature attributions to understand which features are most influential in the model's predictions.