Question 35
Domain 4: ML Solution Monitoring, Maintenance, and SecurityA company deployed an ML model that uses the XGBoost algorithm to predict product failures. The model is hosted on an Amazon SageMaker endpoint and is trained on normal operating data. An AWS Lambda function provides the predictions to the company's application. An ML engineer must implement a solution that uses incoming live data to detect decreased model accuracy over time. Which solution will meet these requirements?
Correct answer: C
Explanation
SageMaker Model Monitor is designed to "detect drift" by comparing incoming live data to a baseline from training data statistics and constraints. Because the model is trained on normal operating data and served through a SageMaker endpoint, a scheduled monitoring job can track changes in input distribution and signal decreased accuracy over time.
Why each option is right or wrong
A. Use Amazon CloudWatch to create a dashboard that monitors real-time inference data and model predictions. Use the dashboard to detect drift.
CloudWatch visualizes metrics and logs, but it does not natively perform model drift analysis.
B. Modify the Lambda function to calculate model drift by using real-time inference data and model predictions. Program the Lambda function to send alerts.
Lambda can run custom code, but hand-building drift detection is not the managed SageMaker monitoring solution.
C. Schedule a monitoring job in SageMaker Model Monitor. Use the job to detect drift by analyzing the live data against a baseline of the training data statistics and constraints.
Amazon SageMaker Model Monitor is the service that continuously evaluates endpoint data for drift by comparing live inference payloads to a baseline captured from the training set, including statistics and constraints generated from that data. Under the SageMaker Model Monitor workflow, a scheduled monitoring job can run at a defined interval against the endpoint’s incoming data and flag deviations that indicate degraded model performance over time, which fits a model hosted on a SageMaker endpoint and invoked by Lambda.
D. Schedule a monitoring job in SageMaker Debugger. Use the job to detect drift by analyzing the live data against a baseline of the training data statistics and constraints.
SageMaker Debugger focuses on training and debugging behavior, not production data drift monitoring.