Question 20
Domain 6: Evaluation and MonitoringYou want to compare a new model-serving configuration against production using real traffic while limiting risk. Which approach is best?
Correct answer: B
Explanation
A canary or shadow rollout lets you expose the new configuration to a small slice of real traffic or mirror production requests while limiting blast radius. This supports safe comparison of live metrics against production before full deployment.
Why each option is right or wrong
A. Send all traffic to the new version and compare later
Full cutover removes the safety buffer and increases impact if the new configuration underperforms.
B. Run a controlled canary or shadow rollout and compare live metrics
A controlled canary or shadow rollout is the standard low-risk way to validate a new serving configuration against production traffic: in a canary, only a small percentage of live requests is routed to the new version, while in a shadow deployment the production request stream is mirrored without affecting user responses. There is no fixed statutory timeframe or fee here; the relevant operational condition is that the rollout remains limited and metrics such as latency, error rate, and throughput are compared against the baseline before any broader cutover.
C. Wait until the offline score is perfect before deploying anything
Offline metrics help, but they cannot fully predict real-world production behavior.
D. Rebuild the vector index first even if retrieval was unchanged
Vector index rebuilding is unrelated when the retrieval setup has not changed.