Question 24
Domain 3: Application DevelopmentAfter changing the system prompt to be more concise, which rollout plan is strongest?
Correct answer: B
Explanation
A concise system prompt can change model behavior, so the safest rollout is to verify it first with offline regression tests and then expose it to a small canary slice. Monitoring canary traffic for “quality, latency, and cost” checks both output impact and operational impact before full deployment.
Why each option is right or wrong
A. Deploy globally and wait for complaints
Global rollout skips controlled validation and turns users into the test signal.
B. Run offline regression tests, then monitor canary traffic for quality, latency, and cost
A prompt change is a production configuration change, so the safest deployment pattern is staged validation: first run offline regression against a fixed evaluation set to detect behavior drift before any users are exposed, then release to a small canary slice and watch live metrics. The canary should explicitly track quality, latency, and cost because a shorter system prompt can alter response quality and token usage, and can also shift inference latency and spend before broader rollout.
C. Only verify that JSON still parses in one local example
One local parse check verifies syntax only, not response quality, latency, or spend impact.
D. Re-embed all documents before release
Prompt wording changes generation behavior; embeddings usually matter only if retrieval content changes.