Offline evaluation narrows risk, but production traffic still reveals differences in request shape, tool behavior, provider latency, and cost. A canary release limits exposure while collecting that evidence.
Version every behavioral input
Treat prompts, model policy, tool definitions, retrieval configuration, and application code as one release. A canary is only interpretable when its boundaries are clear.
Route representative traffic
Start with a small percentage and preserve important segments. A random sample that excludes long-running or high-value workflows may report a healthy release that has not faced the real workload.
Compare a balanced scorecard
Review completion rate, validation outcomes, latency, errors, tool retries, token usage, and cost per successful run. No single metric captures the release tradeoff.
Graduate or roll back automatically
Set promotion thresholds and stop conditions before rollout. If the canary breaches an objective, restore the stable version and retain its traces for investigation.