It never answers twice the same. Ship it anyway.

You stop testing for an exact output and start testing for correct behaviour at an acceptable rate. Batch evaluation runs a set of real utterances and scores whether the agent selected the expected topic and the expected action — those are checkable even when the wording varies. Guardrail cases are stricter and binary: a refusal either held or it did not. Response wording itself is assessed by sampling and human review rather than string matching, because exact-match assertions on generated text produce a suite that fails constantly and teaches everyone to ignore it.

Batch testing agents at scale. You supply a set of test cases — typically an utterance plus the expected topic and expected actions — and it runs them together and reports actual against expected, so you can evaluate many scenarios in one pass rather than typing questions by hand. It is the mechanism that makes the evaluation layer practical. What it does not supply is the test set itself, the judgement about what good looks like, or the discipline of running it before every promotion; those are the parts that determine whether it is useful.

Largely yes — agent definitions, topics, actions, Apex, Flows and prompt templates deploy as metadata through the same tooling as the rest of your Salesforce work, which is what makes source control and rollback possible. The parts that do not travel are environment-specific: indexed content has to be seeded and re-indexed per environment, credentials and endpoints differ, and channel deployments are usually configured per org. Getting those substitutions handled cleanly is most of what building the pipeline involves.

Big enough that a few cases flipping does not change the verdict, and small enough that you will actually run it. In practice that means covering every live topic with several phrasings each, including the boundary cases between adjacent topics, plus the full guardrail suite and everything that has broken in production before. Start smaller than feels rigorous and grow it from real failures — a set built entirely from imagination is both larger and less useful than one grown from what actually went wrong.

Yes. Running an agent consumes AI capacity whether it happens in a sandbox or in production, so evaluation runs draw on the same budget. That has a real design consequence: you cannot run a large batch evaluation on every commit the way you would run Apex tests. We structure the layers so the cheap deterministic tests run constantly and the expensive evaluation runs at meaningful gates — before a promotion, after a topic change, on a scheduled cycle — and we put a number on that cadence during planning rather than letting it surprise someone.

Carefully, and it is exactly why we rehearse it. Reverting the metadata is straightforward; the questions that need answering in advance are what happens to sessions already in progress, whether any action the new version took needs compensating, and whether reverting the agent also requires reverting a Flow or a permission change that shipped with it. Staged exposure limits the blast radius so the answer usually affects a handful of conversations rather than a day's traffic — which is the real argument for staging.