Courses
Agent Harness with Spring AI
Part 5: Observability, Evaluation, and Cost
20. A small eval set

Chapter 20: A small eval set

An eval case has fixed input and expected properties. Check the stop reason, whether an alert posted, and required phrases. Do not check exact text, because model output varies.

Figure 20.1. One eval case checks properties, never exact text. A case can pass with different wording on every run.

Write these cases first: a duplicate that should be suppressed, a critical error that needs approval, a flood that hits the rate limit, a novel error with no runbook, and an empty window that ends with no model call.

CaseWhat it proves
duplicate-suppressedA duplicate is suppressed
critical-needs-approvalA critical error waits for approval
flood-hits-rate-limitA flood hits the rate limit
novel error, no runbookThe watcher copes with a novel error that has no runbook
empty windowAn empty window ends with no model call

Table 20.1. The first five cases. The last two have no case name in the code yet.

@ParameterizedTest
@ValueSource(strings = {"duplicate-suppressed", "critical-needs-approval", "flood-hits-rate-limit"})
void evalCase(String caseName) throws Exception {
    // read the case JSON, run the watcher, assert the expected properties
}

Run the eval before every change. Use plain JUnit 5.

docs