Part 5: Observability, Evaluation, and Cost | Garage Education

Chapter 19: Step logging

Sat, 10 Oct 2026 00:00:00 +0000

Write one JSON line per model call and per tool call. Each line holds the run id, step number, kind, tool name, arguments, token counts, duration, and stop reason. A failure is easy to find when you search by run id.

Figure 19.1. Both kinds of call write to the same log. Because every line carries the run id, one search returns the whole run in step order.

An illustrative pair of lines, one per kind. The field names are an example; the chapter fixes the content, not the spelling.

{"runId":"run-42","step":1,"kind":"model","tool":null,"args":null,"tokensIn":812,"tokensOut":41,"durationMs":930,"stopReason":null}
{"runId":"run-42","step":1,"kind":"tool","tool":"countErrors","args":{"service":"checkout","minutes":15},"tokensIn":0,"tokensOut":0,"durationMs":4,"stopReason":null}

Chapter 20: A small eval set

Sat, 10 Oct 2026 00:00:00 +0000

An eval case has fixed input and expected properties. Check the stop reason, whether an alert posted, and required phrases. Do not check exact text, because model output varies.

Figure 20.1. One eval case checks properties, never exact text. A case can pass with different wording on every run.

Write these cases first: a duplicate that should be suppressed, a critical error that needs approval, a flood that hits the rate limit, a novel error with no runbook, and an empty window that ends with no model call.

Case	What it proves
`duplicate-suppressed`	A duplicate is suppressed
`critical-needs-approval`	A critical error waits for approval
`flood-hits-rate-limit`	A flood hits the rate limit
novel error, no runbook	The watcher copes with a novel error that has no runbook
empty window	An empty window ends with no model call

Table 20.1. The first five cases. The last two have no case name in the code yet.

@ParameterizedTest
@ValueSource(strings = {"duplicate-suppressed", "critical-needs-approval", "flood-hits-rate-limit"})
void evalCase(String caseName) throws Exception {
 // read the case JSON, run the watcher, assert the expected properties
}

Run the eval before every change. Use plain JUnit 5.

Chapter 21: Failure drills

Sat, 10 Oct 2026 00:00:00 +0000

Break one part on purpose and watch the result. Remove a skill description. Return a large tool output. Remove the step cap. Loosen the fingerprint. Remove the citation check. Record the eval score before and after each drill, then revert.

Drill	What it breaks	Built in
Remove a skill description	Skill selection and the model’s choice	,
Return a large tool output	A small parent context
Remove the step cap	The `STEP_LIMIT` stop
Loosen the fingerprint	Duplicate suppression and the triage cache key	,
Remove the citation check	Cited answers

Table 21.1. Each drill undoes one earlier chapter. The middle column is a teaching guide; the eval score is the real measure.

Figure 21.1. One drill at a time, always reverted, so each score change has one cause.

Chapter 22: Cost optimization

Sat, 10 Oct 2026 00:00:00 +0000

Measure tokens per run first. Then add a cache with Spring Cache and Caffeine, keyed by error fingerprint, with a ten-minute expiry. Clear it when a runbook changes. Redis can replace Caffeine later with no code change.

@Cacheable(cacheNames = "triage", key = "#fingerprint", unless = "#result == null")
public TriageDecision triage(String fingerprint, String stackTrace) { ... }

Figure 22.1. A repeat fingerprint within ten minutes costs no tokens. A null result is never cached, so a failed triage is retried.

Provider prompt caching does not apply to LM Studio, so skip that step. Each step must lower the token count in the eval run. Remove any step that does not.

Figure 22.2. Every cost step has to show its saving in the eval run.