Golden Set

A golden set is a small, fixed collection of real tasks — usually twenty to fifty — where someone has already reviewed the output and confirmed it is good enough to send. The set does not change between runs, and that fixity is the entire point. When you edit a prompt, switch Claude models, or add documents to a Claude Projects knowledge base, you re-run the same tasks and place the new outputs next to the approved ones. This is easy to confuse with spot-checking, but the two answer different questions. Spot-checking asks whether today's output is acceptable. A golden set asks whether today's system is better or worse than last week's. It is also not a test suite in the engineering sense, because the correct answer is a judgment rather than an exact string match — two differently worded meeting summaries can both be right.

When you edit a prompt, you watch the one problem you were trying to fix. You do not notice that two of the ten cases that used to work quietly broke. Model versions get updated, a task gets moved from one model to another, someone adds files to the knowledge base, a scheduled task's trigger condition gets nudged — all of these change output behaviour, and almost no office keeps a record of what the previous version looked like. Quality discussions then collapse into competing impressions: one person says it has clearly improved, another says it has been strange lately, and neither can produce anything to compare. A golden set exists for exactly one reason: to turn 'what the previous version looked like' from memory into a file you can pull up, so the effect of a change becomes a visible difference rather than an argument.

Start with selection. The bulk of the set should be the ordinary tasks you run every week — the boring ones nobody wants to discuss — plus a handful of edge cases you already know cause trouble: a scanned attachment, an email mixing two languages, source data with numbers that contradict each other. Store each input paired with its approved output, in a spreadsheet or a Claude Projects knowledge base; what matters is that anyone can retrieve it. To run a check, use batch processing to push the whole set through at once, then compare each item on three things: are the facts wrong, are the required fields present, does the tone meet your external standard. The passing bar is not perfection — it is that the new version must not be worse than the last. Two forms exist in practice. Strict comparison suits tasks with fixed output shapes such as fields, figures and dates. Rubric scoring suits open-ended writing, where a person grades each item one to five and you watch the total move. For upkeep, retire a few items each quarter once they always pass, and add whatever new failure you just hit.

Building a golden set costs roughly half a day: assemble twenty to thirty inputs, paste in the approved outputs, write down the comparison criteria. After that, each re-run takes maybe fifteen minutes. Whether it is worth it comes down to two conditions: you run this same kind of task more than a few times a week, and the output goes straight into a client inbox, an executive deck, or an external document. If both hold, half a day buys you the ability to find out immediately when a change breaks something, rather than three weeks later when a client points it out. A one-off task does not need this — there is no next run to compare against. The real risk to watch is set rot. Over time the set fills up with problems you fixed long ago, every run passes at one hundred percent, and it feels reassuring while measuring nothing. The test is simple: if two consecutive re-runs produce zero failures, that is not a stable system, it is a set that needs refreshing.