> ## Content Index
> Fetch the complete content index at: https://vanta-plus.planethemes.com/llms.txt
> Use this file to discover other available public pages before exploring further.

# Evals before prompts
- URL: https://vanta-plus.planethemes.com/evals-before-prompts/
- Published: 2026-09-10T09:00:00.000Z
- Updated: 2026-09-10T09:00:00.000Z
- Description: Tweaking a prompt without a test set is redecorating in the dark. Write twenty examples first.
- Author: Elena Ruiz
- Tags: AI, #Import 2026-10-23 17:46

The fastest way to improve an AI feature is not a better prompt. It's a list of twenty real inputs and what a good answer looks like for each.

## Start with twenty

Collect them from support tickets, logs and your own use. Include the weird ones. Especially the weird ones.

## Grade by hand, then automate

Read every output yourself the first few times. Once you can describe what 'good' means in a sentence, turn that sentence into a check.

```python
for case in cases:
    out = run(case.input)
    assert case.must_include in out, case.id
```

## Keep the set alive

Every bug report becomes a new case. After a year you have the most valuable asset in the project, and it isn't the prompt.