Why Vibes Are Not Enough: The Case for Prompt Evaluation

If you have done the basics of prompting, you already know how to write a clear instruction, add context, give examples, and ask for a specific format. So why do your prompts still feel unreliable? Because most people improve prompts by vibes: they run a prompt once, eyeball the answer, tweak a word, run it again, and stop when one output looks good. That feels productive, but it is guessing. The output that looked great might fail on the next ten inputs you never tried.

This course is about replacing guessing with measurement. Advanced prompt engineering is less about knowing more tricks and more about building a feedback loop that tells you, objectively, whether a change made your prompt better or worse. That loop is called an evaluation, or eval for short.

What You'll Learn

• Why single-example testing gives you a false sense of confidence
• What an eval is and the three pieces every eval needs
• The difference between subjective "looks good" judgments and objective scoring
• How professional teams treat prompts like code they test, not text they tweak
• A simple mental model you will use for the rest of this course

The Trap of the Single Example

Imagine you write a prompt that turns customer emails into a short summary plus a priority label. You test it on one email, it nails the summary, labels it "High," and you ship it. The next day it labels an angry refund demand as "Low" and routes it to the wrong queue.

What happened? Your one test email happened to be easy. Real inputs vary: some are long, some are sarcastic, some are in mixed languages, some are blank. A prompt that works on one example tells you almost nothing about how it performs across the distribution of inputs it will actually see.

The core problem with vibe-based prompting:

• You test on inputs that are too easy. You naturally pick clean examples.
• You can't compare versions. Was v2 actually better than v1, or did you just get a luckier output?
• You can't catch regressions. Fixing one case often breaks another, and you never notice.
• You can't hand it off. "It feels good" is not something a teammate can verify.

What an Eval Actually Is

An eval is a repeatable test for a prompt. At its simplest, an eval has three parts:

1. A set of test cases — a collection of realistic inputs, ideally including the hard and weird ones, not just the easy ones.
2. A way to score each output — a rule, a checklist, or a judge that says how good each output is. This can be exact-match, a rubric score, or a pass/fail check.
3. An aggregate number — one score that summarizes performance across all cases, such as "84% pass" or "average 4.1 out of 5."

Once you have those three pieces, prompt engineering changes completely. You stop asking "does this output look good?" and start asking "did my score go up?" You can change one word, re-run the whole eval, and see the number move. That is the entire game.

From Subjective to Objective

There is a spectrum from purely subjective to fully objective scoring, and where you land depends on the task:

• Exact / programmatic checks (most objective): The output must equal a known answer, contain a required field, be valid JSON, or stay under a word limit. A script decides pass or fail. No opinion involved.
• Rubric scoring (structured judgment): You score against named criteria such as accuracy, completeness, and tone, each on a defined scale. The judgment is still human-like, but it is consistent because the rubric is fixed.
• Pairwise preference (relative judgment): You don't score one output in isolation; you ask which of two outputs is better. This is often more reliable than absolute scores because comparing is easier than grading.

You will use all three in this course. The skill is matching the method to the task. Extracting a date from an invoice? Use an exact check. Judging whether a summary is well written? Use a rubric or a pairwise comparison.

Prompts Are Software, So Test Them Like Software

Here is the mindset shift that separates advanced practitioners from everyone else: a prompt is a program written in English, and the model is its runtime. You would never ship code without tests and then change it by feel. A prompt deserves the same discipline because it has the same risk: small changes cause invisible breakage.

This does not mean you need to be an engineer or write code. Throughout this course you will build evals you can run by hand in a chat window, in a spreadsheet, or with simple tooling. The principle is what matters, not the tooling: define what good looks like, test against real cases, and let the score tell you the truth.

A Quick Self-Check

Try this thought experiment with a prompt you already use at work.

Most people fail questions 2 and 3 the first time. That is exactly the gap this course closes.

Key Takeaways

• Improving prompts by eyeballing single outputs is guessing, not engineering.
• An eval has three parts: test cases, a scoring method, and an aggregate score.
• Scoring runs from objective (exact checks) to structured judgment (rubrics) to relative judgment (pairwise).
• Treat a prompt like software: define "good," test against realistic cases, and trust the number.
• For the rest of this course, every optimization you make will be judged by whether it moves a measured score.