You had an evaluation set.
Fifty questions,
each with a correct answer.
The old prompt
scored seventy eight percent.
You rewrote the instructions
and the new one
scored eighty two.
Four points better.
You wrote it up,
shipped it,
and moved on.
Look at what four points means
on fifty questions.
Each question is worth two points.
Four points is two questions.
Two questions
out of fifty
went the other way.
That is not a result.
That is a coin
landing differently twice.
Here is the arithmetic.
At around eighty percent,
the standard error
on fifty questions
is close to six points.
A rough ninety five percent range
is plus or minus eleven.
Your true score
could be anywhere
from about seventy
to about ninety.
Both prompts
fit easily inside that.
And the model
is not perfectly repeatable either.
Run the old prompt again
and it may move
a question or two by itself.
Same prompt.
Same questions.
A different afternoon.
So do not compare totals.
Compare question by question.
How many did the new prompt fix?
How many did it break?
Fixed seven
and broke five
is not the same change
as fixed two
and broke none,
even though both
add up to plus two.
Read the five it broke.
They are often
the important ones.
Then make the set bigger.
To see a few points of difference
with any confidence,
you need hundreds of questions,
not dozens.
Run each version
more than once,
and look at how much
a version disagrees with itself
before you trust
how much it disagrees with another.
Keep a note of the noise.
If two runs of the same prompt
differ by three points,
a three point improvement
means nothing.
A score is a measurement.
Every measurement
comes with an error bar,
whether you draw it or not.
If you never draw it,
you will spend months
chasing changes
that were never there.
– Serguey Asael Shinder
