12. Measuring performance, part 1: the IC developer

Originally published at 4thwithme.dev.

Performance review is a hard problem. Every company does it in a different way. So let's look at the approaches that exist. For each one, we ask what it measures. This post is about the individual developer.

The approaches fall into three layers. Each layer measures something different.

[ THREE LAYERS ]

only the bottom layer is about one person

1. Team system. How fast and how safely does this service ship? DORA and DX Core 4 live here.
2. Experience. What slows people down? How do they feel about it? SPACE and surveys live here.
3. Individual. Does this person work at their level? A ladder, a rubric and evidence live here.

The top two layers show where the bottlenecks are. Only the bottom layer belongs in a review. Let's go from top to bottom.

[ DORA: THE DELIVERY PIPELINE ]

DORA is a long research program from Google. It studies how teams ship software. Most people know its "four keys". The current guide has five metrics in two groups.

MTTR got a new name. Rework rate is new

The main finding is simple. Speed and stability do not fight each other. Top teams do well on all five metrics. Weak teams do badly on all five. Dave Farley puts it well. The real choice is "better software faster" or "worse software slower".

DORA has rules. Most teams skip them:

• Measure one app or service at a time. A mobile app is not a mainframe job.
• Do not use one metric for everything. Pick a few that pull against each other.
• Do not let teams compete on the numbers. Each team competes with its own past.
• Do not make the metric the goal. Forced daily deploys give you empty deploys.

That last rule is Goodhart's law. A number stops working once you chase it. You will see this idea in every section below.

[ SPACE: FIVE LENSES, NOT FIVE METRICS ]

SPACE came from Microsoft Research in 2021. Its authors include Nicole Forsgren and Margaret-Anne Storey. SPACE is not a list of metrics. It is a list of five things to look at. It stops you from looking only at the easy one.

DEF.1

Satisfaction and well-being.
Performance: does the software work?
Activity: counts of builds, PRs and releases.
Communication and collaboration.
Efficiency and flow.

Rule: look at two or three dimensions together. When they disagree, pay attention.
SPACE in one box

Most teams already measure activity, because it is free. It is also the weakest sign of productivity. Say activity is high but people are unhappy. That team is not productive. It runs at a pace it cannot keep.

The SPACE authors suggest a written rule. These metrics are for the team, not for rating people. Write it down before the first dashboard goes live. After that, nobody will believe you.

[ DX CORE 4: ONE SCORECARD ]

DX Core 4 is the newest framework. Abi Noda and Laura Tacho lead it. It folds DORA, SPACE and DevEx into four dimensions. Each dimension has one key metric.

Core 4 holds DORA inside it. Impact is the new part

Look at the first row. Diffs per engineer is a key metric. Meta, Microsoft and Uber use it. But DX sets three conditions. First, balance it with the experience survey. Second, attach no targets and no rewards to it. Third, roll it out with care. The framework says "not at individual level". Remember that sentence above all others.

Here are the three frameworks side by side:

My advice is simple. No delivery data at all? Start with DORA. Need one page for a CFO and a team lead? Use Core 4. Use SPACE as a checklist. It reminds you about happiness and flow.

[ SURVEYS: DATA FROM HUMANS ]

You may know this story. A boss asks for productivity metrics. A group picks lead time, deploy rate and PRs per engineer. They build dashboards. A leader sees "2-day lead time, low performer". Then the leader asks: is anything actually wrong? Nobody knows.

Abi Noda and Tim Cochran have an answer. Measure with data from people first. There are two kinds of questions:

• Attitude. Feelings and opinions. "How happy are you with your build tools, 1 to 10?"
• Behavior. Facts about the work. "How long does a deploy to production take?"

Few teams ask the second kind. Yet it is not new. DORA's own benchmarks came from surveys. People picked answers like "under one hour" or "over six months". So "surveys are not rigorous" is a weak argument.

Surveys see what logs cannot. Google tested 117 possible tech-debt metrics from system data. Not one of them worked, alone or combined. Tech debt is a gap between your code and your ideal code. Only a human can see that gap.

ask first, then measure, then track both

These are the rules I follow now:

• One question asks one thing. Keep the words the same each time. Otherwise you cannot compare.
• Split the results. Group by team, role and time at the company. An average can hide five unhappy mobile developers.
• Close the loop. Long surveys do not tire people out. Surveys with no follow-up do. Share the results and one action.
• Sample people. Peloton surveys a random half of developers each time. Each person answers once a year. The data still holds up.

The same warning applies here. These surveys describe friction, not people. Say people think answers affect their rating. Then they stop being honest. You lose your best data.

[ THE INDIVIDUAL: LADDER FIRST ]

Now the bottom layer. Every source agrees on one point. Without a written career ladder, a review is just an opinion. Will Larson says it plainly. Invest in one thing only? Invest in the ladder.

A ladder is a grid. Skills go on one axis. Levels go on the other. Each cell says what "good" means at that level. CircleCI needed about 8 months to build theirs. The main reason: nobody owned it full time. They cut about 50 skills down to 27.

The best idea is on the level axis. Higher levels mean bigger scope, not more output.

E1 to E3: do it well. E4 to E6: help others do it well

Dropbox's framework writes each point in the first person. One example: "I look for ways to reduce future toil." Their 2022 update fixed a common bias. The old ladder rewarded big, visible projects. The new one also gives credit for on-call, upkeep, docs and glue work.

Dropbox and Larson add two warnings:

• The ladder says "this is not a checklist". People will use it as one anyway. Write it with that in mind.
• Never move the goalposts during a cycle. Show changes early. Announce them. Give people time before you rate against new text.

[ TWO WAYS TO SCORE A PERSON ]

A ladder says what good looks like. You still need a way to score against it. I found two concrete methods. They work well together.

Method 1: the skill matrix

Philipp Hauer's matrix has 11 skills in three groups. They are coding, behavior, and responsibility. Each skill gets a score of 1, 2 or 3. One means "needs work". Three means "doing very well".

The trick is in the process. The manager scores the whole team in one sitting. That keeps the bar equal for everyone. Each engineer also scores themselves. In the review, you compare the two columns.

the totals say nothing. the gaps say everything

Hauer says not to trust the total. The chart shows why. Scores of 26 and 27 look like agreement. But three rows differ. Those three rows are the real review.

Method 2: weekly points

Vardan Torosyan's framework goes further. It turns performance into a sum of four parts:

performance = w₁·input + w₂·output + w₃·outcome + w₄·impact 

Input is effort. Output is what you deliver. Outcome is whether it worked. Impact is the long-term effect on the business. The weights (w) change with the situation. A fast startup phase cares most about output. A strategic phase cares most about impact.

You score events every week. Here are some examples from the framework:

A cycle is six months, or about 24 weeks. These are the bands:

made-up data. the point is when you see the stall

By my math, the top band needs 9 to 11 points a week. But the final number is not the point. The running total is an early warning. In the chart, engineer B stalls at week 6. A 1:1 catches it at week 8. With a yearly review, you find out in month 12.

I would not use this method for everyone as written. The framework counts merged PRs. DX warns against exactly that. Still, it works as a personal tracking tool. The engineer and manager agree on it together. Then it is honest and open.

[ GOALS: CONTRIBUTE, DON'T COUNT ]

Rob Zuber is the CTO of CircleCI. Zuber writes on LeadDev that delivery metrics rarely motivate anyone. A CTO saying "deploy more often" changes nothing. Now give a team a business number to own. It will ask for delivery data on its own. It wants to know why it missed.

So the order is:

1. Set a team goal. Link it to a company goal.
2. Pick a business number the team can own. Think sales, cost or retention. Not deploy count.
3. Use delivery metrics to debug when the business number slips.

For one person, I turn this into two goals per cycle:

• A contribution goal. "Own the checkout speed work. It supports the team's sales goal." Not "ship 40 PRs".
• A growth goal from the ladder. Pick one skill from the next level. Example: "lead one design review across two teams".

Add one habit: the brag document from Julia Evans. It is a running list of your work. It also says why each item mattered. Write each win as before and after. Don't write "I shipped X". Write "I shipped X, and page load fell to 300 ms".

Gergely Orosz's self-review builds on this. It has four parts. First, your goals. Second, your wins, with numbers. Third, how you worked with others. Fourth, how you compare to the ladder.

Your manager does not know everything you did. They also remember last month best. A brag doc fixes both problems.

[ THE REVIEW ITSELF ]

All this measuring only matters if the review talk goes well. I use Orosz's rules for a fair review:

• No surprises, good or bad. Give feedback in 1:1s all year. The review only writes it down.
• Use evidence from real work. Code, design docs, code reviews, incident notes and thank-you messages.
• Rate each skill against the rubric. Then compare reviews side by side. Look for bias.
• Send the written review at least one day early.
• Keep money out of the review meeting. Once a number appears, people stop hearing feedback.
• Spend half the time on the past. Spend the other half on the future.

[ THE SMALL PRINT ]

This post sums up about 30 sources. Where I give my own view, I say so. The two engineers in FIG.6 and FIG.7 are made up. Some numbers come from DX's own customers. One example is "3 to 12% better efficiency". Treat those as marketing, not research. None of these frameworks was tested on your team. Start small. Tell people how you will use the data. Keep team metrics off the review form.

[ ELSEWHERE ]

GitHub · LinkedIn · Dev.to · Substack · 4thwithme.dev/blog

May the --force be with you. See you next week.