CompilerSutraPerf episode cover

Why this lesson exists

When you add a custom metric or experiment to csperf you risk turning the evidence into fan‑fiction. This episode shows how to extend csperf safely while keeping the artifact honest.

Recap — where we are in the series

The misconception

Many developers think adding a plugin is just a one‑liner and that the resulting artifact will still be trustworthy. In reality, a poorly‑designed plugin can silently alter the timing, drop metadata, or introduce nondeterminism.

What problem csperf solves (this episode's slice)

csperf gives you:

  1. A plugin API that enforces a contract for metadata and timing.
  2. Built‑in checks that any plugin must pass before its data is merged into the artifact.
  3. A clear separation between experiment definition and artifact generation.

Mental model

plugin ──► csperf core ──► artifact │ │ │ │ └─► validation ──► reject if broken │ └─► user‑defined metrics 

The core never trusts the plugin; it validates the output structure, timestamps, and that the plugin did not modify the baseline run.

Lab: install and first commands

  1. Install the latest csperf:
 pip install --upgrade compilersutra-perf 
  1. Verify the installation:
 csperf doctor 

The doctor command will print the compiler version, machine metadata, and any known plugin issues.

Lab: what we ran on this machine

We attempted to add a simple --time-alloc experiment that counts memory allocations. The plugin was placed in plugins/time_alloc.py.

csperf list-experiments 

The output lists the built‑in experiments plus time_alloc.

csperf run --experiment time_alloc matrix_traversal.cpp 

The run failed with:

ERROR: Plugin 'time_alloc' failed validation: missing 'execution_time_ms' field. 

The doctor command explains that the plugin must return a JSON with execution_time_ms and metadata.

Results (real numbers only)

Trial wall times from this csperf run

Graph: trial wall_time_ms from this run's artifacts (not invented).

Artifacts missing for this episode — numbers withheld.

How to read the artifacts

When a plugin passes validation, its output is merged into the timing_summary field of the artifact. The artifact will contain:

  • execution_time_ms – wall‑time of the experiment.
  • metadata – any user‑supplied key/value pairs.
  • plugin_version – the version of the plugin used.

You can inspect the artifact with:

csperf visualize artifact.json 

The visualization will show the baseline timing alongside the plugin‑added metric.

Common mistakes (teacher checklist)

  1. Skipping validation – always run csperf doctor after adding a plugin.
  2. Missing required fields – ensure execution_time_ms and metadata exist.
  3. Hard‑coding timestamps – let csperf generate timestamps.
  4. Altering baseline artifacts – plugins must not modify the original run data.

Try this next (homework)

Do this tonight — Episode 29 starts by assuming you did.

Closing

You now know how to extend csperf with plugins while preserving the honesty of your performance evidence. Next, we’ll show you how to teach this lab to teammates in a single hour.

The series so far

References