The most expensive webhook bug I know is boring. Your order handler receives order.paid, stores the event, creates the fulfillment, sends the confirmation email, and then takes a little too long to answer. WooCommerce times out and posts the same event again. Your handler sees a new delivery id, decides this is a new event, and ships the order twice.

A coding agent asked to "make the webhook idempotent" will often write exactly the wrong fix: store X-WC-Webhook-Delivery-ID and skip repeats. That header is a fresh log id for every attempt, so the fix does nothing for this case.

That gap, between what a capable generalist writes and what someone who has debugged a store at 2 a.m. writes, is what I tried to close with commerce-agent-skills. It is a library of 29 skills and matching rules for coding agents that build e-commerce and marketplace products: WooCommerce, Shopify, Stripe, Iranian payment gateways, inventory, returns, tax, feeds, search, accessibility, and more.

This post is not a tour of the catalog. It is about the engineering around it: how a skill is structured, how I test that it teaches what it claims to teach, how the validator keeps 29 skills consistent, and how the installer gets them into a project. I will use the webhook redelivery case throughout, because it is the clearest example of why this needs tests at all.

What a skill is

The skills follow the Agent Skills format: a folder with a SKILL.md that has name and description frontmatter, plus optional references and scripts. An agent reads the description to decide whether a skill applies, reads the body when it does, and opens references and scripts only when the body points to them.

The webhook skill looks like this on disk:

skills/order-and-webhook-reliability/ SKILL.md references/woocommerce-webhooks.md scripts/verify-hmac-signature.mjs evals/tasks.json evals/examples/verify-raw-body.md evals/examples/dedupe-resource-version.md evals/examples/redelivery-after-timeout.md rules/order-and-webhook-reliability.md

And the frontmatter:

name: order-and-webhook-reliability description: "Accept order and payment webhooks with a raw-body signature check, durable deduplication, and reconciliation against the source of truth. Use when handling order events, payment notifications, or outbound webhook delivery." license: MIT compatibility: "The signature checker requires Node.js 20 or newer. No network access." metadata: author: Naser Sadrmanesh version: "0.3.2"

Every skill also has a rule: a short plain-Markdown file of constraints for agents that keep always-on guidance in context. A skill is for a task. A rule is for every task in that area. The webhook rule includes this line:

- Treat a redelivery after a timeout as a duplicate: if the inbox row is already `applied`, return 2xx and run no side effect again.

Write for the failure, not the happy path

The skill body is a procedure, then the details, then Gotchas, Verification, and Evaluation sections. The part about redelivery is written around the failure, because the happy path is what the agent already knows:

The most common duplicate is a redelivery after a timeout. The handler stored and applied the event, but the sender timed out before the 2xx arrived and posts the same event again, often with a new delivery id. The retry must find the inbox row already applied, return 2xx, and run no side effect: no second fulfillment, no second confirmation email, no second stock decrement. This only holds when each side effect is recorded in the same transaction that marks the row applied, or is keyed on the event id where it runs.

It also gives a usable dedupe key instead of a principle. Use the provider event id when it is stable across retries. Otherwise, hash the topic, the resource id, and the resource's date_modified_gmt. And it records the platform facts an agent cannot infer:

  • WooCommerce counts a delivery as failed when the status is outside 2xx, 301, and 302.
  • After more than five consecutive failures it disables the webhook. The threshold comes from the woocommerce_max_webhook_delivery_failures filter, which defaults to 5.
  • WC_Webhook::deliver() posts with redirects disabled, so a 301 does not mean the JSON reached the next URL.

That last point matters operationally. A handler that keeps returning 500 because of a bug does not just lose events. Five more failures and the store stops sending webhooks at all, which is why the skill says to return a retryable error only when the event was not stored.

Scripts where a check beats a paragraph

Some things are easier to check than to describe. WooCommerce signs base64(hmac_sha256(secret, raw_body)), but it HTML-decodes the secret first with wp_specialchars_decode( $secret, ENT_QUOTES ). A secret containing & or a quote produces a mismatch that looks like a bug in your code. The skill ships a small checker with no dependencies:

export function decodeWebhookSecret(secret) { return secret .replaceAll("&quot;", '"') .replaceAll("&#039;", "'") .replaceAll("&#39;", "'") .replaceAll("&lt;", "<") .replaceAll("&gt;", ">") .replaceAll("&amp;", "&"); } export function woocommerceSignature(secret, rawBody) { const key = decodeWebhookSecret(secret); return createHmac("sha256", key).update(rawBody).digest("base64"); } export function signaturesMatch(expected, received) { if (typeof expected !== "string" || typeof received !== "string") { return false; } const left = Buffer.from(expected); const right = Buffer.from(received); if (left.length === 0 || left.length !== right.length) { return false; } return timingSafeEqual(left, right); } 
WEBHOOK_SECRET="..." node scripts/verify-hmac-signature.mjs \ --body payload.json \ --signature "$SIGNATURE"

It reads the secret from the environment or --secret-file and never prints it. Exit code 0 means the signature matches, 1 means a mismatch, and 2 means a usage error.

Evals: making a skill falsifiable

A skill that "covers idempotency" is not a claim you can check. I wanted every skill to make claims that can fail. Each skill has an evals/tasks.json with at least two tasks, and each task lists text that a correct answer must contain and text it must not contain:

{ "id": "redelivery-after-timeout", "task": "The order.paid handler stored and applied the event and created the fulfillment, but WooCommerce timed out before the 2xx arrived and posted the same event again. Make the retry safe.", "expects": { "must_include": ["applied", "2xx", "side effect", "timeout"], "must_not_include": ["X-WC-Webhook-Delivery-ID"] } }

The must_not_include is the important half. It encodes the specific wrong answer I described at the top. The validator requires at least one negative check per skill, because a test that only looks for good words is easy to pass with a long, vague answer.

Each task also has a worked example under evals/examples/, a short answer that shows what a passing response looks like:

Look up the inbox row by topic, order id, and `date_modified_gmt`. When its status is already `applied`, return 2xx with the original result and run no side effect: no second fulfillment, no second confirmation email, and no second stock decrement.

The scorer is deliberately simple. Every must_include and must_not_include entry is one check:

export function checkAnswer(answer, expects) { const errors = []; const text = String(answer); for (const needle of expects?.must_include ?? []) { if (!text.includes(needle)) { errors.push(`missing required text: ${needle}`); } } for (const needle of expects?.must_not_include ?? []) { if (needle !== "" && text.includes(needle)) { errors.push(`contains forbidden text: ${needle}`); } } return errors; }

Running it

Without arguments, the runner scores every worked example and requires all checks to pass. That is a self-check: it proves the fixtures and examples agree. It runs in CI on every pull request.

$ node scripts/run-evals.mjs --skill order-and-webhook-reliability 100% order-and-webhook-reliability (3/3 tasks) Overall 100%: 10/10 checks, 3/3 tasks passed (examples).

Across the library it reports Overall 100%: 211/211 checks, 60/60 tasks passed (examples).

The more useful mode is scoring real answers. You give your agent the task text, save each answer as answers/<skill>/<task-id>.md, and run the scorer with a threshold. Here is the wrong fix from the top of this post, scored:

$ node scripts/run-evals.mjs --answers ./answers --skill order-and-webhook-reliability --min-score 0.8 60% order-and-webhook-reliability (2/3 tasks) redelivery-after-timeout: missing required text: applied; missing required text: 2xx; missing required text: side effect; contains forbidden text: X-WC-Webhook-Delivery-ID Overall 60%: 6/10 checks, 2/3 tasks passed (answers). $ echo $? 1

The answer file said: "Store each X-WC-Webhook-Delivery-ID in a table and skip the request if the id was seen before. Make the handler faster so the timeout does not happen." It fails on the forbidden header, and it fails on three missing concepts. It exits with status 1 below the threshold, so you can put it in a pipeline that compares agent output with and without the skill installed.

Notice what it did not catch: the answer gets credit for timeout just by mentioning it. That is the honest limit of substring scoring, and I come back to it below.

The validator: consistency at 29 skills

With one skill, you can keep everything straight by reading it. With 29, drift starts on the first release. scripts/validate-skills.mjs runs in CI and before publish, and fails on any of these:

- metadata.version must match package.json exactly - description must be between 100 and 320 characters - SKILL.md must have Gotchas, Verification, and Evaluation sections - a skill with scripts/ must set compatibility - every relative link must resolve and stay inside the skill directory - every file under references/ and scripts/ must be referenced from SKILL.md - every script must pass `node --check` - the rule must exist, be plain Markdown, start with the same heading as SKILL.md, and be 8 to 80 lines - evals: at least two tasks, kebab-case ids, a worked example per task that passes its own checks, and at least one must_not_include per skill - every must_include term must appear in the skill, its rule, or its references - every skill must be in the catalog order and in exactly one website category - no em dashes and no zero-width non-joiners in prose

Three of these deserve a word.

Every eval term must be taught. This is the check I am most glad I added. Without it, it is easy to write an eval that expects date_modified_gmt while the skill only says "use the resource version". Then the eval tests the agent's prior knowledge, not the skill. The check is a case-insensitive search through the skill, rule, and references:

export function untaughtTerms(tasks, taughtText) { const haystack = taughtText.toLowerCase(); const missing = []; for (const task of tasks) { for (const needle of task?.expects?.must_include ?? []) { if (typeof needle === "string" && !haystack.includes(needle.toLowerCase())) { missing.push({ task: task.id, term: needle }); } } } return missing; }

The description length window. The description is what the agent uses to choose a skill, so it has to name both what the skill does and when to use it. Under 100 characters, it is usually missing the "use when". Over 320, it starts to crowd out other skills in the selection step.

Unreferenced files are errors. An agent opens references only when the skill points to them. A reference nobody links to is dead weight that will never be read, so the validator treats it as a bug.

The release workflow ties it together. A vX.Y.Z tag must match package.json, then the validator, evals, and tests run before anything is published. npm test runs 50 tests covering the installer, scorer, validator, and catalog generator.

The installer

The format does not name an install path, and different agents look in different places. So the package ships a small CLI with no dependencies, which copies files and downloads nothing after the package itself:

npx @sadrmanesh/commerce-agent-skills init

That copies every skill to .agents/skills/ and every rule to .agents/rules/. The rest of the surface:

commerce-agent-skills list [--json] commerce-agent-skills init [options] commerce-agent-skills add <skill> [<skill> ...] [options] --target <dir> Project directory to install into --layout <name> agents (.agents/skills and .agents/rules, default) or root (skills/ and rules/) --skills-dir <path> Skills directory, relative to --target --rules-dir <path> Rules directory, relative to --target --only <a,b> init: install only these skills --no-rules Do not copy the matching rules --force Replace skills that are already installed --dry-run Print what would be copied and write nothing 
$ npx @sadrmanesh/commerce-agent-skills add order-and-webhook-reliability --dry-run Would add order-and-webhook-reliability at .agents/skills/order-and-webhook-reliability and .agents/rules/order-and-webhook-reliability.md Planned: 1 of 1 skills. Skills: .agents/skills, rules: .agents/rules

Two decisions about existing files. init is meant to be rerun, so it skips skills that are already installed. add names specific skills, so an existing one is an error unless you pass --force. Both validate every name and build the full plan before writing anything, so a typo in the third name does not leave you with two of three skills installed. Exit codes are 0 for success, 1 for a skill or file error, and 2 for a usage error.

Installed skills are copies, not links. You can edit them for your shop, and you can review them in a pull request like any other file in the repository. The trade-off is that upgrades are explicit: rerun with --force and read the diff.

Where this connects to MCP

One of the skills, mcp-store-tooling, covers exposing a store to an agent through the Model Context Protocol: tool annotations, input and output schemas, scopes, and dry runs with confirmation for risky writes. I wrote it alongside a real server, woocommerce-mcp, where risky writes go through a preview, a confirmation, and an audit log.

The design is in Safe write tools for agents.

Limitations

I want to be precise about what the evals prove.

  • Scoring is substring matching. It checks that the right concepts appear and that known wrong answers do not. It does not check that the reasoning is correct. An answer can mention applied, 2xx, side effect, and timeout in a wrong order and pass. The negative checks catch the common wrong answer, not every wrong answer.
  • The 100% figure is a self-check. 211 of 211 checks pass against worked examples I wrote. It shows the fixtures are consistent, not that any agent performs well.
  • The runner does not call an agent. You collect answers yourself and score them. That keeps it provider-neutral and free to run in CI, but it means a real comparison takes work on your side.
  • Platform facts drift. Header names, limits, and filters are documented against WooCommerce and Shopify behavior I checked, and the reference notes say to confirm against your version when you debug a live problem.

What's next

From the roadmap: graded eval rubrics beyond substring checks, more store platforms written to the same level of API detail as the WooCommerce and Shopify skills, more payment providers written like the Stripe and Zarinpal skills, and runnable reference services for the payout ledger and a feed upload, with the same checks the local scripts apply.

Install 

npx @sadrmanesh/commerce-agent-skills list npx @sadrmanesh/commerce-agent-skills add order-and-webhook-reliability stripe-payments npx @sadrmanesh/commerce-agent-skills init --layout root

Node.js 20 or newer. The package is on npm and, under the same name, on GitHub Packages. The skills are also published as a dataset on Hugging Face at sadrmanesh/commerce-agent-skills.

MIT licensed. If you have a commerce bug that an agent keeps getting wrong, open an issue with the task and the wrong answer. That is exactly the shape of a new eval.