LLM Regression Testing for Jira with CogniRunner

LLM regression testing means running the same labelled inputs again after you change a prompt or a model, to see what the change broke. This tutorial does that for an AI check on a Jira workflow validator. Here the test set is a Jira filter of 8 labelled requests. An 80-line Node script runs it. Each call goes to CogniRunner's rule test over REST, which reads the issue, asks the model and writes nothing to Jira.

I ran it on 13 October 2026 because I wanted to move a validator from openai/gpt-6-luna to anthropic/claude-haiku-5.5. Over three rounds, the site's model matched 6, 7 and 7 of the 8 labels. The candidate matched 8 of 8. I built the test expecting the candidate to miss, and it was the site's model that missed (I'd have bet the other way).

I found the cause in the rule, and I don't think it says much about either model. Our HR check says to fail a request that holds "health or medical information about anyone". Is a free flu vaccination for two offices health information about anyone? Our rule didn't say. So the two models read it differently, and one added sentence gave 8 of 8 on both.

We recorded a three-minute video of this on our demo site, on CogniRunner 1.34.0. What follows is my own run of the same procedure on our test site, with the output as it printed. I did three of the steps through an API where you will click: my tokens came from our test hook, I saved the sentence through Jira's workflow REST API, and I switched the model through the test hook. The video shows those screens, and I give the timestamp at each step.

https://www.youtube.com/watch?v=lMpvV03upQA

[!NOTE]
Prerequisites

• A Jira Cloud site with CogniRunner 1.34.0 or later. The Marketplace calls that build 7.16.0, released 12 October 2026. I ran a development build of 1.34.0 on our test site.
• An AI provider saved in CogniRunner's Settings. Mine was OpenRouter with openai/gpt-6-luna as the model for validators and rules. If you haven't set one up, the provider tutorial covers it.
• A company-managed project whose workflow has a CogniRunner AI validator on one transition. Mine was called Send for Review.
• An email and API token for Jira's REST API. The script reads the published workflow. I ran it as a Jira admin.
• A CogniRunner admin, to create the REST API token in step 2.
• Node 18 or later. I used Node v24.15.0. The script has no dependencies.

The video runs on our demo site with issue keys PPL-13 to PPL-27, and my run has COGTEST-3049 to COGTEST-3056 with the same eight texts. Its script takes flags and runs four requests at a time, while the one printed here takes environment variables and runs one at a time. In the video the first round is 7 of 8. Mine was 6 of 8.

[[steps]]

1. Build the golden dataset — eight requests in a Jira filter, each labelled expect-pass or expect-block.
2. Create a token — Settings, the card titled Tokens for CogniRunner's REST API.
3. Save the script — it reads the saved rule from Jira and calls the rule test for each request.
4. Run the baseline — the site's own model, three rounds.
5. Run the candidate — one variable, and the site's model stays as it is.
6. Read the mismatch — decide whether the model or the rule is wrong.
7. Try the sentence, save it, run both again — first unsaved, then saved.
8. Switch the site's model — and find the switch and the tests in the audit log.

What one rule test does: one field, one prompt, nothing written

A CogniRunner AI validator sits on a workflow transition. Somebody moves the request. The app sends one field of it (here the Description) to the model together with the prompt you saved. Back comes pass or fail, with a reason. Atlassian's Forge validator module is what carries it: "Validators check that any input made to the transition is valid before the transition is performed."

You may know the Test button in the rule editor. CogniRunner's Rules REST API has the same thing as a call: POST ?resource=rules&action=test. Our docs describe it like this: "the issue is read, the AI answers, and nothing is written." The body carries the issue key, the field and the prompt. Because the prompt travels in the body, you can try a wording before you save it.

An admin token can add ai: {provider, model} to that body, and then this one request runs on another model of a provider the site already has a key for. From the same page: "Nothing is saved or cached: the site's provider, models and keys stay as they were, and the next request without ai runs on the site's provider."

So the whole procedure is a loop over a filter. The same call takes other kinds of rule as well, and I tested validators. The rest of this API is in the release post for Marketplace version 3.3.0 and on the product page. Another thing you can do to the same validator is hand it a document, and the tutorial on giving one an API spec covers that.

Step 1. Build the golden dataset: a Jira filter of 8 labelled issues

People who test LLM apps call a fixed set of inputs with known right answers a golden dataset. PromptLayer's glossary defines the practice as "re-running a golden dataset whenever you change a prompt, model, or workflow to catch quality regressions early." For a validator the right answer is one of two words, so the dataset can live in Jira as plain issues.

Create eight requests in the project the validator guards. Give each the label gate-test, and then expect-pass or expect-block. Mine were four of each:

All of these people are made up. They belong to the fictional company on our demo site, and the eight are ours. Write yours from requests your own people sent, with the names changed.

Then save the search as a filter. Mine is called Gate tests: HR health check, with this JQL:

project = COGTEST AND labels = gate-test ORDER BY key ASC 

Note the filter's id from its URL. Mine was 12450.

Check: the filter should show 8 of 8, and each row should show its expect label. If a row shows another label first, remove the extra labels. Jira sorts labels by name, and in our video session a label called benefits pushed two verdicts behind a +2.

Is eight enough? Probably not for a rule that matters a lot. Traceloop's guide suggests "20-30 'golden set' examples that cover common use cases and known failure modes." Our eight came out of 28 candidates. Our video session ran those 28 on both models, 600 dry runs in all. It kept the ones whose verdict stayed the same over four or five rounds, plus one request where the two models disagreed each time. Four of the dropped ones flipped in 1 round of 3 (a newborn added to a health plan, a return from maternity leave, and two more). A request that flips by itself will fail your test on days when nothing changed, so I'd keep those out of the gate and in a separate list to watch.

Step 2. Create a token for CogniRunner's REST API

Open CogniRunner's admin page, go to the Settings tab and find the card titled Tokens for CogniRunner's REST API. It opens on Viewer and Their own rules. Change those to Admin and Everyone's rules. My tokens had both, and I haven't tried a narrower one. Type a name and click Create token. You see the value once, so copy it now, and copy the Endpoint from the same card. The video shows this card at 0:34, with its two tokens already made.

Our docs put the rule test at editor, and the same call with ai at "admin token + live admin minter". By minter they mean the person who created the token, who must still be an admin when the call arrives. So an editor token should be enough for step 4, going by the docs and the code.

Tokens run out. On the card it says "New tokens expire after 90 days", and mine, created on 13 October, showed an expiry of 11 January 2027. Also treat the endpoint address as private too. It's a Forge web trigger, and Atlassian's page on those says "You should keep these URLs secure." CogniRunner asks for the bearer token on top. I still mask the address everywhere, this page included.

Put the values in your shell:

export JIRA_SITE=https://wolfaenpak.atlassian.net JIRA_EMAIL=<admin email> JIRA_API_TOKEN=<jira api token> export RULES_API_URL=<rules-api> RULES_API_TOKEN=cgr_<masked> export WORKFLOW="Software Simplified Workflow for Project COGTEST" TRANSITION="Send for Review" FILTER=12450 

That first value is our test site, so put your own there. WORKFLOW is the workflow's name as Jira's workflow list shows it. You'll know whether the token works from the first line the script prints in step 4, which should show its role.

Step 3. Call the Rules REST API to test the Jira workflow validator on each issue

Save this as gate-test.mjs. It needs no packages. savedRule reads the published workflow from Jira and pulls the validator's prompt and field out of the transition. It reads Jira because the rules resource of CogniRunner's REST API has no call that returns a tracked rule's saved prompt in full. testSet reads the filter and takes the expected verdict from the label. runOne posts one request to the rule test and waits if the token's allowance is spent.

#!/usr/bin/env node // gate-test.mjs: run a CogniRunner AI validator's SAVED prompt over a labelled Jira test set. Node 18+, no dependencies. // Every call is a dry run (POST ?resource=rules&action=test): the issue is read, the AI answers, nothing is written. // JIRA_SITE=https://you.atlassian.net JIRA_EMAIL=... JIRA_API_TOKEN=... \ // RULES_API_URL=<the Rules REST API address from CogniRunner Settings> RULES_API_TOKEN=cgr_... \ // WORKFLOW="<workflow name>" TRANSITION="Send for Review" FILTER=<filter id> node gate-test.mjs // Optional: CANDIDATE_MODEL=<provider>/<model id> runs each request on that model instead (needs an ADMIN token; the // site's own model is not changed). CANDIDATE_PROVIDER defaults to openrouter. ADD_SENTENCE="..." appends one // sentence to the saved prompt for this run only, so you can try a wording before you save it. const E = process.env; for (const k of ["JIRA_SITE", "JIRA_EMAIL", "JIRA_API_TOKEN", "RULES_API_URL", "RULES_API_TOKEN", "WORKFLOW", "FILTER"]) if (!E[k]) { console.error(`Set ${k}`); process.exit(2); } const TRANSITION = E.TRANSITION || "Send for Review"; const JAUTH = "Basic " + Buffer.from(`${E.JIRA_EMAIL}:${E.JIRA_API_TOKEN}`).toString("base64"); const sleep = (ms) => new Promise((r) => setTimeout(r, ms)); const pad = (s, n) => String(s ?? "").padEnd(n).slice(0, n); async function jira(path, body) { const r = await fetch(E.JIRA_SITE + path, { method: "POST", headers: { Authorization: JAUTH, Accept: "application/json", "Content-Type": "application/json" }, body: JSON.stringify(body) }); if (!r.ok) throw new Error(`Jira ${path}: ${r.status} ${(await r.text()).slice(0, 200)}`); return r.json(); } async function rules(method, query, body) { const r = await fetch(`${E.RULES_API_URL}?${query}`, { method, headers: { Authorization: `Bearer ${E.RULES_API_TOKEN}`, "Content-Type": "application/json" }, body: body ? JSON.stringify(body) : undefined }); const text = await r.text(); let json = null; try { json = JSON.parse(text); } catch { /* not JSON */ } return { status: r.status, json, retryAfter: Number(r.headers.get("retry-after") || 0) }; } // 1. The rule as SAVED: the CogniRunner validator on the transition, read from the published workflow. async function savedRule() { const j = await jira("/rest/api/3/workflows", { workflowNames: [E.WORKFLOW] }); for (const w of j.workflows || []) for (const t of w.transitions || []) { if (t.name !== TRANSITION) continue; for (const v of t.validators || []) { if (!/ai-text-field-validator/.test(v.parameters?.key || "")) continue; const c = JSON.parse(v.parameters.config); return { prompt: c.prompt, fieldId: c.fieldId || "description", enableTools: c.enableTools === true, version: w.version?.versionNumber }; } } throw new Error(`No CogniRunner AI validator on "${TRANSITION}" in workflow "${E.WORKFLOW}"`); } // 2. The test set: every issue in the filter; the expected verdict is its label. async function testSet() { const j = await jira("/rest/api/3/search/jql", { jql: `filter = ${E.FILTER} ORDER BY key ASC`, fields: ["summary", "labels"], maxResults: 100 }); return (j.issues || []).map((i) => ({ key: i.key, expected: i.fields.labels.includes("expect-block") ? "BLOCK" : i.fields.labels.includes("expect-pass") ? "PASS" : "?" })) .sort((a, b) => Number(a.key.split("-")[1]) - Number(b.key.split("-")[1])); } // 3. One dry run through the test door. A 429 means the token's AI allowance is spent: wait for Retry-After. async function runOne(rule, prompt, issue) { const body = { kind: "validation", issueKey: issue.key, prompt, fieldId: rule.fieldId, enableTools: rule.enableTools }; if (E.CANDIDATE_MODEL) body.ai = { provider: E.CANDIDATE_PROVIDER || "openrouter", model: E.CANDIDATE_MODEL }; for (let attempt = 0; attempt < 4; attempt++) { const t0 = Date.now(); const r = await rules("POST", "resource=rules&action=test", body); if (r.status === 429) { const s = r.retryAfter || 60; console.error(` 429 on ${issue.key}: allowance spent, waiting ${s} s`); await sleep(s * 1000); continue; } if (r.status === 202) return { ...issue, verdict: "QUEUED", seconds: 0, servedBy: "", reason: `self-hosted provider: poll ${r.json.poll}` }; const res = r.json?.result || {}; const verdict = r.status !== 200 ? `HTTP ${r.status}` : res.success && res.isValid === true ? "PASS" : res.success && res.isValid === false ? "BLOCK" : "NOT JUDGED"; return { ...issue, verdict, seconds: Math.round((Date.now() - t0) / 100) / 10, servedBy: r.json?.ai?.servedBy?.map((s) => s.model).join(",") || res.modelUsed || "", reason: res.reason || res.error || r.json?.error || "" }; } return { ...issue, verdict: "GAVE UP", seconds: 0, servedBy: "", reason: "429 four times" }; } const who = await rules("GET", "resource=whoami"); if (who.status !== 200) { console.error(`whoami: HTTP ${who.status} ${who.json?.error || ""}`); process.exit(1); } const spend = who.json.aiSpend || {}; console.log(`Token role: ${who.json.effective?.role} · AI actions used ${spend.window?.used}/${spend.window?.limit} in this 10-minute window, ${spend.today?.actions}/${spend.today?.limit} today`); const rule = await savedRule(); const prompt = E.ADD_SENTENCE ? `${rule.prompt} ${E.ADD_SENTENCE.trim()}` : rule.prompt; const set = await testSet(); console.log(`Rule: ${E.WORKFLOW} · ${TRANSITION} · reads ${rule.fieldId} · workflow version ${rule.version}`); console.log(`Prompt (${E.ADD_SENTENCE ? "saved + one sentence, this run only"<...