Stop Digging Through CI Logs: One Test Report for Python, Java, JS, .NET, Go and Robot Framework

It is 9:40 in the morning. The nightly build is red. The CI log is 14,000 lines long, and somewhere in it are the four failures you care about. Before you can fix anything, you have to answer the same questions every team asks after every red build:

1. What broke since yesterday? Not "what is failing", but what is newly failing.
2. Is it one problem or ten? Forty tests failing with the same ConnectionError is one broken Redis, not forty bugs.
3. Is this test flaky, or did we really break it?
4. Can the person who isn't a Python developer read the result? The QA lead, the release manager, the Java team whose service you call.

Most test runners answer none of these out of the box. They print pass/fail to a terminal and leave the rest to you.

I built pyrept to answer all four, for every test framework a real company runs, in one consistent report. This article is a practical, engineering-focused tour: what it produces, how it works inside, and copy-paste recipes for Python, Java, JavaScript/TypeScript, .NET, Go, Robot Framework, Ruby, Rust and shell tests, plus GitHub Actions, GitLab, Jenkins and Azure DevOps.

Try it before reading further: here is the live demo report and the demo history page. Open them on your phone, they work there too.

TL;DR

pip install pyrept pytest --pyrept 

Open report.html. That's it for pytest. For everything else:

Every path produces the same four outputs:

• report.html: one self-contained page. No CDN, no external fonts, no server. Open it offline, attach it to a CI run, email it.
• report.json: the same data, machine-readable, for dashboards, bots and the next run's comparison.
• junit.xml (optional): so GitLab, Jenkins, Azure DevOps and friends show results in their native test tabs.
• A Markdown summary (optional): for GitHub job summaries, pull request comments and chat.

The only runtime dependency is jinja2. Python 3.8 to 3.14 are tested in CI, plus the 3.15 pre-releases.

Part 1: The technical spec

Before the recipes, it helps to know the model pyrept builds, because every feature falls out of it.

1.1 The pipeline

 pytest plugin ───────┐ unittest runner ─────┤ nose2 plugin ────────┤ ┌──> report.html (self-contained) behave formatter ────┼──> ReportCollector ─────┼──> report.json (machine-readable) Python API ──────────┤ │ ├──> junit.xml (CI test tabs) │ ├─ baseline diff ├──> summary.md (job summary / PR) pyrept convert ──────┘ ├─ failure groups └──> Slack / Teams / webhook (JUnit, TRX, NUnit, └─ quality gates xUnit, Robot, TAP, Allure, Cucumber, Playwright, ...) 

Every integration, native or imported, feeds one ReportCollector. Everything downstream (HTML, JSON, JUnit, Markdown, comparison, gates, notifications) is written once and behaves identically no matter where the tests came from. That's the whole design: normalise early, render once.

1.2 Four outcomes, mapped the same way everywhere

Every test ends as exactly one of passed, failed, error or skipped. The mapping is deliberate and consistent across all frameworks and importers:

Pass rate = passed ÷ (total − skipped).

Engineering note: why skipped tests don't lower the pass rate. If you skip 200 tests because a sandbox is down, your code didn't get 200 tests worse. Counting skips as failures trains teams to delete skip markers instead of fixing environments. Skips are shown prominently, they just don't pollute the quality signal.

Engineering note: why failed and error are different. A failed test means your code produced the wrong answer. An error usually means the test couldn't even run: a fixture blew up, an import failed, a dependency was down. When 30 errors appear at once, you look at infrastructure first. When 3 failures appear, you look at the diff. Keeping them separate saves real triage time.

1.3 The JSON contract

report.json is the stable interface between pyrept and everything else (dashboards, bots, the next run). Abbreviated:

{ "test_report_title": "Checkout suite", "test_summary": {"total": 51, "passed": 46, "failed": 2, "error": 2, "skipped": 1, "percentage": 92.0}, "timestamp": "2026/10/07 11:58:46 UTC", "duration": 13.29, "environment": {"Python": "3.13.2", "Framework": "pytest 9.0 + Playwright", "Build": "#1842"}, "slowest_tests": [{"name": "tests/test_checkout.py::test_pay_with_card", "duration": 3.42}], "failure_groups": [{"message": "redis.exceptions.ConnectionError: Error 111 connecting to cache:6379. Connection refused.", "count": 2, "tests": ["tests/test_api.py::test_rate_limit", "tests/test_api.py::test_session_cache"]}], "comparison": { "percentage_delta": -4.0, "new_failures": ["tests/test_checkout.py::test_pay_with_card"], "fixed": [], "still_failing": [], "new_tests": [], "removed_tests": [], "slower": [{"name": "...", "before": 1.2, "after": 3.42, "ratio": 2.85}] }, "test_results": [ {"name": "tests/test_checkout.py::test_pay_with_card", "result": "failed", "description": "Pays with a saved Visa card.", "traceback": "AssertionError: ...", "metadata": {"duration": 3.42, "location": "tests/test_checkout.py:42", "tags": ["smoke", "payments"]}} ] } 

test_results lists errors first, then failures, skips and passes. comparison is null when no baseline was given. Because the format is plain JSON, a pyrept report from one job is also a valid input to another: that's how sharded runs and multi-language monorepos get merged (Part 3).

1.4 Regression diff: comparing with the previous run

Give any integration an earlier report.json as a baseline, and every test gets a change label:

Engineering note: why "slower" needs two thresholds. A ratio alone flags a 2 ms test that became 5 ms (2.5×, irrelevant). An absolute delta alone flags a 60 s integration test that jittered to 60.6 s. Requiring both 1.5× and +0.5 s catches real regressions and ignores noise.

Engineering note: the baseline can be the file you're about to overwrite. pyrept reads the baseline before writing, so --pyrept-json=report.json --pyrept-baseline=report.json always compares with the previous run. A missing or corrupt baseline never fails the run; it logs a warning and writes the report without a comparison. That's exactly what you want on the first CI run of a new branch.

Tests are matched by name, so compare reports that come from the same source.

1.5 Common failure causes

Failures that share an error message are grouped. In the demo, two API tests fail with the same redis.exceptions.ConnectionError; the report shows one group of 2, not two separate mysteries. On a bad day this turns "38 failures" into "1 broken Redis + 2 real bugs", which is a very different stand-up.

1.6 What gets captured per test

• Description (docstrings, BDD scenario text), tags (pytest markers, NUnit categories, xUnit traits, Robot tags, Allure labels), duration and file:line.
• For failures: captured stdout, stderr and logs, the traceback, and its first line as the message.
• For browser tests: a full-page screenshot taken while the browser is still open, the browser console output and page errors (Playwright), and links to Playwright traces and videos.
• Retries and flakiness (pytest-rerunfailures, Surefire reruns, Playwright retries, Allure history).
• Environment: Python, platform, framework, plus anything you add.

1.7 Security and robustness details

These are the boring things that make a report tool safe to run on untrusted CI output:

• Everything is HTML-escaped, including tracebacks that contain <script>-like text, and attachment data from imported files. javascript: links in attachment paths are never rendered.
• Images up to 5 MB are embedded as data URIs, so the report stays a single file. Larger files stay linked.
• JUnit output strips characters XML forbids (such as ANSI colour codes), so CI parsers never reject it.
• The Markdown summary stays far below GitHub's 1 MiB job-summary limit, however big the run. It shows the first 20 failures with trimmed errors.
• Webhook URLs are treated as secrets: never logged, and a failed notification never fails your build.
• Exit codes are predictable: 0 OK, 1 a gate failed, 2 bad input (missing file, wrong format).

Part 2: Python recipes

2.1 pytest in 30 seconds

Here's a tiny shop test suite I'll use for the rest of the Python section:

# tests/test_cart.py import pytest @pytest.mark.smoke def test_add_item(): """Adding an item updates the cart count.""" assert 1 + 1 == 2 def test_apply_coupon(): """AUTUMN15 takes 15% off.""" assert 85.0 == 85.0 def test_legacy_tax_rules(): """Known broken since the tax refactor (JIRA-123).""" assert round(0.1 + 0.2, 2) == 0.31 @pytest.mark.skip(reason="payment sandbox down") def test_refund(): pass 

pytest --pyrept 

The plugin is installed with the package but stays off until you ask for a report, so it never changes a normal pytest run. Docstrings become descriptions, markers become tags, skip reasons are kept.

All the options:

pytest --pyrept-html=reports/report.html --pyrept-json=reports/report.json pytest --pyrept --pyrept-title="Nightly regression" pytest --pyrept --pyrept-junit=reports/junit.xml # also JUnit XML pytest --pyrept --pyrept-baseline=reports/report.json # compare with the previous run pytest --pyrept --pyrept-markdown=reports/summary.md # Markdown summary pytest --pyrept --pyrept-github-summary # GitHub job summary 

Or set it once in pytest.ini / pyproject.toml:

[pytest] pyrept = true pyrept_html = reports/report.html pyrept_json = reports/report.json pyrept_junit = reports/junit.xml pyrept_baseline = reports/report.json pyrept_github_summary = true 

Engineering note: path resolution. Paths on the command line are relative to where you run pytest. Paths in the ini file are relative to the ini file, so the report always lands in the same place no matter which subdirectory a developer runs pytest from. Small thing, saves a lot of "where did my report go?".

2.2 The regression workflow, for real

I ran the suite above twice. On the second run, a coupon calculation broke and the Redis cache was down. pytest's own output tells you 4 things failed. pyrept adds one line that tells you what matters:

------ pyrept: 3 new failures, 0 fixed, 1 still failing (vs report.json) ------- FAILED tests/test_cart.py::test_apply_coupon - assert 90.0 == 85.0 FAILED tests/test_cart.py::test_legacy_tax_rules - assert 0.3 == 0.31 ERROR tests/test_api.py::test_rate_limit - ConnectionError: Error 111 connect... ERROR tests/test_api.py::test_session_cache - ConnectionError: Error 111 conn... 

test_legacy_tax_rules was already broken yesterday. The other three are new, and two of them share one root cause. That's the triage done before you've opened a single traceback.

2.3 Quality gates for legacy suites: the ratchet

Every mature codebase has a few known-broken tests. Teams usually respond by either ignoring the red build (bad) or deleting the tests (worse). pyrept gives you a third option: fail only on new failures, with a floor.

pytest --pyrept-json=reports/report.json --pyrept-baseline=reports/report.json \ --pyrept-ignore-known-failures --pyrept-fail-under=90 

With the known-broken tax test only:

- pyrept: 1 known failure ignored: all of them were already failing in the baseline - 1 failed, 4 passed, 1 skipped in 0.04s $ echo $? 0 

The moment a previously passing test breaks:

--- pyrept: 3 new failures, so --pyrept-ignore-known-failures does not apply --- $ echo $? 1 

The fail-under floor stops known failures from quietly piling up. Over time you raise it: that's the ratchet.

Engineering note: gates never hide other failures. If pytest-cov's --cov-fail-under (or any other plugin) failed the run, pyrept keeps it failed. Collection errors (a test module that doesn't even import) are never ignored. A gate that can accidentally turn a broken build green is worse than no gate.

2.4 Parallel, retries, subtests, BDD

These just work with the same flags:

• pytest-xdist (-n auto): workers send results to the controller, which writes one report. Descriptions, tags and screenshots survive the trip.
• pytest-rerunfailures: reruns aren't counted as extra tests; the final attempt is recorded with retries, and marked flaky if it passed after a retry.
• Subtests (built into pytest 9+, or pytest-subtests): each failing subtest gets its own entry, like test_x (i=1).
• pytest-bdd: each scenario shows the feature, the scenario and every step with its status:

Feature: Checkout Scenario: Pay with card Given a cart with 2 items [passed] When I pay with a declined card [failed] Then I see the confirmation page [skipped] 

2.5 Playwright and Selenium (Python)

UI test failures without a screenshot are guesswork. When a test using a pytest-playwright page fixture, or a Selenium driver / selenium / browser fixture, fails, pyrept attaches a full-page screenshot (taken before the browser closes), the browser console output and uncaught page errors, and links to the trace and video:

pip install "pyrept[playwright]" && playwright install chromium pytest --pyrept --tracing retain-on-failure --video retain-on-failure 

Open traces at trace.playwright.dev. Links are relative to the report, so upload reports/ and test-results/ together as one CI artifact.

2.6 Extend it from conftest.py

Three hooks let you add environment rows, attachments or anything else without forking anything:

# conftest.py import os from pyrept import make_attachment def pytest_pyrept_environment(config): return {"Build": os.environ.get("BUILD_ID", "local"), "Commit": os.environ.get("GITHUB_SHA", "")[:7]} def pytest_pyrept_attachments(item, report): # Attach the API server's log to every failing test  if report.when == "call" and report.failed and os.path.exists("server.log"): with open("server.log") as fh: return [make_attachment("server.log", "text/plain", text=fh.read()[-20000:])] def pytest_pyrept_context(config, context): context["test_report_title"] += " (nightly)" 

pytest_pyrept_attachments runs where the test runs, including xdist workers, so it returns plain data.

2.7 unittest, nose2, behave

unittest, with zero extra dependencies. It accepts all the usual python -m unittest arguments:

python -m pyrept.unittest_runner discover -s tests \ --pyrept-junit=reports/junit.xml --pyrept-baseline=reports/report.json 

Or from code:

import unittest from pyrept.unittest_runner import PyreptTestRunner suite = unittest.defaultTestLoader.discover("tests") result = PyreptTestRunner(html_path="reports/report.html", json_path="reports/report.json", title="Unit tests", verbosity=2).run(suite) raise SystemExit(not result.wasSuccessful()) 

nose2 (pip install "pyrept[nose2]"), in nose2.cfg:

[unittest] plugins = pyrept.html_report [html-report] always-on = True html-report-path = test_results/report.html json-report-path = test_results/report.json 

behave (Cucumber BDD for Python):

pip install "pyrept[behave]" behave -f pyrept.behave_formatter:PyreptFormatter -o /dev/null -f pretty \ -D pyrept_html=reports/report.html -D pyrept_json=reports/report.json 

Gotcha worth knowing: behave pairs each -o with the -f before it. Put -f pyrept... -o /dev/null first; if you write -f pretty -f pyrept -o /dev/null, your console output disappears into /dev/null. (Windows: -o NUL.)

Part 3: Every other stack

Here's the part I'm most proud of. Your QA team writes Robot Framework, the backend is Java, the frontend is TypeScript, the payment service is .NET, and the CLI is Go. They all produce some report file. pyrept convert turns each of them into the same HTML/JSON/JUnit/Markdown output, and the project you convert from doesn't need to be a Python project. You only need Python on the CI runner, which every hosted runner already has.

3.1 The format matrix

Each converter does more than read pass/fail:

• JUnit XML: properties become tags, Surefire flakyFailure/rerunFailure become retries, [[ATTACHMENT|...]] screenshots are embedded, system-out/err are attached.
• TRX: data-driven rows become separate tests, result files are linked or embedded.
• Robot Framework: long names like Suite.Sub suite.Test, tags, documentation, durations for both the Robot 7 elapsed attribute and older start/end timestamps.
• TAP 12 to 14: plans, SKIP / TODO directives, YAML diagnostics, and Bail out!.
• Allure: retries folded into one test with history, labels as tags, attachments kept.

Mapping is consistent across all of them: crashes, timeouts and broken tests become error; ignored, inconclusive and TODO tests become skipped.

3.2 One report for a polyglot release

Here's a real run I did while writing this article: a Java Surefire file, a Jest JUnit file, a .NET TRX file, a Robot Framework output.xml and a Node TAP file, merged into one release report.

pyrept convert --from junit target/surefire-reports/*.xml --json parts/java.json --html parts/java.html pyrept convert --from junit web/junit.xml --json parts/web.json --html parts/web.html pyrept convert --from trx TestResults/*.trx --json parts/dotnet.json --html parts/dotnet.html pyrept convert --from robot results/output.xml --json parts/robot.json --html parts/robot.html pyrept convert --from tap cli/results.tap --json parts/cli.json --html parts/cli.html # Merge every stack into one report (pyrept JSON is itself an input format) pyrept convert --from pyrept parts/*.json --title "Release 2026.10 - all stacks" \ --html all/report.html --json all/report.json --junit all/junit.xml 

pyrept: 23 tests (10 passed, 6 failed, 2 errors, 5 skipped) pyrept HTML report: all/report.html pyrept JSON report: all/report.json pyrept JUnit report: all/junit.xml 

Your release manager gets one link. Each test keeps its framework metadata (junit, trx, robot, tap), so you can still tell where it came from.

The same trick merges sharded CI jobs: run 8 shards in parallel, have each upload its junit.xml or report.json, then merge in a final job:

pyrept convert --from junit shard-*/junit.xml --html reports/report.html --json reports/report.json 

3.3 Gates and baselines work for every stack

Everything from Part 2 applies to converted reports:

# Robot Framework suite: fail the job only on tests that passed last time, # and never let the pass rate fall below 95% pyrept convert --from robot output.xml \ --baseline reports/report.json --json reports/report.json --junit reports/junit.xml \ --fail-on-failure --ignore-known-failures --fail-under 95 

So a Java team gets "3 new failures, 1 fixed" from their Surefire XML exactly like a Python team does from pytest.

Part 4: CI recipes

4.1 GitHub Actions: job summary and a sticky PR comment

The pyrept GitHub Action puts the result on the run's summary page and keeps one pull request comment up to date (it edits its own comment instead of spamming a new one per push).

name: tests on: [push, pull_request] permissions: contents: read pull-requests: write # for the PR comment jobs: test: runs-on: ubuntu-latest steps: - uses: actions/checkout@v4 - uses: actions/setup-python@v5 with: python-version: '3.13' # Keep the previous report so every run is compared with the one before it - uses: actions/cache/restore@v4 with: path: reports/report.json key: pyrept-${{ github.ref_name }}-${{ github.run_id }} restore-keys: pyrept-${{ github.ref_name }}- - run: pip install -r requirements.txt pyrept - run: > pytest --pyrept-html=reports/report.html --pyrept-json=reports/report.json --pyrept-baseline=reports/report.json continue-on-error: true # let the action decide whether the job fails - uses: pankajnayak1994/pyrept@v1 with: report: reports/report.json fail-on-failure: true ignore-known-failures: true fail-under: 90 - uses: actions/cache/save@v4 if: always() with: path: reports/report.json key: pyrept-${{ github.ref_name }}-${{ github.run_id }} - uses:...