Anyone can say an AI rewriting tool "works". Few people measure it. Here is a small, fair experiment you can run yourself in under an hour, with a Python script to do the counting.

The idea

Take the same input, run it through a tool, and compare the output against the original using the same measurements every time. The goal is not to prove a tool is "best". It is to see what actually changes.

Step 1: Build a small test set

Pick 5 to 10 short passages (80 to 150 words each) from different types of writing:

  • A technical explanation
  • A product description
  • A casual blog paragraph
  • A formal email

Save each one as a separate text file, for example samples/01_before.txt. Using different types matters, because a tool can do well on one style and badly on another.

Step 2: Run each sample through the tool

Paste each passage into the tool, copy the result, and save it as samples/01_after.txt. Use the same settings for every sample, and write them down.

Step 3: Measure what changed

This script compares each before/after pair on three simple signals.

import re import statistics import textstat # pip install textstat from pathlib import Path def sentence_lengths(text): sentences = [s for s in re.split(r"[.!?]+\s", text) if s.strip()] return [len(s.split()) for s in sentences] def report(text): lengths = sentence_lengths(text) spread = round(statistics.pstdev(lengths), 1) if len(lengths) > 1 else 0 return { "words": len(text.split()), "flesch": round(textstat.flesch_reading_ease(text), 1), "spread": spread, } for before_file in sorted(Path("samples").glob("*_before.txt")): after_file = before_file.with_name(before_file.name.replace("_before", "_after")) if not after_file.exists(): continue b = report(before_file.read_text()) a = report(after_file.read_text()) print(before_file.stem.replace("_before", "")) print(" before:", b) print(" after: ", a)

Step 4: Check what numbers cannot tell you

Numbers do not show whether the meaning stayed the same. For each pair, ask:

  1. Is every fact still correct?
  2. Did any important detail disappear?
  3. Does it still sound like the same writer?
  4. Is it easier to read aloud?

Give each pair a score from 1 to 5 for meaning and for readability. Write the scores in a simple table.

How to read the results

  • A higher Flesch score means easier reading, but a score that jumps a lot may mean the text became too simple.
  • A higher spread means sentence lengths vary more, which usually feels more natural.
  • A big drop in words can mean details were cut.

These are rough signals. They are useful for comparing tools on the same samples, not for judging quality on their own.

Keep the test fair

  • Use the same samples and settings for every tool.
  • Don't pick only the examples that look good.
  • Share your method, so others can repeat it.

If you want a tool to include in your own test, you can try RewriteAI alongside any others and compare the numbers side by side.

Final thought

A short, repeatable test tells you more than any marketing page. Try it on your own writing and see what changes.

What would you add to this test? Share your ideas in the comments.