How many invoices does it take to teach a model?

Any fine-tuning project for invoice extraction starts with one question: how many of our invoices do you need before this works? For this task, the measured answer is about 100. Those 100 labelled invoices took the model to 98% on known layouts. The next 900 added 1.6 points, and none of them helped on layouts the model had never seen.

A guess is not an answer, so I measured the curve: the same model, the same settings, six training-set sizes, three seeds each.

This post follows "Prompting vs Fine-Tuning: Invoice Extraction at Scale" (link to add). That post trained one LoRA adapter on 240 invoices and found it beat prompting (86% vs 84%) at about a third of the token cost. It listed "no sweep over training-set size" as a limitation. This post closes that gap.

The setup in a few lines

Every run uses the same task, model and settings; only the number of training invoices (and the seed) changes.

• Task: pull the invoice number, date, vendor, line items and total out of a text invoice (German, UK English or French) and return JSON.
• Model: Qwen2.5-1.5B-Instruct, run locally on an RTX 5070 Ti (16 GB).
• Method: LoRA on the attention layers (rank 16, alpha 32), 3 epochs, learning rate 2e-4, batch size 4.
• Data: synthetic invoices generated from code, so the correct answer is exact by construction. Training invoices are drawn evenly across 8 layouts and 3 languages.
• Test sets, frozen and never trained on: 192 invoices in the 8 layouts the model trains on ("seen") and 45 invoices in 3 layouts it never trains on ("unseen").
• Sizes and seeds: 25, 50, 100, 250, 500 and 1,000 training invoices, each trained 3 times. A seed changes which invoices are picked and the adapter's starting values. For one seed, the smaller sets sit inside the larger ones.
• Baseline: no training, just the best prompt from the first post (instructions plus one worked example).
• Metric: invoice accuracy, the share of invoices with every field correct. One wrong field means a person has to check the invoice, so this is the number a business cares about.

The result: accuracy against training invoices

On known layouts, accuracy climbs from 81% at 25 invoices to 98% at 100 and 100% at 1,000; on unseen layouts it levels off at 77-81%, no better than the prompt alone.

Invoice accuracy, mean of 3 seeds with the range in brackets. Every adapter is 16.7 MB, and peak GPU memory was about 12 GB in every run.

Where the curve flattens, and what it means for a budget

The curve flattens at about 100 invoices: going from 100 to 1,000 adds 1.6 points on known layouts (98.4% to 100%) for ten times the data.

The steps before that are large. From 25 to 50 invoices the gain is 12.2 points, and from 50 to 100 it is another 5.3. Training starts to beat the prompt at about 50 invoices on known layouts (93.1% vs 84.9%).

At 98.4%, roughly 3 of the 192 test invoices needed a person to look at them. That is the price of stopping at 100.

The GPU is not the cost. The longest run, 1,000 invoices, trained in 3.2 minutes. For a client, the real cost is collecting and labelling invoices, so the useful question is whether the extra 1.6 points are worth labelling 900 more.

The surprise: 25 invoices are worse than none

On average, a model trained on 25 invoices scores below the same model with a good prompt: 80.9% vs 84.9% on known layouts.

On known layouts the picture is mixed. Seed 1 reached 85.4%, one invoice above the baseline, while the other two seeds scored 77.1% and 80.2%. On unseen layouts it is clear: all three seeds land at 42-49%, against 82.2% for the prompt.

25 invoices is about one per layout-and-language group (there are 24). I did not test why that is too little; the sweep only shows that it is. If you have a handful of labelled invoices, put them in the prompt as examples instead of training on them.

Where it did not work: layouts the model never saw

More training data did not fix new layouts. From 100 to 1,000 invoices, unseen accuracy stays at 77-81% on average, and no size beats the prompt's 82.2%.

Single runs do reach it: two runs tie at 82.2%, and one seed at 250 invoices hit 86.7%. But the unseen set has only 45 invoices, so one invoice is about 2.2 points, and these differences are within noise. Meanwhile, accuracy on seen layouts rises to 100%. Training teaches the layouts it sees.

This matches the finding of the first post. If a client's invoices come in layouts that are not in the training data, a prompt with an example is at least as good, and fine-tuning should not be sold as the fix.

What to do with 20, 100 or 1,000 client invoices

The size of the labelled set decides the approach.

• About 20 invoices: use them as examples in the prompt, not for training. The sweep tested a prompt with one worked example; I did not test more.
• About 100 invoices: train. That was enough for 98.4% on known layouts. Cover every layout and language you expect, because this test drew the training invoices evenly across all of them.
• About 1,000 invoices: the extra 900 bought 1.6 points on the layouts already covered. I did not test whether more layouts in the training set would help unseen ones, and that is the obvious next experiment.

In every case, budget for the labelling, not the GPU.

Before you trust these numbers

These numbers describe this task and this model, and they come with limits.

• Synthetic data. The invoices are cleaner than real ones, so real documents may need more data than this sweep suggests.
• Scope of the claim. For this task and this model, the curve flattened around 100 invoices. That is a measurement, not a universal rule.
• A small unseen set. It has 45 invoices, so one invoice is about 2.2 points. The 250-invoice range (77.8-86.7%) shows how much that moves things.
• Three seeds. The ranges are the lowest and highest of three runs, not confidence intervals. The prompt-only baseline ran once.
• Batched evaluation. Test invoices are generated 16 at a time with greedy decoding. I re-ran one run (25 invoices, seed 0) one invoice at a time. Invoice accuracy was identical (77.1% seen, 42.2% unseen); field accuracy differed by 0.3 and 0.4 points. Evaluation took about 1 minute batched and 12.9 minutes one at a time.
• Every number in this post matches a row in results.csv, and plot.py prints the same table.

How to rerun it

The whole experiment is one script, sweep.py, and it picks up where it stopped if interrupted: runs already in results.csv are skipped.

python sweep.py # baseline, then 6 sizes x 3 seeds (0, 1, 2) python sweep.py --sizes 100 --seeds 0 1 2 python sweep.py --skip-baseline python plot.py # prints the table, saves the chart 

sweep.py expects an invoice_common folder in its parent directory, with the invoice data and the prompt, parsing and scoring helpers.

Three parts of the script matter for reading the results. First, how a training set is picked: invoices are drawn round-robin from the 24 layout-and-language groups, so every size is balanced and, for one seed, a smaller set is the start of a larger one.

def pick_subset(n: int, seed: int) -> list[str]: """n pool invoices, spread evenly over the 24 format x language groups.""" answers = json.loads((POOL_DIR / "answers.json").read_text(encoding="utf-8")) assert n <= len(answers), f"pool only has {len(answers)} invoices" groups = defaultdict(list) for filename, info in sorted(answers.items()): groups[(info["format"], info["language"])].append(filename) rng = random.Random(seed) keys = sorted(groups) for key in keys: rng.shuffle(groups[key]) rng.shuffle(keys) # with n < 24, which groups get the spare places varies by seed  picked = [] while len(picked) < n: for key in keys: if groups[key] and len(picked) < n: picked.append(groups[key].pop()) return picked 

Second, the training settings, which never change between runs. The seed is set before the adapter is created, so a seed repeats the adapter's starting values too.

EPOCHS = 3 RANK, ALPHA = 16, 32 set_seed(seed) # before get_peft_model, so the adapter's starting values repeat too model = AutoModelForCausalLM.from_pretrained(MODEL_DIR, dtype=torch.bfloat16).to( "cuda" ) model = get_peft_model( model, LoraConfig( r=RANK, lora_alpha=ALPHA, target_modules=["q_proj", "k_proj", "v_proj", "o_proj"], lora_dropout=0.05, task_type="CAUSAL_LM", ), ) trainer = Trainer( model=model, args=TrainingArguments( output_dir=str(out_dir), num_train_epochs=EPOCHS, per_device_train_batch_size=4, learning_rate=2e-4, logging_steps=10, save_strategy="no", report_to="none", bf16=True, seed=seed, ), train_dataset=Dataset.from_list(examples), data_collator=collate, ) 

Third, evaluation: test invoices are generated 16 at a time, padded on the left so every prompt ends where its answer starts, with greedy decoding.

# Pad on the LEFT for generation, so every prompt ends right where its answer # starts. (Training pads by hand in collate(), so this does not affect it.) tokeniser.padding_side = "left" # ... then, for each batch of 16 test invoices: inputs = tokeniser(chat_texts, return_tensors="pt", padding=True).to("cuda") n_in = inputs["input_ids"].shape[1] # same for all: padded to the longest with torch.no_grad(): output = model.generate(**inputs, max_new_tokens=400, do_sample=False) 

The full code

Save the files below in one folder. plot.py needs only matplotlib and results.csv; sweep.py needs a GPU and the invoice_common folder described above.

sweep.py

import argparse import csv import gc import json import random import sys import time from collections import defaultdict from pathlib import Path import torch from datasets import Dataset from peft import LoraConfig, PeftModel, get_peft_model from transformers import ( AutoModelForCausalLM, AutoTokenizer, Trainer, TrainingArguments, set_seed, ) COMMON = Path(__file__).resolve().parent.parent / "invoice_common" sys.path.insert(0, str(COMMON / "evaluations")) from parsing import parse_output from prompts import build_ft, build_p2 from score import score_one POST_DIR = Path(__file__).resolve().parent POOL_DIR = COMMON / "data" / "pool" MODEL_DIR = COMMON / "qwen15b_base" ADAPTERS_DIR = POST_DIR / "adapters" RESULTS_CSV = POST_DIR / "results.csv" SIZES = (25, 50, 100, 250, 500, 1000) SEEDS = (0, 1, 2) EVAL_BATCH = 16 # test invoices generated at once; 1 at a time leaves the GPU mostly idle TEST_BUCKETS = {"test_seen": "seen", "eval_unseen": "unseen"} FIELDS = ("invoice_number", "date", "vendor_name", "line_items", "total") COLUMNS = [ "run", "n_train", "seed", "rank", "epochs", "train_minutes", "peak_vram_gb", "adapter_mb", "field_acc_seen", "invoice_acc_seen", "field_acc_unseen", "invoice_acc_unseen", "eval_minutes", ] # Fixed in every run - only the number of invoices and the seed change. MAX_LENGTH = 768 # longest example measured in the first post: 538 tokens EPOCHS = 3 RANK, ALPHA = 16, 32 tokeniser = AutoTokenizer.from_pretrained(MODEL_DIR) if tokeniser.pad_token is None: tokeniser.pad_token = tokeniser.eos_token def pick_subset(n: int, seed: int) -> list[str]: """n pool invoices, spread evenly over the 24 format x language groups.""" answers = json.loads((POOL_DIR / "answers.json").read_text(encoding="utf-8")) assert n <= len(answers), f"pool only has {len(answers)} invoices" groups = defaultdict(list) for filename, info in sorted(answers.items()): groups[(info["format"], info["language"])].append(filename) rng = random.Random(seed) keys = sorted(groups) for key in keys: rng.shuffle(groups[key]) rng.shuffle(keys) # with n < 24, which groups get the spare places varies by seed  picked = [] while len(picked) < n: for key in keys: if groups[key] and len(picked) < n: picked.append(groups[key].pop()) return picked def build_example(prompt: str, answer: str) -> dict: """Tokenise one pair; labels of -100 mean the prompt is read but not graded.""" prompt_text = tokeniser.apply_chat_template( [{"role": "user", "content": prompt}], tokenize=False, add_generation_prompt=True, ) full_text = prompt_text + answer + tokeniser.eos_token prompt_ids = tokeniser(prompt_text, add_special_tokens=False)["input_ids"] full_ids = tokeniser(full_text, add_special_tokens=False)["input_ids"][:MAX_LENGTH] labels = list(full_ids) for i in range(min(len(prompt_ids), len(labels))): labels[i] = -100 return {"input_ids": full_ids, "labels": labels} def collate(batch: list[dict]) -> dict: """Pad each batch to its longest sequence; padding is neither learned nor attended.""" longest = max(len(item["input_ids"]) for item in batch) input_ids, labels, attention = [], [], [] for item in batch: pad = longest - len(item["input_ids"]) input_ids.append(item["input_ids"] + [tokeniser.pad_token_id] * pad) labels.append(item["labels"] + [-100] * pad) attention.append([1] * len(item["input_ids"]) + [0] * pad) return { "input_ids": torch.tensor(input_ids), "labels": torch.tensor(labels), "attention_mask": torch.tensor(attention), } def train_adapter(files: list[str], seed: int, out_dir: Path) -> dict: """Train one LoRA adapter on the given pool invoices; return its cost.""" answers = json.loads((POOL_DIR / "answers.json").read_text(encoding="utf-8")) examples = [ build_example( build_ft((POOL_DIR / f).read_text(encoding="utf-8")), json.dumps(answers[f]["ground_truth"], ensure_ascii=False), ) for f in files ] set_seed(seed) # before get_peft_model, so the adapter's starting values repeat too  model = AutoModelForCausalLM.from_pretrained(MODEL_DIR, dtype=torch.bfloat16).to( "cuda" ) model = get_peft_model( model, LoraConfig( r=RANK, lora_alpha=ALPHA, target_modules=["q_proj", "k_proj", "v_proj", "o_proj"], lora_dropout=0.05, task_type="CAUSAL_LM", ), ) model.print_trainable_parameters() trainer = Trainer( model=model, args=TrainingArguments( output_dir=str(out_dir), num_train_epochs=EPOCHS, per_device_train_batch_size=4, learning_rate=2e-4, logging_steps=10, save_strategy="no", report_to="none", bf16=True, seed=seed, ), train_dataset=Dataset.from_list(examples), data_collator=collate, ) torch.cuda.reset_peak_memory_stats() start = time.perf_counter() trainer.train() minutes = (time.perf_counter() - start) / 60 peak_gb = torch.cuda.max_memory_allocated() / 1024**3 model.save_pretrained(out_dir) adapter_mb = (out_dir / "adapter_model.safetensors").stat().st...