Let a small model critique and revise its own answers, then fine-tune on the revisions (on a rented GPU)

Short answer: write a short list of principles and a set of questions, have a small model answer them, critique its own answers against your principles and revise them, filter the revisions with a script and read a sample yourself, train a LoRA adapter on the result with LLaMA-Factory, then compare the base model and the adapter on questions you held back. Everything runs on one rented GPU; nothing is sent to a third-party API, and the guide starts no server.

Disclosure: I work on CloudGPU, and the GPU rental in this guide is ours. Each step links the documentation it follows and ends with a "What to check" line, because versions, template names and model access change.

Where the idea comes from, and what this guide does not claim

Anthropic's Constitutional AI work (2022-12-15) describes a supervised phase in which the model samples answers, critiques them against a list of principles, revises them, and is then fine-tuned on the revisions: research page, paper. OpenAI's deliberative alignment post (2024-12-20) says its training data is generated automatically from written specifications, without human-labeled completions: post.

This guide is a small, single-developer version of that pattern. It is our own adaptation, not a result from those sources, and we make no claim that the adapter you train will be better than the base model. A model's critique of its own answer can be wrong, and a revision can be worse than the original. That is why the last step is a comparison you read yourself, with a rule for when to throw the round away.

Read these things first

Your base model's license. You will train on text a base model wrote, so the license of that model matters. The walkthrough uses Mistral 7B Instruct v0.3, whose model card lists the Apache-2.0 license (read 2026-10-27). Two other small models have conditions that touch this exact workflow. The Llama 3.2 Community License says that if you use the Llama Materials "or any outputs or results of the Llama Materials to create, train, fine tune, or otherwise improve an AI model, which is distributed or made available", the model name must begin with "Llama". The Gemma Terms of Use define Model Derivatives to include models created by "methods based on the generation of synthetic data Outputs by Gemma for training that model". Read the current text of the license of whichever model you use; this is an excerpt, not legal advice.
Your data. The questions you write must be ones you have the right to use. CloudGPU's acceptable use policy (section 1.8) prohibits training on datasets you do not have rights to use. Also read the terms.
Labelling. This guide generates data with your own model, not with an API. If you would rather use an API as the generator, the guide on generating a training set with DeepSeek, from this same series, uses DeepSeek only. DeepSeek's Terms of Use (Last Update: March 27, 2026) say in section 3.1 that if you publish or disseminate outputs, you must clearly indicate that the content is generated by artificial intelligence. If you publish any dataset or example from this guide, label it AI-generated as well.

What you need

A CloudGPU account with balance. GPU instances are billed per minute, rounded up, with a 1-minute minimum, and billing stops when you destroy the instance (terms). Rates are on the pricing page. Payment is by card or USDT (payments).
A text editor and a terminal on your own machine. You write three small files there before you rent anything.

What to check: you can open a terminal and run ssh -V and scp (both print a usage line or version).

Step 1: write the principles and the questions (on your own machine)

Create three plain-text files in one folder.

principles.txt is a short list, in your own words, of what a good answer looks like for your use case. An example to replace with yours:

1. Answer only what the question asks. 2. If the answer depends on information you were not given, say what is missing instead of guessing. 3. Do not invent names, numbers, links or quotes. 4. Keep the answer under 150 words unless the question asks for more.

prompts.txt is the training questions, one per line, as many as you can write (a few hundred is a reasonable start; the more varied they are, the more the loop has to work with). heldout.txt is 20 or so questions in the same style that you will not use for training. Write them separately, and do not copy lines between the files.

What to check: wc -l prompts.txt heldout.txt shows both files non-empty, and comm -12 <(sort prompts.txt) <(sort heldout.txt) prints nothing (no question appears in both).

Step 2: rent a GPU

On Deploy, choose a card. The CloudGPU docs say two things that matter here: every template is an image with root SSH, and for anything a template does not cover you can rent a raw GPU (Deploy → Raw GPU) and install what you need over SSH. For this guide, pick a template that includes PyTorch, or choose Raw GPU and install it yourself. A 7B model in 16-bit needs a card with enough memory for the weights plus the generation cache; nvidia-smi shows real use, and a smaller model or a card with more memory is the fix for an out-of-memory error.

Connect with the SSH details of your running instance. Then upload your three files (scp takes the port with a capital -P):

scp -P <ssh-port> principles.txt prompts.txt heldout.txt <ssh-user>@<ssh-host>:/workspace/

What to check: connect with ssh -p <ssh-port> <ssh-user>@<ssh-host>, then ls /workspace lists your three files, nvidia-smi lists a GPU, and python3 -c "import torch; print(torch.cuda.is_available())" prints True. If torch is missing, install PyTorch with CUDA from the PyTorch install page first.

Step 3: install LLaMA-Factory

These are the install commands from the LLaMA-Factory README (read 2026-10-27; the repository name is spelled LlamaFactory there). The repository's pyproject.toml lists torch, transformers, accelerate and peft as dependencies, which is all the scripts below use. The docs FAQ lists a LLaMA-Factory template, but don't count on it being installed or current: check, and install if needed.

cd /workspace git clone --depth 1 https://github.com/hiyouga/LlamaFactory.git cd LlamaFactory pip install -e . pip install -r requirements/metrics.txt

What to check: llamafactory-cli help prints the command list, and python3 -c "import transformers, peft; print(transformers.__version__, peft.__version__)" prints two versions without an error.

Step 4: generate, critique, revise

Two files. common.py loads the base model and answers a list of prompts in batches (the transformers docs say to pad on the left for batched generation, and the chat templating docs describe apply_chat_template):

# /workspace/common.py import torch from transformers import AutoModelForCausalLM, AutoTokenizer MODEL = "mistralai/Mistral-7B-Instruct-v0.3" def load(): tok = AutoTokenizer.from_pretrained(MODEL, padding_side="left") if tok.pad_token is None: tok.pad_token = tok.eos_token model = AutoModelForCausalLM.from_pretrained( MODEL, torch_dtype=torch.bfloat16, device_map="auto" ) model.eval() return tok, model def chat(tok, model, prompts, max_new_tokens=400, sample=True, batch=8): outs = [] for i in range(0, len(prompts), batch): texts = [ tok.apply_chat_template( [{"role": "user", "content": p}], tokenize=False, add_generation_prompt=True, ) for p in prompts[i : i + batch] ] enc = tok(texts, return_tensors="pt", padding=True, add_special_tokens=False) enc = enc.to(model.device) kw = {"do_sample": True, "temperature": 0.7, "top_p": 0.9} if sample else {"do_sample": False} with torch.no_grad(): gen = model.generate( **enc, max_new_tokens=max_new_tokens, pad_token_id=tok.pad_token_id, **kw ) for row in gen[:, enc["input_ids"].shape[1]:]: outs.append(tok.decode(row, skip_special_tokens=True).strip()) return outs

loop.py runs the three passes: answer, critique against your principles, revise. Principles go inside the user message so the same scripts work with chat templates that have no system role.

# /workspace/loop.py import json from common import load, chat NONE = "NONE" principles = open("/workspace/principles.txt", encoding="utf-8").read().strip() prompts = [l.strip() for l in open("/workspace/prompts.txt", encoding="utf-8") if l.strip()] tok, model = load() answers = chat(tok, model, prompts) critique_in = [ "Principles:\n" + principles + "\n\nQuestion:\n" + p + "\n\nAnswer:\n" + a + "\n\nIn at most three short bullet points, say where the answer breaks a principle. " "If it breaks none, reply with exactly " + NONE + "." for p, a in zip(prompts, answers) ] critiques = chat(tok, model, critique_in, max_new_tokens=200) todo = [i for i, c in enumerate(critiques) if c.strip().strip(".").upper() != NONE] revise_in = [ "Principles:\n" + principles + "\n\nQuestion:\n" + prompts[i] + "\n\nAnswer:\n" + answers[i] + "\n\nProblems found:\n" + critiques[i] + "\n\nRewrite the answer so that it follows the principles and fixes these problems. " "Reply with the rewritten answer only." for i in todo ] revised = chat(tok, model, revise_in) revised_by_index = dict(zip(todo, revised)) with open("/workspace/loop_raw.jsonl", "w", encoding="utf-8") as f: for i in range(len(prompts)): rec = {"prompt": prompts[i], "answer": answers[i], "critique": critiques[i], "revised": revised_by_index.get(i)} f.write(json.dumps(rec, ensure_ascii=False) + "\n") with open("/workspace/loop_revised.jsonl", "w", encoding="utf-8") as f: for i in todo: rec = {"prompt": prompts[i], "answer": answers[i], "critique": critiques[i], "revised": revised_by_index[i]} f.write(json.dumps(rec, ensure_ascii=False) + "\n") print("prompts:", len(prompts), "revised rows:", len(todo))

Run it from /workspace:

cd /workspace python3 loop.py

If your transformers version prints a warning that torch_dtype is deprecated, replace it with dtype in common.py.

What to check: the script ends by printing the number of prompts and revised rows. Open loop_revised.jsonl and read five rows yourself: does the critique point at a real problem, and is the revised answer actually better than the first one? If critiques are vague or revisions are not better, stop here and change your principles or your prompts before you go on.

Step 5: filter and read a sample

The loop writes everything the model said, including its mistakes. Filter before you train. This script drops rows where the revision is empty, identical to the first answer, very long, repeated, or contains words that show the model talked about the task instead of answering:

# /workspace/filter.py import json import random LEAK = ("principles", "critique", "rewritten", "revised answer", "problems found") rows, seen = [], set() for line in open("/workspace/loop_revised.jsonl", encoding="utf-8"): r = json.loads(line) rev = (r["revised"] or "").strip() if not rev or rev == r["answer"].strip(): continue if len(rev) > 2500 or len(rev) < 20: continue low = rev.lower() if any(w in low for w in LEAK): continue key = " ".join(low.split()) if key in seen: continue seen.add(key) rows.append({"instruction": r["prompt"], "input": "", "output": rev}) json.dump(rows, open("/workspace/selfloop_sft.json", "w", encoding="utf-8"), ensure_ascii=False, indent=2) random.seed(1) with open("/workspace/sample_to_read.md", "w", encoding="utf-8") as f: for r in random.sample(rows, min(30, len(rows))): f.write("## " + r["instruction"] + "\n\n" + r["output"] + "\n\n") print("kept:", len(rows))

python3 filter.py

Then read sample_to_read.md yourself. Edit selfloop_sft.json by hand and remove any row you would not want the model to learn from, or tighten the filter and run it again. If you publish the filtered file or examples from it, label them as AI-generated.

What to check: the script prints how many rows it kept. If that number is very small, the loop is not producing usable revisions for your prompts; go back to Step 1. If you read the sample and many rows are wrong or off-topic, do not train on them.

Step 6: register the data and train a LoRA adapter

Copy the dataset into LLaMA-Factory's data/ folder and register it in data/dataset_info.json, which is how LLaMA-Factory finds custom datasets:

cd /workspace/LlamaFactory cp /workspace/selfloop_sft.json data/ python3 - <<'EOF' import json p = "data/dataset_info.json" info = json.load(open(p, encoding="utf-8")) info["selfloop_sft"] = { "file_name": "selfloop_sft.json", "columns": {"prompt": "instruction", "query": "input", "response": "output"}, } json.dump(info, open(p, "w", encoding="utf-8"), ensure_ascii=False, indent=2) EOF

Save this as selfloop_lora.yaml in the LlamaFactory folder. The keys follow the official LoRA example in the repo's examples/train_lora/ folder; the model, template and dataset lines are the parts you change (the mistral template name is from the README's model table):

### model model_name_or_path: mistralai/Mistral-7B-Instruct-v0.3 ### method stage: sft do_train: true finetuning_type: lora lora_rank: 8 lora_target: all ### dataset dataset: selfloop_sft template: mistral cutoff_len: 2048 preprocessing_num_workers: 4 ### output output_dir: saves/mistral7b/lora/selfloop logging_steps: 10 save_steps: 100 plot_loss: true overwrite_output_dir: true ### train per_device_train_batch_size: 1 gradient_accumulation_steps: 8 learning_rate: 1.0e-4 num_train_epochs: 3.0 lr_scheduler_type: cosine warmup_ratio: 0.1 bf16: true ### eval val_size: 0.1 per_device_eval_batch_size: 1 eval_strategy: steps eval_steps: 50

llamafactory-cli train selfloop_lora.yaml

Compare your YAML with the current LoRA example in the LLaMA-Factory examples folder: key names change between releases.

What to check: the log prints a training loss every 10 steps and an evaluation loss every 50; saves/mistral7b/lora/selfloop contains adapter_model.safetensors and adapter_config.json. Make sure no earlier script is still holding the GPU (nvidia-smi shows what holds memory). If you choose another base model, change MODEL in common.py, model_name_or_path, and template together, and set a Hugging Face token for a gated model.

Step 7: compare on questions the model never saw

compare.py answers your held-out questions with the base model and with the adapter, greedily (no sampling, so the two columns are comparable), and writes both next to each other:

# /workspace/compare.py from peft import PeftModel from common import load, chat ADAPTER = "/workspace/LlamaFactory/saves/mistral7b/lora/selfloop" qs = [l.strip() for l in open("/workspace/heldout.txt", encoding="utf-8") if l.strip()] tok, model = load() base = chat(tok, model, qs, sample=False) tuned_model = PeftModel.from_pretrained(model, ADAPTER) tuned_model.eval() tuned = chat(tok, tuned_model, qs, sample=False) with open("/workspace/compare.md", "w", encoding="utf-8") as f: for q, b, t in zip(qs, base, tuned): f.write("## " + q + "\n\n**Base:**\n\n" + b + "\n\n**Adapter:**\n\n" + t + "\n\n") print("questions:", len(qs))

cd /workspace python3 compare.py

Read compare.md against your principles, question by question. Decide for yourself what "better" means, and write it down before you read. Stop rule: if the adapter is not clearly better on your principles, do not keep it. Change the principles, the prompts or the filter and run the loop again; each round uses more GPU minutes, so decide in advance when to stop.

What to check: the script prints the number of questions and compare.md exists with two answers per question. Remember that 20 questions is a small test, and that you wrote the principles that decide the outcome.