LoRA vs full fine-tuning: when is LoRA good enough?

LoRA is good enough for most fine-tuning: on normal-sized instruction and reasoning datasets, and in reinforcement learning, it matches full fine-tuning when it covers every layer and uses a higher learning rate. Nodus runs both on its nodus/sft training runtime, where the switch is one parameter, method: LoRA or method: Full. Full fine-tuning earns its roughly 7.7 times larger memory bill when you add more new knowledge than an adapter can hold, as in continued pretraining on billions of tokens.

What does the research say about LoRA vs full fine-tuning?

The two best-known comparisons agree once you look at dataset size. The LoRA paper introduced the method: freeze the pretrained weights and train small low-rank matrices beside them. Later work tested where that shortcut stops being free.

LoRA Learns Less and Forgets Less (Biderman et al., 2024) compared the two on code and math. It used about 100K instruction pairs and, separately, 20B tokens of continued pretraining. In standard low-rank settings, LoRA substantially underperformed full fine-tuning on the target domain. It also kept more of the base model's performance outside that domain.

The authors found full fine-tuning learns weight changes with a rank 10 to 100 times greater than typical LoRA settings.

LoRA Without Regret from Thinking Machines ran supervised and RL experiments with a learning-rate sweep for every setting. On small-to-medium instruction-tuning and reasoning datasets, LoRA performed the same as full fine-tuning. It fell behind once the dataset exceeded the adapter's capacity. In RL it matched full fine-tuning even at small ranks.

Two details decide which result you get. Thinking Machines found attention-only LoRA underperforms, so apply it to all weight matrices, especially the MLP layers. They also found the best LoRA learning rate is consistently about 10 times the best full fine-tuning rate.

How much more memory does full fine-tuning need?

Full fine-tuning with mixed-precision Adam needs about 16 bytes for every parameter, before activations. The ZeRO paper breaks it down: 2 bytes for the 16-bit weights, 2 for the gradients, then 4 each for an fp32 copy of the weights, Adam's momentum and its variance.

LoRA pays those 16 bytes only for the adapter. The frozen base costs its 2 bytes per parameter and nothing else. We sized a rank-16 adapter on all seven linear projections of each layer, from the layer sizes in the Qwen3-8B and Qwen3-32B configs. Each projection adds 16 x (input size + output size) parameters.

Parameter counts from the Hugging Face model pages; layer sizes from each model's config.json. Totals use exact counts, so rows may not add up after rounding.

Optimizer state is what makes full fine-tuning big: a LoRA run never pays those 12 bytes per parameter for the frozen base.

The ratio is about 7.7 times at both sizes. The adapter is only 0.53% of Qwen3-8B's parameters and 0.41% of Qwen3-32B's. Activations come on top for both methods and grow with sequence length and batch size.

What GPUs does each method need, and what do they cost per hour?

Memory decides how many GPUs you need. By our math, a LoRA run on Qwen3-8B fits on one 48 GB card. Full fine-tuning of the same model is a multi-GPU job: the nodus/sft runtime reference ships a Full preset for Qwen3-8B on 8 x H100.

At 32B, LoRA's 67.7 GB fits one H200 with room for activations. Full fine-tuning needs 524.2 GB sharded across GPUs, which the runtime supports through its distributed.strategy parameter (FSDP, ZeRO1 to ZeRO3). Eight H200s give 1,128 GB, enough headroom for activations.

Prices checked November 12, 2026: RTX A6000 listed at $0.333/hr, H100 at $2.60/hr and H200 at $3.593/hr on the Nodus pricing page.

At 8B the full fine-tuning setup costs about 62 times more per hour; at 32B, on the same GPU type, 8 times more.

Does full fine-tuning cost 8 times more per job?

No. The hourly gap overstates the job gap, because eight GPUs also finish the work sooner. What you pay for is GPU-hours, and those follow compute.

Thinking Machines calculates that LoRA takes slightly more than two thirds of the FLOPs of full fine-tuning per pass. The backward pass skips the gradient for the frozen weights. So on the same GPU type, at the same utilization, full fine-tuning the same data needs about 1.5 times the GPU-hours.

Here is the 32B case worked through. Suppose LoRA on one H200 takes 10 hours: 10 x $3.593 = $35.93. Full fine-tuning needs about 15 H200-hours of compute, which on 8 x H200 is 15 / 8 = 1.9 hours of wall clock. That costs 15 x $3.593 = $53.90.

Full fine-tuning costs about 1.5 times as much per job here, not 8 times, and it finishes sooner.

Treat $53.90 as a floor. It assumes perfect scaling across eight GPUs, and sharded training loses some time to communication. We have not benchmarked this pair on Nodus, so we will not quote the overhead.

The bigger multipliers sit outside the single run. Every learning-rate sweep pays the 1.5 times again. Every full fine-tune also produces a whole model: a 16-bit Qwen3-8B is 16.4 GB, while its rank-16 adapter is about 87 MB at 2 bytes per parameter. On Nodus, the outputs land under model/ for full fine-tuning and adapter/ for LoRA, per the training guide.

When is full fine-tuning worth the GPUs?

Full fine-tuning is worth it when the job is closer to pretraining than to post-training. Our rule, built from the two studies above:

1. Start with LoRA on all linear layers, at about 10 times your full fine-tuning learning rate. This covers instruction tuning, format and style changes, and reasoning datasets of normal post-training size.
2. Use LoRA for RL. Thinking Machines found RL needs very little capacity, so even small ranks matched full fine-tuning.
3. Switch to full fine-tuning when you are teaching a lot of new knowledge, such as billions of tokens of domain text. That is where Biderman et al. measured LoRA falling well short.
4. Switch when a properly configured LoRA run still trails on a held-out eval you care about. Try a higher rank first, since capacity is the usual limit.
5. Stay on LoRA if keeping general abilities matters more than the last points on the target task. Biderman et al. found it forgets less, and our post on catastrophic forgetting after fine-tuning covers how to measure that on your own model.

Also stay on LoRA if you plan to serve many variants of one base. Thinking Machines notes that inference engines such as vLLM and SGLang can batch requests across many adapters on one server.

How do you switch between LoRA and full fine-tuning on Nodus?

Change method and the learning rate, and let the presets pick the GPUs. The training guide says to leave out the lora block for full fine-tuning, and that presets for the Full method pick larger GPUs. A LoRA job from the guide looks like this:

apiVersion: nodus.dev/v1beta1 kind: TrainingJob metadata: name: support-sft spec: runtime: nodus/sft model: uri: hf://Qwen/Qwen3-0.6B@c1899de289a04d12100db370d81485cdf75e47ca secret: hf-token # only for gated repositories  data: volume: support-chats # a Volume holding chats.jsonl  path: chats.jsonl format: JSONL columns: {prompt: question, completion: answer} parameters: steps: 2000 lora: {r: 16, alpha: 32} maxCostUSD: "20.00" 

For a full fine-tune, replace the lora line with method: Full and set learningRate yourself. The runtime's default learningRate is 0.0002, a typical LoRA rate. Going by the 10 times finding, start full fine-tuning nearer 0.00002 and sweep around it.

Run nodus apply -f sft.yaml --dry-run -o yaml first. The dry run shows the resources the preset picked and a cost estimate before anything is charged. Keep maxCostUSD on full runs too, since they are the ones that grow.

If the 16-bit base does not fit your budget at all, the next step down is QLoRA, covered with its own break-even rule in LoRA vs QLoRA.

FAQ

What is better than LoRA?

For large, knowledge-heavy datasets, full fine-tuning learns more, per Biderman et al. For a base model that is too big for your GPU, QLoRA keeps LoRA's adapters on a 4-bit base. For most post-training data, a well-configured LoRA already matches full fine-tuning, so the better move is usually fixing its settings: all layers, a higher learning rate, enough rank.

Is full fine-tuning faster than LoRA?

Per GPU, no. LoRA does about two thirds of the compute per pass. A full fine-tune spread over more GPUs can finish sooner in wall-clock time, but it costs more GPU-hours.

Can I merge a LoRA adapter into the base model?

Yes. The LoRA paper describes computing W0 + BA once for deployment, so the model then adds no inference latency, and subtracting BA recovers the base for another task. Keep the adapter unmerged if you want to serve several variants from one base.

Does Nodus support full fine-tuning?

Yes, in Beta. The nodus/sft runtime accepts Full, LoRA and QLoRA for its method parameter, with Full presets from one GPU for Qwen3-0.6B up to 8 x H100 for Qwen3-8B. Pricing for every GPU is on the Nodus pricing page.

Originally published on the Nodus blog.