Fine-tuning usually ends with a choice. You train several versions of a model, with different seeds, learning rates, or data orders, evaluate them all, and keep the best one. The rest are discarded. Model soups challenge that habit. Instead of picking a single winner, you average the weights of several fine-tuned models and use the average as your model. Surprisingly often, the average beats every ingredient that went into it.
This is not the same as ensembling. An ensemble keeps several models and averages their predictions at inference time, paying the cost of running all of them. A model soup is a single model, built by averaging weights, so inference costs nothing extra. The fact that this works at all tells us something important about the geometry of fine-tuning.
The basic recipe
Start from one shared pretrained model. Fine-tune it several times, changing only things that do not change the starting point: the random seed, the data order, the learning rate, or a light hyperparameter. Each run gives a set of weights. The soup is their element-wise average.
If the fine-tuned weights are θ1,θ2,…,θk , the uniform soup is:
That is the whole method. No extra training, no extra data, no distillation step. A greedy variant builds the soup one model at a time, adding a candidate only if it improves validation accuracy, which protects against a bad ingredient spoiling the mix. In practice, the greedy version is the safer default, and the uniform version is a good first check.
import torch # state_dicts: list of fine-tuned checkpoints from the SAME pretrained start def make_soup(state_dicts): soup = {k: v.clone().float() for k, v in state_dicts[0].items()} for key in soup: for sd in state_dicts[1:]: soup[key] += sd[key].float() soup[key] /= len(state_dicts) return soup soup_sd = make_soup(fine_tuned_checkpoints) model.load_state_dict(soup_sd) Why averaging weights is not nonsense
Averaging arbitrary neural network weights is usually nonsense. Two networks trained from different random initialisations learn the same function in different internal coordinate systems: a neuron in one may correspond to a permuted neuron in the other, so averaging their weights mixes unrelated quantities. Fine-tuned models are different, because they share a starting point and stay close to it.
Fine-tuning from a shared pretrained model moves weights only a modest distance, and the different runs tend to stay in the same broad basin of the loss landscape. In that region, the loss surface is approximately flat along the directions that connect the solutions, a property often called linear mode connectivity. When solutions are linearly connected, the line segment between them stays in a low-loss region, and so does their midpoint. Averaging is then not mixing unrelated coordinates, it is interpolating between nearby solutions that already agree on most of the function.
A simple quadratic picture makes the intuition precise. Suppose the loss near a solution is approximately:
where θ is the optimum, and H is the Hessian. If each fine-tuned solution θi is a noisy estimate of a good region, centred near θ , then averaging reduces variance. For independent errors, the variance of the average shrinks roughly by a factor of k , the number of models, while the bias stays about the same. That variance reduction is the same reason ensembling helps, but here it happens in weight space rather than in prediction space, and it only works because the solutions are close enough for weight space to be meaningful.
There is a second effect. Different fine-tuning runs make different small errors: one overfits a little to one slice of data, another to a different slice. Averaging cancels part of those idiosyncratic errors, much like averaging noisy measurements cancels noise. The soup is therefore often flatter and more robust than any single run, which is one reason it tends to generalise better, especially under distribution shift.
When soups work, and when they fail
Soups work best when all ingredients satisfy three conditions.
First, a shared initialisation. Every ingredient must be fine-tuned from the same pretrained checkpoint. If you average models trained from scratch, or from different pretrained models, the permutation problem returns and the soup will usually be much worse than any ingredient. This is the most common failure, and it is easy to diagnose: the soup loss is far higher than the individual losses.
Second, similar training recipes. Ingredients should differ only in seed, data order, or modest hyperparameters. If one ingredient was trained with a very different objective, a very different number of steps, or a different architecture, it may sit in a different basin, and averaging pulls the soup out of both basins.
Third, comparable quality. A soup is not a rescue for a bad run. If one ingredient is clearly broken, uniform averaging lets it drag the rest down. Greedy souping handles this by testing each addition on a held-out set, and it is worth using whenever you have more than a handful of candidates.
A useful diagnostic is to check the soup on validation data before trusting it, and to compare it against both the best single model and a simple alternative such as the model trained longest. If the soup does not beat the best ingredient, keep the best ingredient. Soups are a cheap experiment, not a doctrine.
Soups, SWA, and ensembles: what is different
Stochastic weight averaging, or SWA, is a close cousin. SWA averages checkpoints along a single training trajectory, usually with a cyclical learning rate, to find a flatter point in the same basin. A model soup averages the endpoints of several independent trajectories. Both exploit flatness, but soups exploit diversity across runs, while SWA exploits diversity along one run. They can be combined: average within each run, then average across runs.
Ensembles are more powerful in principle, because they do not require weights to be averaged at all, so they can combine models from different basins. They are also more expensive at inference, since every model must run. Soups trade that generality for a single model at no extra inference cost. When the shared-initialisation condition holds, that trade is usually worth making. When it does not, an ensemble, or a distillation of the ensemble into one model, is the better tool.
There is also a practical point about calibration. Averaged weights often produce better calibrated predictions than a single sharp minimum, because the flatter region corresponds to a less overconfident fit. This is not guaranteed, and you should still check calibration on a held-out set, but it is a common side benefit, and it matters in settings where a probability is used for a decision, not just a label.
A practical workflow
A workflow that works well in practice looks like this.
- Fine-tune from one pretrained checkpoint with a small grid of seeds and learning rates, say five to ten runs, keeping everything else fixed.
- Evaluate each run on a held-out validation set and sort them by score.
- Build a greedy soup: start with the best run, then try adding each remaining run in order, keeping it only if validation accuracy improves.
- Evaluate the final soup on a separate test set, or at least on data not used for greedy selection, because greedy selection on the same set can overfit it slightly.
- If the soup wins, ship the soup. If it does not, ship the best single run and note that soups did not help this time.
Two pitfalls deserve emphasis. First, do not select ingredients on the test set, and do not report the soup's test score as if it were untouched if greedy selection used that set. Second, do not average checkpoints with different tokenisers, vocabularies, or layer layouts. Weight averaging assumes the same coordinates mean the same thing in every ingredient, and a shape mismatch is a sign that the assumption is broken, not a detail to work around by reshaping.
Limitations and common pitfalls
Soups do not fix a bad pretrained model, a bad objective, or bad data. They average out run-to-run noise, not systematic bias. If every ingredient makes the same mistake, the soup makes it too.
They also do not help much when fine-tuning moves weights a long way, for example when training from a pretrained model on a very different task for many steps. In that regime, different runs can leave the shared basin, and averaging becomes unreliable. A quick check is to interpolate between two ingredients and watch the validation loss along the line: if it stays low, soups are plausible; if it spikes in the middle, they are not.
Finally, soups are not a substitute for proper evaluation. Because they are so cheap, it is tempting to try many soup combinations and report the best. That is a form of selection overfitting, and it inflates the apparent gain. Keep the selection set and the reporting set separate, and treat a small soup gain as noise unless it holds up on fresh data.
Takeaway
Model soups are a reminder that the end of fine-tuning does not have to be a single winner. When several runs start from the same place and stay in the same basin, their average is a better, flatter, more robust model than any one of them, at no extra inference cost. The method is almost embarrassingly simple, which is exactly why it is worth trying before reaching for a more complex one: train a few variants, average their weights, and let validation data decide whether the soup or the best single model ships.
Connect with me on LinkedIn: linkedin.com/in/umeshtharukaofficial
