How to fine-tune an embedding model with LoRA adapters

To fine-tune an embedding model with LoRA, freeze the base encoder, attach a small LoRA adapter to its linear layers, and train only the adapter with a contrastive loss on (query, relevant passage) pairs from your own domain. With Sentence Transformers this is a few lines: build a LoraConfig with task_type=FEATURE_EXTRACTION, call model.add_adapter(), and train with MultipleNegativesRankingLoss. A few hundred to a few thousand clean pairs is usually enough. The adapter comes out at a few megabytes. Afterwards you must re-embed your whole corpus, because the fine-tuned model's vectors are not comparable with the old ones.

Why LoRA fits embedding models

Full fine-tuning updates every weight and leaves you with a second full copy of the model. LoRA freezes the weight matrix W and learns a low-rank correction B·A on top of it, scaled by alpha / r. Only A and B train.

For an encoder that matters in three ways:

• Small files. The Sentence Transformers docs show a LoRA adapter for bert-base-uncased at 9.44 MB, 2.14% of the base model's weights.
• Less forgetting. The base weights never change, so the model keeps most of what it knew before.
• Many domains, one base. You can keep one encoder loaded and switch adapters per tenant, language or domain.

What your training data looks like

An embedding model learns from pairs, not from next-token prediction. Each row is an anchor (usually a query) and a positive (the passage that should come back for it). A hard negative, something that looks relevant and is not, is optional and often helps.

Good sources of pairs: search logs with clicks, support tickets matched to the doc that resolved them, and headings matched to their sections. Clean pairs matter more than volume.

The code

This uses Sentence Transformers 3 or later with peft installed (pip install sentence-transformers peft datasets).

from datasets import Dataset from peft import LoraConfig, TaskType from sentence_transformers import ( SentenceTransformer, SentenceTransformerTrainer, SentenceTransformerTrainingArguments, ) from sentence_transformers.losses import MultipleNegativesRankingLoss # Load the base encoder in fp32 for training model = SentenceTransformer( "BAAI/bge-base-en-v1.5", model_kwargs={"torch_dtype": "float32"}, ) # Attach a LoRA adapter; only these weights train peft_config = LoraConfig( task_type=TaskType.FEATURE_EXTRACTION, inference_mode=False, r=16, lora_alpha=32, lora_dropout=0.1, ) model.add_adapter(peft_config) # Your (anchor, positive) pairs; add a "negative" column if you have hard negatives train = Dataset.from_dict({ "anchor": ["torque spec for M8 flange bolt", "refund window for annual plans"], "positive": [ "M8 flange bolts on the intake manifold are tightened to 25 Nm.", "Annual plans can be refunded in full within 30 days of purchase.", ], }) args = SentenceTransformerTrainingArguments( output_dir="bge-base-lora", num_train_epochs=1, per_device_train_batch_size=32, learning_rate=2e-4, warmup_ratio=0.1, ) trainer = SentenceTransformerTrainer( model=model, args=args, train_dataset=train, loss=MultipleNegativesRankingLoss(model), ) trainer.train() model.save_pretrained("bge-base-lora-adapter") 

MultipleNegativesRankingLoss treats every other positive in the batch as a negative for each anchor, so larger batches give it more to learn from. LoRA trains comfortably at a higher learning rate than full fine-tuning; 2e-4 is a common starting point.

To use the result, load the adapter folder like any model: SentenceTransformer("bge-base-lora-adapter"). You can also load the base model and call model.load_adapter(...). The Sentence Transformers PEFT docs cover both.

Settings that matter

• Rank and alpha. r=16 with lora_alpha at r or 2r is a sound default. Raise r only when an evaluation shows the adapter is underfitting.
• Target modules. If you leave target_modules out, PEFT picks defaults for known architectures (for BERT, the attention query and value projections). Adding the other linear layers gives the adapter more capacity for a little more memory. The PEFT LoRA reference lists the options.
• Evaluation before and after. Hold out a few hundred of your own queries with known answers and measure recall@10 or nDCG@10 on the base model first. Sentence Transformers' InformationRetrievalEvaluator does this. If the base model already scores well, a better base model, hybrid search or a reranker may help more than fine-tuning.
• Forgetting. Check a general benchmark too. A large drop there means the learning rate was too high or you trained too long.

The step most tutorials skip: re-embedding

Fine-tuning changes the geometry of the vector space. Vectors from the base model and vectors from the fine-tuned model cannot be compared, so they cannot live in the same index. After training you re-embed every document with the new model, and every query has to use the same adapter.

On a large corpus, that re-embedding pass costs more than the training. Plan it like a migration: build the new index next to the old one, compare results on your evaluation set, then switch traffic.

Where Mixpeek fits

Disclosure: I work on Mixpeek.

Mixpeek stores and searches vectors you compute yourself, including ones from a fine-tuned encoder, next to the documents and metadata they came from. During a re-embed you can keep the old and new vectors in separate collections and compare them on the same queries before you switch. Two longer write-ups on our site go deeper: how LoRA fine-tuning works, including QLoRA and DoRA, and changing embedding models without breaking your index.