Fine-Tuning LLMs on Consumer Hardware: LoRA Guide 2024

Train custom models on your GPU using LoRA/QLoRA — no cloud credits needed.

Introduction

Fine-tuning used to require A100s and thousands of dollars. With LoRA (Low-Rank Adaptation) and QLoRA (Quantized LoRA), you can fine-tune 7B-70B models on a single 24GB GPU (RTX 3090/4090) or even 12GB with quantization.

What is LoRA?

LoRA freezes the base model and trains only small adapter matrices:

Base Model (frozen) + LoRA Adapters (trainable) = Fine-tuned Model

Benefits:

  • 100-1000x fewer trainable parameters
  • Fast training (minutes to hours)
  • Small checkpoint files (MBs not GBs)
  • Mergeable back into base model

Hardware Requirements

Model Size Method VRAM Needed GPU Example
7B LoRA (fp16) ~16 GB RTX 3090/4090
7B QLoRA (4-bit) ~8 GB RTX 3060 12GB
13B QLoRA (4-bit) ~12 GB RTX 3080 12GB
33B QLoRA (4-bit) ~20 GB RTX 3090/4090
70B QLoRA (4-bit) ~40 GB Dual 3090/4090

Setup Environment 

# Create environment conda create -n lora python=3.10 -y conda activate lora # Install dependencies pip install torch torchvision torchaudio --index-url https://download.pytorch.org/whl/cu121 pip install transformers peft trl accelerate bitsandbytes datasets wandb

Prepare Dataset 

// data/train.jsonl {"text": "### Instruction:\nWrite a Python function to reverse a string\n### Response:\ndef reverse_string(s):\n return s[::-1]"} {"text": "### Instruction:\nExplain what a decorator does in Python\n### Response:\nA decorator is a function that takes another function and extends its behavior..."}

Format: Alpaca/ShareGPT style with ### Instruction: and ### Response: markers.

Training Script (QLoRA) 

# train_lora.py import os import torch from datasets import load_dataset from transformers import ( AutoModelForCausalLM, AutoTokenizer, BitsAndBytesConfig, TrainingArguments ) from peft import LoraConfig, get_peft_model, prepare_model_for_kbit_training from trl import SFTTrainer # Config MODEL_ID = "mistralai/Mistral-7B-v0.1" # or "meta-llama/Llama-2-7b-hf" OUTPUT_DIR = "./mistral-7b-code-lora" DATA_PATH = "data/train.jsonl" # 4-bit quantization config bnb_config = BitsAndBytesConfig( load_in_4bit=True, bnb_4bit_quant_type="nf4", bnb_4bit_compute_dtype=torch.bfloat16, bnb_4bit_use_double_quant=True ) # Load model model = AutoModelForCausalLM.from_pretrained( MODEL_ID, quantization_config=bnb_config, device_map="auto", trust_remote_code=True ) model = prepare_model_for_kbit_training(model) # LoRA config lora_config = LoraConfig( r=64, # Rank (higher = more capacity)  lora_alpha=16, # Scaling factor  target_modules=["q_proj", "k_proj", "v_proj", "o_proj", "gate_proj", "up_proj", "down_proj"], lora_dropout=0.05, bias="none", task_type="CAUSAL_LM" ) model = get_peft_model(model, lora_config) model.print_trainable_parameters() # Trainable params: ~0.5% of total  # Tokenizer tokenizer = AutoTokenizer.from_pretrained(MODEL_ID, trust_remote_code=True) tokenizer.pad_token = tokenizer.eos_token tokenizer.padding_side = "right" # Dataset dataset = load_dataset("json", data_files=DATA_PATH, split="train") # Training arguments training_args = TrainingArguments( output_dir=OUTPUT_DIR, per_device_train_batch_size=2, gradient_accumulation_steps=4, warmup_steps=10, max_steps=100, # Adjust for your dataset  learning_rate=2e-4, fp16=True, logging_steps=5, save_strategy="steps", save_steps=50, report_to="wandb", # or "none"  optim="paged_adamw_8bit" ) # Trainer trainer = SFTTrainer( model=model, train_dataset=dataset, dataset_text_field="text", max_seq_length=512, tokenizer=tokenizer, args=training_args, packing=False ) # Train trainer.train() # Save trainer.model.save_pretrained(OUTPUT_DIR) tokenizer.save_pretrained(OUTPUT_DIR) print(f"Model saved to {OUTPUT_DIR}")

Run Training 

# Single GPU python train_lora.py # Multi-GPU (if available) accelerate launch --multi_gpu train_lora.py

Merge & Export 

# merge_lora.py from peft import PeftModel from transformers import AutoModelForCausalLM, AutoTokenizer import torch BASE_MODEL = "mistralai/Mistral-7B-v0.1" LORA_PATH = "./mistral-7b-code-lora" OUTPUT_PATH = "./mistral-7b-code-merged" # Load base model = AutoModelForCausalLM.from_pretrained( BASE_MODEL, torch_dtype=torch.float16, device_map="auto" ) # Load & merge LoRA model = PeftModel.from_pretrained(model, LORA_PATH) model = model.merge_and_unload() # Save merged model.save_pretrained(OUTPUT_PATH) tokenizer = AutoTokenizer.from_pretrained(BASE_MODEL) tokenizer.save_pretrained(OUTPUT_PATH)

Convert to GGUF (llama.cpp) 

# Install llama.cpp git clone https://github.com/ggerganov/llama.cpp cd llama.cpp && make # Convert python convert-hf-to-gguf.py ../mistral-7b-code-merged \ --outfile ../mistral-7b-code.Q4_K_M.gguf \ --outtype q4_k_m

Inference with Ollama 

# Create Modelfile cat > Modelfile << 'EOF' FROM ./mistral-7b-code.Q4_K_M.gguf TEMPLATE "{{ .System }}\n\n{{ .Prompt }}" PARAMETER temperature 0.7 PARAMETER top_p 0.9 EOF # Create model ollama create mistral-code -f Modelfile # Test ollama run mistral-code "Write a Python decorator for timing functions"

Cost Breakdown (RTX 3090 24GB)

Item Cost
Electricity (4 hrs @ 350W) ~$0.20
GPU depreciation ~$1.00
Total per run ~$1.20

vs. Cloud (A100 40GB): $1.50-3.00/hr → $6-12 for same training.

Tips for Success

  1. Dataset quality > quantity — 500 clean examples beat 5000 noisy ones
  2. Start small — 100 steps, check loss, then scale
  3. Monitor VRAM — Reduce batch size if OOM
  4. Use gradient checkpointing — model.gradient_checkpointing_enable()
  5. Eval regularly — Keep a validation set

Common Issues

Error Fix
CUDA OOM Reduce batch size, increase grad accumulation, use 4-bit
Loss not decreasing Increase learning rate, check data format
NaN loss Lower learning rate, add gradient clipping
Slow training Enable flash attention, use packing=True

Conclusion

LoRA/QLoRA democratized LLM fine-tuning. With a consumer GPU, you can create domain-specialized models for coding, writing, analysis — whatever your use case. Start with 7B QLoRA, iterate on data, and deploy locally via Ollama or llama.cpp.

Next: Try DPO/ORPO for preference alignment, or merge multiple LoRAs for multi-task models.

Part of the LLM Fine-Tuning series on Dev.to