Capstone — fine-tuning with LoRA, QLoRA, and Unsloth

A startup in 2026 does not train a foundation model from scratch — the compute bill is in the eight figures and the data is gated. Instead, you take a pretrained open model (LLaMA-3 8B/70B, Mistral 7B, Gemma-2 9B, Qwen-2 7B, Phi-3) and fine-tune it on your task-specific data.

This is the most common deep learning job in industry by a wide margin: load a checkpoint, prepare your dataset, train, evaluate, deploy. The capstone walks the full pipeline — using a small open model so you can run everything on a single consumer GPU — but the shape of the work is identical for any production fine-tune.

Three flavors of fine-tuning in order of cost:

1. Full fine-tune: every weight is trainable. Maximum capacity, requires multi-GPU for 7B+. Best quality if you have the compute. 2. LoRA / QLoRA: train tiny adapter matrices, freeze the base model. ~1% of full-FT memory, ~95% of the quality on most tasks. The default. 3. Prompt / prefix tuning, soft prompts: train just a few learnable tokens prepended to the input. Cheapest, weakest. Niche.

For 99% of tasks in 2026, LoRA or QLoRA is the right choice. We'll spend the rest of the chapter on it.

Low-Rank Adaptation (LoRA) is built on a single observation: when you fine-tune a pretrained model, the *change* in any weight matrix is typically low-rank. So instead of updating the full d × d matrix W, factor the update as a product of two skinny matrices:

W_new = W + ΔW   where ΔW = B @ A,  A: r × d,  B: d × r,  r << d

Freeze W, train only A and B. If d = 4096 and r = 16, you trained 2 * d * r = 131,072 parameters instead of d² = 16,777,216 — a 128× reduction.

In PyTorch:

class LoRALinear(nn.Module):
    def __init__(self, base: nn.Linear, r=16, alpha=32):
        super().__init__()
        self.base = base
        for p in self.base.parameters(): p.requires_grad = False
        self.A = nn.Parameter(torch.empty(r, base.in_features))
        self.B = nn.Parameter(torch.zeros(base.out_features, r))
        nn.init.kaiming_uniform_(self.A, a=5**0.5)
        self.scale = alpha / r

    def forward(self, x):
        return self.base(x) + (x @ self.A.T @ self.B.T) * self.scale

Key hyperparameters:

• `r` (rank): 4-64. Higher = more capacity but more memory. 16 is a strong default.
• `alpha`: scaling factor. alpha = 2r is the convention; effectively a per-LoRA learning rate.
• Target modules: which layers to LoRA-fy. The big wins are q_proj, k_proj, v_proj, o_proj (attention) and the MLP gate_proj, up_proj, down_proj. Embedding layers usually frozen.
• Dropout: 0.05-0.1 inside the LoRA path.

LoRA is mergeable at inference: W_merged = W + B @ A * scale. No latency penalty in production.

QLoRA (2023) goes further: the frozen base model is quantized to 4-bit (NF4), while the LoRA adapters stay in bf16. Result: a 70B model that needs ~140GB in bf16 fits in ~35GB quantized — runnable on a single 48GB A6000 or two consumer 24GB cards.

Three pieces of magic:

1. NF4 (4-bit NormalFloat): a non-uniform 4-bit quantization optimized for the empirical distribution of pretrained weights (zero-mean Gaussian). Better quality than uniform int4 at the same bitwidth. 2. Double quantization: even the quantization constants are quantized, saving ~0.4 bits per weight. 3. Paged optimizer: NVIDIA Unified Memory swaps optimizer state between GPU and CPU during long sequences. Avoids OOM when sequence lengths spike.

In HuggingFace transformers + bitsandbytes + peft:

from transformers import AutoModelForCausalLM, BitsAndBytesConfig
from peft import LoraConfig, get_peft_model, prepare_model_for_kbit_training

bnb = BitsAndBytesConfig(
    load_in_4bit=True,
    bnb_4bit_quant_type='nf4',
    bnb_4bit_compute_dtype=torch.bfloat16,
    bnb_4bit_use_double_quant=True,
)
base = AutoModelForCausalLM.from_pretrained('meta-llama/Llama-3-8B', quantization_config=bnb, device_map='auto')
base = prepare_model_for_kbit_training(base)

lora = LoraConfig(r=16, lora_alpha=32, lora_dropout=0.05,
                  bias='none', task_type='CAUSAL_LM',
                  target_modules=['q_proj','k_proj','v_proj','o_proj','gate_proj','up_proj','down_proj'])
model = get_peft_model(base, lora)
model.print_trainable_parameters()   # e.g. 0.62% trainable

Quality cost: typically 1-2% on benchmark scores vs. full fine-tune. Often imperceptible on the task you actually care about. QLoRA is the production default for fine-tuning 7B-70B models in 2026.

Unsloth (https://unsloth.ai) is an open-source library that rewrites the hot paths of LoRA/QLoRA fine-tuning in fused Triton kernels. Same math, dramatically faster + less memory. Claims (consistent with community benchmarks):

• 2× faster training.
• ~70% less VRAM vs. HuggingFace + bitsandbytes.
• Manual backward pass for LoRA layers (skips the autograd graph entirely).
• Fused RoPE, fused RMSNorm, fused cross-entropy.
• Drop-in compatible with HF Trainer and TRL SFTTrainer / DPOTrainer.

The simplest possible Unsloth fine-tune:

from unsloth import FastLanguageModel
import torch

model, tokenizer = FastLanguageModel.from_pretrained(
    model_name='unsloth/llama-3-8b-bnb-4bit',   # pre-quantized model
    max_seq_length=2048,
    dtype=None,                                   # auto bf16/fp16
    load_in_4bit=True,
)

model = FastLanguageModel.get_peft_model(
    model, r=16, lora_alpha=32, lora_dropout=0.0,
    target_modules=['q_proj','k_proj','v_proj','o_proj','gate_proj','up_proj','down_proj'],
    use_gradient_checkpointing='unsloth',         # unsloth's optimized checkpointing
    random_state=42, use_rslora=False,
)

from trl import SFTTrainer
from transformers import TrainingArguments
from datasets import load_dataset

ds = load_dataset('yahma/alpaca-cleaned', split='train')

def format(row):
    return {'text': f'<|user|>\n{row["instruction"]}\n<|assistant|>\n{row["output"]}'}
ds = ds.map(format)

trainer = SFTTrainer(
    model=model, tokenizer=tokenizer,
    train_dataset=ds, dataset_text_field='text',
    max_seq_length=2048,
    args=TrainingArguments(
        per_device_train_batch_size=2,
        gradient_accumulation_steps=4,           # effective batch 8
        warmup_steps=10, max_steps=300,
        learning_rate=2e-4, bf16=True,
        logging_steps=10, optim='adamw_8bit',
        weight_decay=0.01, lr_scheduler_type='linear',
        seed=42, output_dir='out',
    ),
)
trainer.train()

model.save_pretrained_merged('out/llama3-instruct-merged', tokenizer, save_method='merged_16bit')
model.save_pretrained_gguf('out/llama3-instruct.gguf', tokenizer, quantization_method='q4_k_m')

That single block fits a Llama-3 8B fine-tune in ~8GB of VRAM — a free Colab T4 or any consumer card. Save as merged 16-bit for serving with vLLM, or as GGUF for Llama.cpp / Ollama / LM Studio.

Quality of data dominates everything else in fine-tuning. A pristine 1k-example dataset beats a noisy 100k one. Five rules:

1. Match the format to the chat template of the base model. LLaMA-3 uses <|begin_of_text|><|start_header_id|>user<|end_header_id|>.... Mistral uses <s>[INST] ... [/INST]. Get this wrong and you'll train the model to be confused. Use tokenizer.apply_chat_template(messages, tokenize=False). 2. Be ruthless about quality. Remove duplicates, repeated patterns, copy-paste errors, refused responses. Lima paper: 1000 hand-curated examples can fine-tune a 65B model as well as 50k crowd-sourced ones. 3. Mask the prompt loss. When fine-tuning for instruction following, you should only compute loss on the *response*, not the prompt — otherwise the model wastes capacity 'predicting' the user's question. 4. Mix in a small amount of general data (e.g. 10% Alpaca / OpenHermes / Tulu) so the model doesn't catastrophically forget its general abilities. 5. Evaluate on a held-out set you wrote yourself. Public benchmarks are usually irrelevant to your actual task. Hand-write 50-100 examples that represent your real use case. This is your North Star metric.

Dataset libraries: datasets (HF), litdata, `distilabel` (synthetic data generation), argilla (annotation), lm-eval-harness (evaluation).

After SFT, you often want to refine the model with preference data — pairs of (prompt, chosen, rejected) responses where humans (or a stronger model) judged 'chosen' better.

Using TRL with Unsloth:

from trl import DPOTrainer, DPOConfig
from unsloth import PatchDPOTrainer
PatchDPOTrainer()   # unsloth speedup

model, tokenizer = FastLanguageModel.from_pretrained('out/llama3-instruct-merged', load_in_4bit=True, ...)
model = FastLanguageModel.get_peft_model(model, r=64, lora_alpha=64, ...)

ds = load_dataset('argilla/ultrafeedback-binarized-preferences-cleaned', split='train')

trainer = DPOTrainer(
    model=model, tokenizer=tokenizer,
    train_dataset=ds,
    args=DPOConfig(
        per_device_train_batch_size=1, gradient_accumulation_steps=8,
        max_steps=200, learning_rate=5e-6,    # 10x smaller than SFT
        beta=0.1,                              # KL strength
        max_prompt_length=1024, max_length=1536,
        output_dir='out-dpo',
    ),
)
trainer.train()

Notes:

• Learning rate is much smaller for DPO than SFT (5e-6 vs 2e-4). Preference fine-tuning is delicate.
• `beta` controls how far the policy can drift from the reference. Lower = bigger updates but more reward hacking risk. 0.1-0.3 is typical.
• DPO often wants higher LoRA rank than SFT (32-64 vs 16) — the changes are more global.
• Datasets: UltraFeedback, HH-RLHF, OpenAssistant Conversations, or your own logged preferences. Quality matters; 1k high-signal pairs beats 50k noisy ones.

For most product use cases, SFT alone is enough. Add DPO when you need to refine tone, factuality, safety, or formatting beyond what SFT achieved.

Don't trust loss alone. The actual evaluation framework:

1. Held-out task examples you hand-wrote. The most important metric. 2. LLM-as-judge: ask GPT-4 / Claude / a strong open model to score your outputs vs. a baseline. Use a structured rubric. pairwise comparisons more reliable than absolute scores. 3. Standard benchmarks: only the ones relevant to your task. For instruction-following: MT-Bench, AlpacaEval, Arena-Hard. For reasoning: GSM8K, MATH, BBH. For domain knowledge: MMLU subsets. For coding: HumanEval, MBPP. 4. Latency / throughput at your target inference setup. A 7B that doesn't fit in your latency budget is a 0. 5. Production A/B test when feasible. The only metric that proves ROI.

Watch for catastrophic forgetting: the fine-tune nails your task but breaks general capabilities. Mitigations: small LR, mix in some original SFT data, regularize with KL to the base. If catastrophic forgetting persists, you may need lower LoRA rank or more data diversity.