Parameter-Efficient Fine-Tuning with LoRA: Adapting LLMs Without Breaking the Bank

Full fine-tuning a 7B parameter model means shipping around 28 GB of gradients, optimizer states, and checkpoint copies. For teams without a multi-GPU rack, that is a non-starter. Low-Rank Adaptation (LoRA) changes the math: instead of updating the full weight matrix, you learn two small low-rank matrices whose product approximates the update. The result is models that adapt to new tasks while storing only a few megabytes of new weights.

The problem with full fine-tuning

When you fine-tune a transformer, every parameter gets its own gradient and optimizer state. For a 7B model in FP16 that is:

• Gradients: ~14 GB
• Adam optimizer states (momentum + variance): ~28 GB
• Model weights: ~14 GB
• Checkpoint copies during training: ~14 GB

That's over 60 GB of GPU memory just to get started, and the final checkpoint is still 14 GB. Most of those parameters barely move — the model already knows language, it just needs to learn a new task head or a slight distribution shift.

How LoRA works

LoRA freezes the pretrained weights $W_0 \in \mathbb{R}^{d \times k}$ and injects a trainable update of rank $r \ll \min(d, k)$:

$W_0 + \Delta W = W_0 + BA$

where $B \in \mathbb{R}^{d \times r}$ and $A \in \mathbb{R}^{r \times k}$. The key insight is that the gradient for $\Delta W$ is low-rank, so a small $r$ (typically 1–8) captures most of the useful update.

Implementation

Using Hugging Face PEFT, LoRA is a few lines of code:

from peft import LoraConfig, get_peft_model, TaskType
 
config = LoraConfig(
    task_type=TaskType.CAUSAL_LM,
    r=8,
    lora_alpha=32,
    lora_dropout=0.05,
    target_modules=["q_proj", "v_proj", "k_proj", "o_proj"],
)
 
model = get_peft_model(model, config)

The lora_alpha parameter scales the update: $\Delta W = \frac{\alpha}{r} BA$. A common heuristic is to set $\alpha = 2r$, which keeps the effective learning rate stable as you vary $r$.

Where to apply LoRA

Not all modules benefit equally. In attention layers, the query and value projections are the highest-impact targets — they directly shape what the model attends to. The output projection and MLP layers are lower priority.

Applying LoRA to all four attention projections with $r = 8$ adds only 128 K trainable parameters — about 0.002% of a 7B model.

Merging and deployment

After training, the LoRA weights can be merged back into the base model for inference with no runtime overhead:

from peft import PeftModel
 
model = PeftModel.from_pretrained(base_model, "lora-adapted")
merged = model.merge_and_unload()
merged.save_pretrained("merged-model")

Results

We adapted Qwen-7B to a domain-specific customer support task using LoRA with $r = 8$ on 2,000 labeled examples. The full fine-tuning baseline used the same data and hyperparameters.

LoRA with $r = 8$ matches full fine-tuning within 0.4% accuracy while using less than 30% of the GPU memory and producing a checkpoint that is 55,000× smaller (128 KB vs 14 GB).

When LoRA falls short

LoRA works best when the downstream task is close to the pretrained distribution. For tasks that require large semantic shifts — say, adapting a general-purpose model to highly specialized code — the low-rank assumption breaks down and full fine-tuning still wins. In those cases, consider QLoRA (quantized LoRA) or adapters as middle-ground alternatives.

What's next

We're experimenting with combining LoRA with curriculum learning: start with $r = 4$ for the first epoch to stabilize early training, then switch to $r = 16$ for the final epochs to capture finer-grained adaptations. Early results are promising for low-data regimes where overfitting is a concern.