Fine-tuning a billion-parameter model typically requires modifying billions of weights. That’s prohibitively expensive. LoRA (Low-Rank Adaptation) sidesteps this by updating only a tiny fraction of the model while achieving comparable results.<\/p>\n
The insight is elegant: weight updates during fine-tuning have low rank. You don’t need to update the full weight matrix. You only need to update a low-rank approximation of the update.<\/p>\n
Instead of fine-tuning the full weight matrix W, LoRA decomposes the update as a product of two smaller matrices: \u0394W = B \u00d7 A.<\/p>\n
For a weight matrix of shape (d_out \u00d7 d_in), LoRA introduces:<\/p>\n
A:<\/strong> shape (r \u00d7 d_in), where r is the rank (typically 8-64) During forward pass: output = W \u00d7 input + (B \u00d7 A) \u00d7 input<\/p>\n You only train A and B, freezing the original W. The rank r is typically much smaller than d_in and d_out, so the parameter count explodes down.<\/p>\n For a 70B model with 4k hidden dimensions:<\/p>\n Full fine-tuning:<\/strong> 70B trainable parameters You’re training 0.2% of the model with rank-8 LoRA. The memory and compute savings are massive.<\/p>\n LoRA’s rank is the tuning knob. Higher rank = more expressiveness but more parameters.<\/p>\n Rank 8:<\/strong> Very cheap, fast training. Works for minor domain adaptation. Fine-tuning instructions or specific styles.<\/p>\n Rank 16-32:<\/strong> Sweet spot for most applications. Enough expressiveness for meaningful adaptation without excessive cost.<\/p>\n Rank 64+:<\/strong> Approaching full fine-tuning cost. Use when minor rank isn’t expressive enough.<\/p>\n In practice, rank 16 works for 80% of use cases. Rank 32 works for 95%. Diminishing returns set in fast.<\/p>\n The assumption underlying LoRA is empirically validated: fine-tuning updates have low intrinsic rank. The model doesn’t need to change very much to adapt to new domains or tasks.<\/p>\n This makes sense. The pre-trained model already encodes enormous amounts of knowledge. Adapting to a new domain doesn’t require wholesale rewiring, just targeted adjustments.<\/p>\n LoRA captures these adjustments efficiently.<\/p>\n Using LoRA in code is trivial with the peft library:<\/p>\n Train the model normally. Only A and B matrices get updated. At inference, merge the weights or keep them separate for easy switching between adapters.<\/p>\n A 70B model with LoRA can fine-tune on a consumer GPU. Storage is minimal (rank-8 LoRA for 70B is ~140MB). You can load multiple adapters and switch between them at runtime.<\/p>\n This unlocks a new development model: base models + many specialized adapters. No need to train 10 different full models. Train 10 LoRA adapters instead, at 1% the cost.<\/p>\n
\nB:<\/strong> shape (d_out \u00d7 r)<\/p>\nThe Numbers<\/h2>\n
\nLoRA (rank 8):<\/strong> 70B \u00d7 (8 \/ 4000) \u2248 140M trainable parameters
\nLoRA (rank 64):<\/strong> 70B \u00d7 (64 \/ 4000) \u2248 1B trainable parameters<\/p>\nRank: The Key Tradeoff<\/h2>\n
Why It Works<\/h2>\n
Practical Implementation<\/h2>\n
from peft import LoraConfig, get_peft_model\n\nconfig = LoraConfig(\n r=16,\n lora_alpha=32,\n target_modules=[\"q_proj\", \"v_proj\"],\n lora_dropout=0.05,\n bias=\"none\"\n)\n\nmodel = get_peft_model(model, config)\n<\/code><\/pre>\nThe Practical Advantage<\/h2>\n