Fine-tuning an open-weight LLM used to mean touching every parameter and paying for it in GPU memory. LoRA (Low-Rank Adaptation) changed that calculus. This is a placeholder post; replace it with your own write-up.
The core idea behind LoRA
Instead of updating a weight matrix W directly, LoRA freezes W and
learns a low-rank decomposition ΔW = A·B, where A and B are much
smaller matrices. During training you only update A and B, which can
cut trainable parameters by over 99% while keeping most of full
fine-tuning's quality.
class LoRALayer(nn.Module):
def __init__(self, in_dim, out_dim, rank=8, alpha=16):
super().__init__()
self.A = nn.Parameter(torch.randn(in_dim, rank) * 0.01)
self.B = nn.Parameter(torch.zeros(rank, out_dim))
self.scale = alpha / rank
def forward(self, x):
return x @ self.A @ self.B * self.scale
When LoRA is the right call
- You're adapting style, tone, or a narrow task (classification, structured extraction) rather than teaching genuinely new capabilities.
- You need to serve many fine-tuned variants cheaply — LoRA adapters are small enough to swap per-request on a shared base model.
- You're memory-constrained and full fine-tuning simply won't fit on your hardware.
When you actually need full fine-tuning
- You're trying to inject substantial new knowledge or change deep reasoning behavior, not just surface style.
- Your task distribution is very far from the base model's pretraining distribution.
- You have the compute budget and the quality gap from LoRA is measurable and matters for your use case.
A practical middle ground: QLoRA
QLoRA quantizes the frozen base model to 4-bit and trains LoRA adapters on top, which is how most people fine-tune 7B–70B parameter models on a single consumer or prosumer GPU today. The quality loss from quantization is usually small relative to the memory savings.
Takeaway
Default to LoRA/QLoRA first. Reach for full fine-tuning only after you've measured a real quality gap that matters for your product, not because it feels like the "more serious" option.