17 Adapting Pretrained Models: Prompting, PEFT, Quantization
Chapter 9 left us a transfer rule: importing a representation is most promising when labels are scarce, the target is feature-hungry, and the source supplies useful coverage at a matched scale. Chapter 15 showed the conventional next step—fine-tuning changes every encoder weight. Chapter 16 then planted a second warning: training-optimal is not serving-optimal. Chinchilla’s allocation can spend a pretraining budget well while leaving us with a backbone that is expensive to copy, update, or store once per downstream task.
Let us put those promises on one ledger. A pretrained model creates three different bills:
- the backbone storage—a base shared across tasks, or a full task-specific copy when sharing is abandoned;
- the incremental task state—the prompt, adapter, or unmerged delta that makes one task different from another; and
- the transient work—context tokens, activations, gradients, optimizer state, and caches needed while adapting or running the model.
These bills are related—but they are not interchangeable. Freezing a billion weights removes their gradients—it does not make the billion weights disappear. Packing each weight into fewer bits reduces a storage lower bound—it does not prove that a particular kernel will run faster. Adding demonstrations changes no weight—it still lengthens the sequence the model must process. A merged LoRA or fully tuned checkpoint can collapse base and delta into one artifact—but then each task carries a full backbone-sized copy instead of sharing the original.
The useful organizing questions are therefore not simply “How many parameters?” They are where does task-specific information live, and which object does the method change? Hard prompts and retrieved documents place task information in context. Parameter-efficient fine-tuning (PEFT) places it in a restricted set of learned values. Quantization need not add task information at all—it changes the numerical representation of a base or adapter. QLoRA composes the last two—it is not a fourth kind of objective.
17.1 Prompting: move evidence into context
A decoder in Chapter 14 already defines a conditional distribution over the next token. Prompting changes the prefix on which that fixed distribution is conditioned. For an instruction \(I\), demonstrations \((x_i,y_i)\), and a query \(x_\star\)—a discrete prompted prediction can be written
\[ \widehat y =\argmax_y p_{\theta_0}\!\left( y\mid I,(x_1,y_1),\ldots,(x_k,y_k),x_\star \right), \qquad \theta_{\text{after}}=\theta_{\text{before}}=\theta_0. \tag{17.1}\]
With no demonstrations, this is zero-shot prompting. With a few, it is few-shot prompting; when the model uses those examples to infer the task inside its forward computation, we call the behavior in-context learning (ICL). The word learning names a behavioral change with context—not a checkpoint update.
Brown and colleagues found that few-shot gains often grew with model scale in GPT-3—but that is an empirical result over particular models and tasks. It does not establish universal parameter thresholds, nor does it settle the mechanism by which a Transformer uses demonstrations. Attention makes earlier examples visible and routable; saying it literally “performs analogy” would outrun the evidence. Prompt order, label words, formatting, and example choice can all matter. Controlled ablations have found that demonstration inputs, the label space and its mapping, formatting, and ordering can contribute differently across tasks. Those results constrain simple stories about copying examples, but they do not settle one universal causal mechanism for ICL.
Instructions that ask for intermediate steps can improve some multi-step tasks. Those visible steps are generated tokens, however—not a guaranteed faithful window into the model’s internal computation. The same logits and decoding cautions from Chapter 11 still apply.
NotePrompting is not prompt tuning
A hard prompt is a sequence of ordinary token IDs chosen by a person or program. It receives no gradient. Prompt tuning, introduced later in this chapter, learns continuous vectors by backpropagation and is therefore a PEFT method. Similar names do not mean the same training procedure.
Zero trainable parameters is not zero cost—demonstrations consume context length. They increase prefill work, the initial forward pass that processes the supplied prefix, and occupy entries in the key/value (KV) cache, the stored attention keys and values reused while autoregressively generating later tokens. They also expose task data at inference time. Prompting trades a new checkpoint for a longer, more carefully constructed input.
A frozen-context mechanism test
Can a fixed network genuinely change its answer because examples in the prompt reveal a task? We can test that mechanism without claiming to reproduce natural-language ICL.
Each synthetic episode secretly permutes four group symbols onto four label symbols. A demonstration reveals one group–label pair; demo groups are distinct. A two-shot prompt might read [BOS] group C, label A; group A, label D; [QUERY] group A, whose target is label D. The query asks for another group’s label. A tiny causal Transformer is meta-trained across many such episodes, cycling through one to four demonstrations. During evaluation, it receives a new random mapping and no parameter update.
The information ceiling is known before training. With \(k<4\) demonstrations, the query is already covered with probability \(k/4\). Otherwise, its label is uniform among the \(4-k\) unused labels. Therefore
\[ \Pr(\text{correct}\mid k) = \begin{cases} \dfrac{k}{4}+\left(1-\dfrac{k}{4}\right)\dfrac{1}{4-k} =\dfrac{k+1}{4}, & 0\le k\le3,\\[6pt] 1, & k=4. \end{cases} \tag{17.2}\]
At \(k=3\), even an unseen query group is solvable by elimination—a distinction that will tell us whether the model merely copies a seen pair or uses the whole mapping.
The code uses nn.TransformerEncoderLayer as a compact container for the same pre-LayerNorm attention–residual–FFN structural pattern built from first principles in Chapter 14. Its GELU choice differs from that chapter’s teaching block; the wrapper does not hide the causal or padding masks, embeddings, or classifier. Token IDs have shape \([B,11]\), hidden states \([B,11,48]\), and the query logits \([B,4]\).
- Prepare the inputs and fixed settings for the example.
- Define the reusable helpers:
sample_episode_bank,render_episodes, andContextLearner. - Define the reusable helpers:
evaluate_contextandrun_context_seed. - Meta-train a tiny causal Transformer, then audit frozen-weight in-context adaptation over five seeds.
# [1]
PAD, BOS, QUERY = 0, 1, 2
GROUP0, LABEL0 = 3, 7
VOCAB, SEQUENCE_LENGTH, N_CLASSES = 11, 11, 4
# [2]
def sample_episode_bank(
batch_size: int, generator: torch.Generator
) -> tuple[torch.Tensor, torch.Tensor, torch.Tensor]:
mapping = torch.rand(batch_size, N_CLASSES, generator=generator).argsort(dim=1)
demo_groups = torch.rand(
batch_size, N_CLASSES, generator=generator
).argsort(dim=1)
query_groups = torch.randint(
N_CLASSES, (batch_size,), generator=generator
)
return mapping, demo_groups, query_groups
def render_episodes(
bank: tuple[torch.Tensor, torch.Tensor, torch.Tensor], k: int
) -> tuple[torch.Tensor, torch.Tensor, torch.Tensor]:
mapping, demo_groups, query_groups = bank
tokens = torch.full((len(mapping), SEQUENCE_LENGTH), PAD, dtype=torch.long)
tokens[:, 0] = BOS
for slot in range(k):
group = demo_groups[:, slot]
label = mapping.gather(1, group[:, None]).squeeze(1)
tokens[:, 1 + 2 * slot] = GROUP0 + group
tokens[:, 2 + 2 * slot] = LABEL0 + label
tokens[:, 9] = QUERY
tokens[:, 10] = GROUP0 + query_groups
targets = mapping.gather(1, query_groups[:, None]).squeeze(1)
covered = (
(demo_groups[:, :k] == query_groups[:, None]).any(dim=1)
if k else torch.zeros(len(mapping), dtype=torch.bool)
)
return tokens, targets, covered
class ContextLearner(nn.Module):
def __init__(self, width: int = 48, heads: int = 4, layers: int = 2) -> None:
super().__init__()
self.token = nn.Embedding(VOCAB, width, padding_idx=PAD)
self.position = nn.Embedding(SEQUENCE_LENGTH, width)
layer = nn.TransformerEncoderLayer(
d_model=width, nhead=heads, dim_feedforward=2 * width,
dropout=0.0, batch_first=True, norm_first=True, activation="gelu",
)
self.encoder = nn.TransformerEncoder(
layer, layers, norm=nn.LayerNorm(width), enable_nested_tensor=False
)
self.head = nn.Linear(width, N_CLASSES)
self.register_buffer(
"causal",
torch.triu(torch.ones(SEQUENCE_LENGTH, SEQUENCE_LENGTH, dtype=torch.bool),
diagonal=1),
)
def forward(self, tokens: torch.Tensor) -> torch.Tensor:
positions = torch.arange(SEQUENCE_LENGTH, device=tokens.device)
hidden = self.token(tokens) + self.position(positions)[None]
hidden = self.encoder(
hidden, mask=self.causal, src_key_padding_mask=tokens.eq(PAD)
)
return self.head(hidden[:, -1])
# [3]
@torch.no_grad()
def evaluate_context(
model: nn.Module, seed: int, n: int = 8192
) -> tuple[list[dict], bool]:
model.eval()
before = [parameter.detach().clone() for parameter in model.parameters()]
bank = sample_episode_bank(n, torch.Generator().manual_seed(seed + 10_000))
rows = []
for k in range(5):
tokens, targets, covered = render_episodes(bank, k)
correct = model(tokens).argmax(1).eq(targets)
rows.append({
"k": k,
"accuracy": correct.float().mean().item(),
"covered": correct[covered].float().mean().item()
if covered.any() else float("nan"),
"uncovered": correct[~covered].float().mean().item()
if (~covered).any() else float("nan"),
})
unchanged = all(
torch.equal(old, new) for old, new in zip(before, model.parameters())
)
return rows, unchanged
def run_context_seed(seed: int, steps: int = 1200) -> dict:
torch.manual_seed(seed)
generator = torch.Generator().manual_seed(seed)
model = ContextLearner()
optimizer = torch.optim.AdamW(model.parameters(), lr=3e-3, weight_decay=1e-2)
scheduler = torch.optim.lr_scheduler.CosineAnnealingLR(
optimizer, T_max=steps, eta_min=3e-5
)
model.train()
for step in range(steps):
k = 1 + step % 4
bank = sample_episode_bank(256, generator)
tokens, targets, _ = render_episodes(bank, k)
loss = F.cross_entropy(model(tokens), targets)
optimizer.zero_grad(set_to_none=True)
loss.backward()
nn.utils.clip_grad_norm_(model.parameters(), 1.0)
optimizer.step()
scheduler.step()
rows, unchanged = evaluate_context(model, seed)
return {
"seed": seed,
"rows": rows,
"unchanged": unchanged,
"parameters": sum(parameter.numel() for parameter in model.parameters()),
}
CONTEXT_SEEDS = list(range(6050, 6055))
context_runs = [run_context_seed(seed) for seed in CONTEXT_SEEDS]
context_accuracy = np.array([
[row["accuracy"] for row in run["rows"]] for run in context_runs
])
context_ceiling = np.array([0.25, 0.50, 0.75, 1.00, 1.00])
print("model parameters:", f'{context_runs[0]["parameters"]:,}')
print("k mean accuracy seed sd information ceiling")
# [4]
for k in range(5):
print(
f"{k} {context_accuracy[:, k].mean():.3f} "
f"{context_accuracy[:, k].std(ddof=1):.3f} "
f"{context_ceiling[k]:.2f}"
)
print("all evaluation passes left weights unchanged:",
all(run["unchanged"] for run in context_runs))model parameters: 39,268
k mean accuracy seed sd information ceiling
0 0.252 0.003 0.25
1 0.500 0.005 0.50
2 0.749 0.002 0.75
3 1.000 0.000 1.00
4 1.000 0.000 1.00
all evaluation passes left weights unchanged: True
The 39,268-parameter network lands at mean accuracies \(0.252,0.500,0.749,1.000,1.000\) for \(k=0,1,2,3,4\), respectively. The analytic ceilings are \(0.25,0.50,0.75,1,1\). More revealingly, unseen-query accuracy rises from \(0.252\) to \(0.330\), \(0.499\), and \(1.000\). The model is not only copying a matching demonstration; at \(k=3\) it identifies the remaining label.
WarningAn existence proof in a designed regime
The model was meta-trained on exactly this family of random permutation tasks. The study demonstrates that a frozen causal Transformer can use prompt evidence in this controlled distribution. It does not estimate natural-language ICL quality, explain the mechanism inside a large language model, or show that more examples always help. Across runs, initialization, meta-training episodes, and the finite 8,192-episode evaluation bank all change. The seed SD therefore measures combined run-to-run repeatability—not task or domain diversity, and not optimization alone.
Retrieval is another context lever
A prompt can contain information selected at query time rather than written by hand. Retrieval-augmented generation (RAG) first searches an external corpus, then places retrieved passages beside the instruction and query before generation. The generator’s weights may remain fixed—the task-specific state lives partly in the index and partly in the retrieved context.
This reconnects to Chapter 12. A retriever computes query–document similarities, selects neighbors, and hands their content to a downstream weighted computation. Modern systems can learn the retriever, the generator, or both, but the basic context-augmentation move is distinct from PEFT.
Retrieval moves rather than abolishes failure—a missed document cannot support the answer; a highly ranked irrelevant passage can distract it; retrieved text can contain malicious instructions; and a plausible citation does not prove that the generated claim follows from the cited evidence. RAG is attractive when facts change faster than checkpoints or when provenance matters, but it is a pipeline whose retrieval and generation stages must be evaluated separately.
17.2 What if the prompt itself were learnable?
Hard prompts spend human effort and context tokens. The chapter’s familiar refrain now applies: what if the prompt itself were learnable? Let \(\matr{P}\in\mathbb{R}^{m\times d}\) be \(m\) continuous vectors with the same width as token embeddings. For an \(n\)-token input, define \(\operatorname{Embed}(x)\in\mathbb{R}^{n\times d}\). Prompt tuning prepends the learned rows,
\[ \matr{H}^{(0)}=[\matr{P};\operatorname{Embed}(x)], \qquad \min_{\matr{P}} -\sum_i\log p_{\theta_0,\matr{P}}(y_i\mid x_i), \tag{17.3}\]
while freezing \(\theta_0\). Before any task head, the learned state contains exactly \(md\) values. Those values are not words—and need not be nearest to any vocabulary embedding. They are virtual tokens optimized by gradient descent, and they still occupy \(m\) virtual sequence positions during use.
Prefix tuning moves a related idea deeper. At every Transformer layer \(\ell\), let the ordinary keys and values be \(\matr{K}_\ell,\matr{V}_\ell\in\mathbb{R}^{n\times d}\) and the learned prefixes be \(\matr{P}^K_\ell,\matr{P}^V_\ell\in\mathbb{R}^{m\times d}\). The prefix rows extend the ordinary keys and values:
\[ \matr{K}'_\ell=[\matr{P}^K_\ell;\matr{K}_\ell], \qquad \matr{V}'_\ell=[\matr{P}^V_\ell;\matr{V}_\ell]. \]
A direct implementation with \(m\) prefix slots, width \(d\), and \(L\) layers stores \(2Lmd\) values, although practical parameterizations can differ. Li and Liang used an auxiliary network during optimization—then retained only the learned prefix for inference. Lester and colleagues found that prompt tuning approached full tuning as T5 scale grew in their SuperGLUE study; that result is not a universal size threshold.
Two other PEFT families complete the map. Adapters insert a small bottleneck inside each frozen Transformer block. For bottleneck width \(d_b\), let \(\matr{W}_{\mathrm{down}}\in\mathbb{R}^{d_b\times d}\), \(\vect{b}_{\mathrm{down}}\in\mathbb{R}^{d_b}\), \(\matr{W}_{\mathrm{up}}\in\mathbb{R}^{d\times d_b}\), and \(\vect{b}_{\mathrm{up}}\in\mathbb{R}^{d}\). One common residual form is
\[ \operatorname{Adapter}(\vect{h}) =\vect{h} +\matr{W}_{\mathrm{up}} \sigma(\matr{W}_{\mathrm{down}}\vect{h}+\vect{b}_{\mathrm{down}}) +\vect{b}_{\mathrm{up}}, \]
This module adds \(2dd_b+d+d_b\) parameters. BitFit takes the selective-update idea to an extreme—freeze pretrained weight matrices and update their existing bias terms. The original BitFit classification experiments also trained the task-specific output layer; “bias-only” describes the restriction inside the pretrained backbone. Neither method promises the same accuracy on every task. They are different restrictions on where gradient blame may write.
17.3 LoRA: a low-rank correction beside a frozen path
Consider one frozen linear map
\[ \matr{W}_0\in\mathbb{R}^{d_{\mathrm{out}}\times d_{\mathrm{in}}}, \qquad \vect{h}=\matr{W}_0\vect{x}. \]
Full fine-tuning learns an unrestricted correction \(\Delta\matr{W}\) with \(d_{\mathrm{out}}d_{\mathrm{in}}\) values. Low-Rank Adaptation (LoRA) instead constrains the correction to a product of two thin matrices:
\[ \Delta\matr{W} =\frac{\alpha}{r}\matr{B}\matr{A}, \qquad \matr{A}\in\mathbb{R}^{r\times d_{\mathrm{in}}}, \quad \matr{B}\in\mathbb{R}^{d_{\mathrm{out}}\times r}. \tag{17.4}\]
The forward pass is
\[ \vect{h} =\matr{W}_0\vect{x} +\frac{\alpha}{r}\matr{B}(\matr{A}\vect{x}). \tag{17.5}\]
\(\matr{A}\) compresses to \(r\) coordinates and \(\matr{B}\) expands back to the output width. Because \(\operatorname{rank}(\matr{B}\matr{A})\le r\), task updates can use at most \(r\) independent linear directions in that matrix. The merged matrix is not therefore rank \(r\)—the constraint applies only to the correction.
The trainable count is
\[ P_{\mathrm{LoRA}} =r(d_{\mathrm{in}}+d_{\mathrm{out}}). \tag{17.6}\]
For one \(4096\times4096\) map and \(r=16\), full tuning exposes 16,777,216 values; LoRA exposes \(16(4096+4096)=131{,}072\), or \(0.78125\%\). A model-level percentage requires summing Equation 17.6 over the matrices actually targeted. The original LoRA experiments emphasized query and value projections—later recipes often target different or more numerous matrices. “Rank 16” alone does not specify an adapter’s size.
TipCourse-lab bridge (non-examinable): read PEFT through the algebra
Module 11’s supplied Gemma/PEFT lab expresses Equation 17.4 through a package wrapper. Treat the wrapper’s names as a translation layer, not as new mathematics:
ris the inner dimension of \(A\) and \(B\);lora_alpha / rrepresents the factor \(\alpha/r\);target_moduleschooses which frozen maps play the role of \(W_0\);- the adapter checkpoint carries the learned factors, while the base checkpoint supplies the frozen path.
If the lab targets “all linear” modules, do not infer the adapter size from that phrase. Inspect the selected module names and shapes, sum Equation 17.6 over exactly that set, and print the parameters that still have requires_grad=True. Then apply the four-part freeze audit below: the base stays unchanged, only permitted factors receive gradients, and merged and unmerged outputs agree within a declared numerical tolerance. Package calls and model identifiers can evolve; these algebraic checks are the stable contract.
When the lab also stores the base on a quantization grid, the result is QLoRA, not a different LoRA equation. Keep the frozen base’s storage and compute dtypes separate from the factors’ trainable dtype, as Appendix C requires. The QLoRA section later in this chapter makes that composition explicit.
The usual initialization samples \(\matr{A}\) and sets \(\matr{B}=0\). The initial correction is then exactly zero—yet \(\matr{B}\) can receive a gradient because \(\matr{A}\vect{x}\) is generally nonzero. Setting both factors to zero would block both first gradients. The scale \(\alpha/r\) separates a chosen adapter strength from rank, but its best value remains a tuning decision.
Once trained, a single compatible adapter can be merged:
\[ \matr{W}_{\mathrm{merged}} =\matr{W}_0+\frac{\alpha}{r}\matr{B}\matr{A}. \tag{17.7}\]
Merged and unmerged arithmetic should agree to numerical tolerance. Only the merged form removes the extra branch at inference. Dynamic adapter switching, keeping many adapters active, or combining LoRA with a quantized base changes that systems story; a quantized merge generally requires dequantization and requantization.
TipA four-part freeze audit
Count trainable values; assert the base tensor is bitwise unchanged; inspect that only permitted parameters receive gradients; and compare merged with unmerged outputs. A small checkpoint is not evidence that the intended tensors were frozen. The assertions belong beside the implementation, not in a comment after training.
Make the rank bottleneck fail on purpose
Low rank is a hypothesis about the update—not a free compression theorem. We can make its boundary visible by planting a target correction with exactly six independent diagonal directions. The frozen base is a \(32\times32\) map. Full tuning may move all 1,024 entries; LoRA may use ranks \(1,2,4,6,\) or \(8\). Inputs are fixed Gaussian vectors, targets are noiseless, and five LoRA initializations see the same problem and optimizer schedule.
Write the six planted magnitudes as \(\sigma_1\ge\cdots\ge\sigma_6\). Their values are \(1.20,0.90,0.65,0.40,0.22,0.10\). A rank-\(r\) correction below six must leave some directions behind. This design deliberately favors the LoRA assumption—that is what makes the capacity claim falsifiable before we run it.
Because the six diagonal basis directions are mutually orthogonal, their squared correction energy is \(\sum_{j=1}^{6}\sigma_j^2\). A rank-\(r\) correction can retain at most \(r\) independent directions; spending them on the \(r\) largest magnitudes leaves the concrete relative floor
\[ \epsilon_r =\sqrt{\frac{\sum_{j=r+1}^{6}\sigma_j^2} {\sum_{j=1}^{6}\sigma_j^2}}, \qquad r<6, \tag{17.8}\]
and zero for \(r\ge6\). The plotted “capacity floor” is this calculation—not a result estimated from the optimizer.
- Prepare the inputs and fixed settings for the example.
- Define the reusable helpers:
make_rank_problem,FullUpdate, andLoRAUpdate. - Define the reusable helpers:
train_linear_updateandrelative_update_error. - Compare frozen, full, and low-rank updates against a planted rank-six correction.
- Check the claimed identities, shapes, or invariants.
- Report or visualize the measured result.
# [1]
D_IN = D_OUT = 32
TARGET_RANK = 6
DIRECTION_MAGNITUDES = torch.tensor([1.20, 0.90, 0.65, 0.40, 0.22, 0.10])
# [2]
def make_rank_problem(
seed: int = 1700,
) -> tuple[torch.Tensor, torch.Tensor, torch.Tensor, torch.Tensor,
torch.Tensor, torch.Tensor]:
generator = torch.Generator().manual_seed(seed)
base = torch.randn(D_OUT, D_IN, generator=generator) / math.sqrt(D_IN)
correction = torch.zeros(D_OUT, D_IN)
diagonal = torch.arange(TARGET_RANK)
correction[diagonal, diagonal] = DIRECTION_MAGNITUDES
x_train = torch.randn(1024, D_IN, generator=generator)
x_validation = torch.randn(4096, D_IN, generator=generator)
y_train = F.linear(x_train, base + correction)
y_validation = F.linear(x_validation, base + correction)
return base, correction, x_train, y_train, x_validation, y_validation
class FullUpdate(nn.Module):
def __init__(self, base: torch.Tensor) -> None:
super().__init__()
self.weight = nn.Parameter(base.clone())
def forward(self, x: torch.Tensor) -> torch.Tensor:
return F.linear(x, self.weight)
class LoRAUpdate(nn.Module):
def __init__(
self, base: torch.Tensor, rank: int, generator: torch.Generator
) -> None:
super().__init__()
self.register_buffer("weight0", base.clone())
self.A = nn.Parameter(torch.randn(rank, D_IN, generator=generator) * 0.02)
self.B = nn.Parameter(torch.zeros(D_OUT, rank))
self.rank = rank
self.alpha = rank
@property
def scaling(self) -> float:
return self.alpha / self.rank
def forward(self, x: torch.Tensor) -> torch.Tensor:
low_rank = F.linear(F.linear(x, self.A), self.B)
return F.linear(x, self.weight0) + self.scaling * low_rank
def merged_weight(self) -> torch.Tensor:
return self.weight0 + self.scaling * (self.B @ self.A)
# [3]
def train_linear_update(
model: nn.Module, x: torch.Tensor, y: torch.Tensor, steps: int = 1200
) -> None:
optimizer = torch.optim.Adam(model.parameters(), lr=0.03)
scheduler = torch.optim.lr_scheduler.CosineAnnealingLR(
optimizer, T_max=steps, eta_min=3e-4
)
for _ in range(steps):
loss = F.mse_loss(model(x), y)
optimizer.zero_grad(set_to_none=True)
loss.backward()
optimizer.step()
scheduler.step()
def relative_update_error(
estimate: torch.Tensor, target: torch.Tensor
) -> float:
return (torch.linalg.norm(estimate - target) / torch.linalg.norm(target)).item()
base, target_delta, x_train, y_train, x_val, y_val = make_rank_problem()
target_energy = torch.sum(DIRECTION_MAGNITUDES ** 2)
full_model = FullUpdate(base)
# [4]
train_linear_update(full_model, x_train, y_train)
with torch.no_grad():
full_mse = F.mse_loss(full_model(x_val), y_val).item()
full_relative_error = relative_update_error(
full_model.weight - base, target_delta
)
frozen_mse = F.mse_loss(F.linear(x_val, base), y_val).item()
LORA_RANKS = [1, 2, 4, 6, 8]
lora_runs = []
# [5]
for seed in range(6050, 6055):
for rank in LORA_RANKS:
model = LoRAUpdate(
base, rank, torch.Generator().manual_seed(seed + rank)
)
frozen_before = model.weight0.clone()
train_linear_update(model, x_train, y_train)
with torch.no_grad():
merged = model.merged_weight()
merge_error = (
model(x_val[:64]) - F.linear(x_val[:64], merged)
).abs().max().item()
assert torch.equal(frozen_before, model.weight0)
assert merge_error < 2e-6
learned_delta = merged - base
lora_runs.append({
"seed": seed,
"rank": rank,
"trainable": rank * (D_IN + D_OUT),
"validation_mse": F.mse_loss(model(x_val), y_val).item(),
"relative_error": relative_update_error(learned_delta, target_delta),
"merge_error": merge_error,
})
# [6]
print(f"frozen validation MSE: {frozen_mse:.6f}")
print(
f"full update: MSE={full_mse:.2e}, relative delta error="
f"{full_relative_error:.2e} (1,024 trainable)"
)
print("rank trainable mean validation MSE mean relative update error")
for rank in LORA_RANKS:
selected = [row for row in lora_runs if row["rank"] == rank]
print(
f"{rank:>4} {selected[0]['trainable']:>9,} "
f"{np.mean([row['validation_mse'] for row in selected]):>19.8f} "
f"{np.mean([row['relative_error'] for row in selected]):>26.6f}"
)
print("largest merged/unmerged difference:",
f"{max(row['merge_error'] for row in lora_runs):.2e}")frozen validation MSE: 0.091155
full update: MSE=6.39e-15, relative delta error=2.39e-07 (1,024 trainable)
rank trainable mean validation MSE mean relative update error
1 64 0.04535140 0.709896
2 128 0.02021863 0.471562
4 256 0.00188776 0.142274
6 384 0.00000000 0.000000
8 512 0.00000000 0.000000
largest merged/unmerged difference: 1.43e-06
The failure is orderly. Rank 1 leaves relative correction error \(0.710\); rank 2 leaves \(0.472\); rank 4 leaves \(0.142\). Their mean validation MSEs are \(0.04535,0.02022,\) and \(0.001888\). Rank 6 and rank 8 recover the target to below \(10^{-7}\) relative correction error. The unrestricted update reaches numerical-zero validation MSE and \(2.39\times10^{-7}\) relative correction error. Every frozen base is bitwise intact, and the largest merged–unmerged discrepancy is below \(2\times10^{-6}\).
In this toy \(32\times32\) layer, rank 6 still trains 384 values—37.5% of a full matrix. The point is inspectability, not an impressive compression ratio. Real LoRA savings depend on wide matrices, small ranks, and which layers are targeted. Likewise, the experiment was built from a low-rank shift—a task whose useful correction is not well captured at the chosen rank will retain error no matter how fashionable the adapter is.
17.4 Quantization: put weights on a smaller grid
PEFT restricts which values may change. Quantization asks how precisely values must be represented. If a tensor contains \(N\) values stored at \(b\) bits each, its ideal packed payload is
\[ \text{payload}=\frac{Nb}{8}\ \text{bytes}. \tag{17.9}\]
One billion FP16 weights therefore require 2 GB in decimal units; an ideal packed 4-bit payload requires 0.5 GB. Those are weight-payload lower bounds—not complete checkpoints or working-memory forecasts. Scales, zero-points, unquantized tensors, packing alignment, activations, and caches remain.
This is where a tensor’s dtype becomes more than a software label. Appendix B will gather dtype, shape, stride, and physical layout in one place; Appendix C connects FP16, BF16, rounding, and mixed precision to the finite representations first encountered in Chapter 5. Here we need one narrower fact—fewer nominal bits change both representable values and storage arithmetic.
Start with signed symmetric uniform quantization of a real-valued tensor \(\matr{W}\). For \(b\) bits, let
\[ Q=2^{b-1}-1, \qquad s=\frac{\max_{ij}|W_{ij}|}{Q}. \]
Quantize and reconstruct by
\[ q_{ij}=\operatorname{clip}\!\left( \operatorname{round}(W_{ij}/s),-Q,Q \right), \qquad \widehat W_{ij}=s q_{ij}. \tag{17.10}\]
\(s\) is the distance between adjacent real-valued grid points. This symmetric choice uses codes \(-Q,\ldots,Q\): \(2^b-1\) levels, leaving the most-negative two’s-complement code unused so the endpoints remain balanced around zero. With the maximum used for calibration, values are not clipped beyond the selected range; nearest rounding gives \(|W_{ij}-\widehat W_{ij}|\le s/2\). One extreme value increases \(s\) for everything sharing that scale, so small values may all round to zero.
If every value is zero, choose any positive sentinel scale—say \(s=1\)—and store zero codes. A constant affine range likewise needs a separately defined convention; the range formula below otherwise divides by zero.
An affine grid also stores an integer zero-point \(z\):
\[ q=\operatorname{clip}\!\left( \operatorname{round}(w/s)+z,q_{\min},q_{\max} \right), \qquad \widehat w=s(q-z). \tag{17.11}\]
For a nonconstant calibrated interval \([w_{\min},w_{\max}]\), a standard choice is
\[ s=\frac{w_{\max}-w_{\min}}{q_{\max}-q_{\min}}, \qquad z=\operatorname{clip}\!\left( \operatorname{round}\!\left(q_{\min}-\frac{w_{\min}}{s}\right), q_{\min},q_{\max} \right). \]
The zero-point lives in integer-code coordinates. Mixing it with a real-valued offset—subtracting \(z\) before dividing by \(s\), then adding \(z\) after dequantization—is a units error.
One scale can erase a quiet channel
Scale granularity changes the trade—per-tensor quantization shares one scale across an entire matrix. Per-row quantization gives each output row its own scale; for a linear layer this is often called per-output-channel quantization. More scales cost metadata, but a large row no longer sets the grid spacing for a small row.
The next audit creates a \(64\times256\) weight matrix whose row ranges span \(0.01\) to \(10\). It compares 8-bit and 4-bit symmetric grids, either global or per-row, on 4,096 fixed Gaussian inputs. The code reports relative weight error, relative layer-output error, the distribution of row errors, and ideal packed payload plus FP32 scale metadata. It reports no runtime—the simple Python arithmetic is not a quantized production kernel.
- Prepare the inputs and fixed settings for the example.
- Define the reusable helpers:
make_quantization_problem,symmetric_quantize, andrun_quantization_audit. - Compare per-tensor and per-row symmetric quantization, including output error and metadata.
# [1]
QUANT_D_IN, QUANT_D_OUT = 256, 64
# [2]
def make_quantization_problem(
seed: int = 1701,
) -> tuple[torch.Tensor, torch.Tensor]:
generator = torch.Generator().manual_seed(seed)
directions = torch.randn(QUANT_D_OUT, QUANT_D_IN, generator=generator)
directions = directions / directions.abs().amax(dim=1, keepdim=True)
row_ranges = torch.logspace(-2, 1, QUANT_D_OUT).unsqueeze(1)
weights = directions * row_ranges
inputs = torch.randn(4096, QUANT_D_IN, generator=generator)
return weights, inputs
def symmetric_quantize(
weights: torch.Tensor, bits: int, per_row: bool
) -> tuple[torch.Tensor, torch.Tensor, torch.Tensor]:
qmax = 2 ** (bits - 1) - 1
scale = (
weights.abs().amax(dim=1, keepdim=True) / qmax
if per_row else weights.abs().amax() / qmax
)
scale = torch.where(scale == 0, torch.ones_like(scale), scale)
codes = torch.clamp(torch.round(weights / scale), -qmax, qmax).to(torch.int8)
reconstructed = codes.float() * scale
return codes, reconstructed, scale
@torch.no_grad()
def run_quantization_audit() -> list[dict]:
weights, inputs = make_quantization_problem()
reference = inputs @ weights.T
rows = []
for bits in (8, 4):
for per_row in (False, True):
codes, reconstructed, scale = symmetric_quantize(
weights, bits, per_row
)
row_errors = (
torch.linalg.norm(reconstructed - weights, dim=1)
/ torch.linalg.norm(weights, dim=1)
)
payload_bytes = codes.numel() * bits / 8
metadata_bytes = scale.numel() * 4
rows.append({
"bits": bits,
"granularity": "per-row" if per_row else "per-tensor",
"weight_error": (
torch.linalg.norm(reconstructed - weights)
/ torch.linalg.norm(weights)
).item(),
"output_error": (
torch.linalg.norm(inputs @ reconstructed.T - reference)
/ torch.linalg.norm(reference)
).item(),
"row_errors": row_errors.numpy(),
"row_ranges": weights.abs().amax(dim=1).numpy(),
"payload_bytes": int(payload_bytes),
"metadata_bytes": int(metadata_bytes),
"total_bytes": int(payload_bytes + metadata_bytes),
})
return rows
quant_rows = run_quantization_audit()
print("bits scheme w err out err med row max row payload meta total")
# [3]
for row in quant_rows:
print(
f"{row['bits']:>4} {row['granularity']:<10} "
f"{row['weight_error']:.4f} {row['output_error']:.4f} "
f"{np.median(row['row_errors']):.4f} {row['row_errors'].max():.4f} "
f"{row['payload_bytes']:>7,} {row['metadata_bytes']:>8,} "
f"{row['total_bytes']:>5,}"
)bits scheme w err out err med row max row payload meta total
8 per-tensor 0.0213 0.0213 0.2213 1.0000 16,384 4 16,388
8 per-row 0.0069 0.0069 0.0067 0.0096 16,384 256 16,640
4 per-tensor 0.2688 0.2695 1.0000 1.0000 8,192 4 8,196
4 per-row 0.1228 0.1223 0.1225 0.1772 8,192 256 8,448
The global 8-bit grid has only \(2.13\%\) relative output error overall, which sounds reassuring until we inspect rows—the median relative row error is \(22.13\%\), and the quietest rows are reconstructed as all zeros. Per-row scales lower overall output error to \(0.689\%\) and maximum row error below \(0.964\%\). At 4 bits, per-row scales improve output error from \(26.95\%\) to \(12.23\%\), but do not make the approximation exact.
The packed 4-bit codes require 8,192 bytes. One FP32 scale brings the global total to 8,196 bytes; 64 row scales bring it to 8,448. The live torch.int8 tensor in this teaching cell occupies one byte per code—it is not physically packed into nibbles. Equation 17.9 is the payload a packing format could achieve.
WarningA smaller checkpoint is not automatically faster
Bit count does not determine latency by itself. Real speed depends on supported kernels, hardware, packing, dequantization, memory bandwidth, batch shape, and which tensors remain in higher precision. Dettmers and colleagues even reported cases where LLM.int8() overhead made smaller models slower than FP16. Measure the target system; do not turn a file-size ratio into a runtime claim. The Roofline and measurement contracts in Appendix C make that distinction explicit.
What a quantization workflow protects
Post-training quantization (PTQ) chooses a representation after the base model has trained. It may use calibration inputs, but it does not run ordinary end-to-end training again. Our uniform audit used only weight ranges. More sophisticated PTQ methods protect behavior measured on calibration data. For a layer \(\matr{W}\in\mathbb{R}^{d_{\mathrm{out}}\times d_{\mathrm{in}}}\) and \(n\) calibration activations \(\matr{X}\in\mathbb{R}^{d_{\mathrm{in}}\times n}\), GPTQ frames a layerwise weight-only problem as
\[ \widehat{\matr{W}} =\argmin_{\widetilde{\matr{W}}\in\mathcal{Q}} \left\|\matr{W}\matr{X} -\widetilde{\matr{W}}\matr{X}\right\|_F^2, \tag{17.12}\]
where \(\widetilde{\matr{W}}\) has the same shape as \(\matr{W}\) and \(\mathcal{Q}\) is the chosen quantized set. The objective protects layer outputs—not ordinary weight MSE and not the original task loss. AWQ uses activation evidence to choose channel scaling that protects salient weights. These methods make the calibration distribution part of the result.
Quantization-aware training (QAT) instead inserts fake quantize–dequantize operations into the forward pass while retaining higher-precision shadow weights for optimization. Because rounding is nondifferentiable, training uses a surrogate gradient such as a straight-through estimator. QAT is not integer-only training; conversion to integer execution remains a separate, kernel-dependent step.
Weight-only and weight–activation quantization also answer different bottlenecks. A weight-only method leaves activations and the autoregressive key/value cache at their chosen compute precision. LLM.int8(), by contrast, quantizes both matrix inputs and weights while routing a small activation-outlier subspace through FP16 in its studied regime. “8-bit method” is not a complete algorithm description.
QLoRA: compose a frozen grid with a trainable delta
QLoRA places the two major levers side by side. Store a pretrained base in blockwise 4-bit NormalFloat (NF4), dequantize blocks to BF16 for matrix multiplication, and train higher-precision LoRA factors:
\[ \vect{h} =\operatorname{Dequant}_{\mathrm{NF4}}(\matr{W}_q)\vect{x} +\frac{\alpha}{r}\matr{B}(\matr{A}\vect{x}). \tag{17.13}\]
The base codes \(\matr{W}_q\) remain frozen. Gradient blame passes through the dequantized computation to the LoRA path, but ordinary updates do not rewrite the 4-bit base. QLoRA is therefore not QAT of the backbone, and “4-bit storage” does not mean every multiply or accumulator is 4-bit.
NF4 uses a nonuniform codebook with denser resolution where a standardized normal distribution places more probability mass, plus an exact zero. Its information argument applies to block-normalized, approximately zero-centered normal weights; it is not a universal optimal grid for arbitrary tensors.
QLoRA also uses double quantization—it quantizes its quantization constants. With the paper’s illustrative group sizes, FP32 constants for each 64-weight block cost \(32/64=0.5\) bits per weight. Storing those constants at 8 bits and adding one FP32 second-level constant per 256 first-level constants costs
\[ \frac{8}{64}+\frac{32}{64\cdot256}=0.126953 \quad\text{bits per weight}, \]
a saving of \(0.373047\) bits per weight. That number belongs to those block sizes; it is not a universal number of gigabytes. Paged optimizers address transient memory spikes in the paper’s implementation, while the LoRA factors still carry their own gradients and optimizer state.
NoteQLoRA is a composition, not a new objective
NF4 decides how the frozen foundation is stored; LoRA decides where task gradients may write. The supervised, preference, or other loss still decides what behavior is rewarded. A parameter-efficient route can optimize a poor objective just as efficiently as a good one.
17.5 Choose by the bill you need to reduce
| Method | Task state and permitted writes | Main consequence |
|---|---|---|
| Hard prompt / ICL | State in ordinary context; no gradients during use | No learned task checkpoint; longer context |
| RAG | State in index and retrieved context; writes depend on design | Fresher evidence; retrieval failure surface |
| Prompt tuning | State in continuous input vectors; write prompt vectors, plus any deliberately unfrozen head | Small learned state; virtual positions remain |
| Prefix / adapters | State in per-layer prefixes or bottlenecks; write added modules | Small checkpoint; extra inference path |
| LoRA | State in \(\matr{A},\matr{B}\); write factors, plus any deliberately unfrozen head | Conditionally mergeable; rank bottleneck |
| Full fine-tuning | State in a changed backbone; write all permitted parameters | Maximum update freedom; one full task copy |
| PTQ | No task state—representation only; no gradients in ordinary PTQ | Smaller weight representation; approximation error |
| QLoRA | Task state in LoRA factors; codes/scales describe the frozen base; write factors plus any deliberately unfrozen head | Smaller base representation during PEFT; mixed dtypes |
The instructor’s practical instinct remains useful: when a carefully evaluated prompt or retrieval pipeline solves the task, avoid fine-tuning merely because it is available. Treat that as a search heuristic—not a law. A stable format change may fit a small adapter; genuinely new knowledge may belong in retrieval or further training; a severe source-coverage mismatch may defeat all of them.
That last point harvests Chapter 9’s full promise. PEFT changes the cost after we choose transfer; it does not repeal the coverage condition. A frozen foundation cannot guarantee a reliable representation absent from source coverage, and a low-rank update does not guarantee that scarce target labels can build the missing feature. Quantization preserves or damages an existing computation—it does not add coverage.
WarningTrainable parameters are not total memory
During PEFT, the frozen base still occupies memory and backpropagation still stores or recomputes activations through it. During generation, context and key/value caches can dominate different regimes. Optimizer dtype and implementation change the bill. Report trainable state, base storage, transient training memory, context length, and measured runtime separately. One percentage cannot stand in for all five.
Parameter efficiency also does not guarantee immunity to forgetting or harmful behavior. Restricting an update constrains which coordinates or directions are available—it does not bound the update’s magnitude or certify what the movement preserves. Evaluation must include the target task, retained capabilities that matter, calibration or distribution shifts, and the actual deployment representation.
17.6 The next question: what deserves the update?
Where the update lives is not what the update optimizes. Prompt vectors, low-rank matrices, and full fine-tuning decide where task gradients may write. Quantization decides how values are stored. None decides which answer is helpful, harmless, honest, or preferred.
Chapter 18 will keep the adaptation machinery and change the question from which parameters move? to which objective should move them? Instruction tuning, preference learning, and alignment methods are claims about supervision and objectives—not automatic consequences of choosing LoRA or a full checkpoint.
NoteCheck yourself
Close the book for one minute and separate the adaptation bills.
- Where can task-specific state live when the backbone is frozen?
- What does LoRA’s rank constrain, and what does quantization change instead?
- Why can a method reduce stored bits without reducing every inference cost?
17.7 Okay, so — adaptation has three separate bills
- Adaptation has three bills. Backbone storage, incremental task state, and transient work must be reported separately. Zero trainable weights and fewer stored bits do not mean zero inference or training cost.
- Prompting changes context, not weights. The five-seed permutation study closely tracks the analytical information ceilings while every evaluation weight stays bitwise fixed. RAG is another context lever, with a separate retrieval failure surface.
- PEFT restricts where gradient blame may write. Prompt tuning, prefixes, adapters, BitFit, and LoRA impose different parameterizations; they are not interchangeable labels for “small fine-tuning.”
- LoRA learns a low-rank correction. Its trainable count is \(r(d_{\mathrm{in}}+d_{\mathrm{out}})\) per targeted matrix, and its rank is a real capacity limit. The planted rank-six experiment fails exactly where it should.
- Quantization is a grid, a granularity, and a workflow. Per-row scales rescue quiet rows in the controlled audit, but consume metadata. Payload size does not predict runtime without kernels and hardware.
- QLoRA composes representation and adaptation. A frozen NF4 base is dequantized for compute while higher-precision LoRA factors train. Storage dtype, compute dtype, and trainable dtype are distinct.
- Transfer coverage still governs the choice. None of these methods creates source knowledge for free. The next chapter will ask what the permitted update should optimize.
Sources and further reading
- Brown et al., Language Models are Few-Shot Learners: GPT-3’s zero-, one-, and few-shot evaluation and its model-scale dependence.
- Min et al., Rethinking the Role of Demonstrations: What Makes In-Context Learning Work?: controlled ablations of demonstration inputs, labels, and formatting that qualify simple imitation accounts.
- Lewis et al., Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks: learned retrieval combined with a pretrained generator.
- Houlsby et al., Parameter-Efficient Transfer Learning for NLP: bottleneck adapters inserted into a frozen Transformer.
- Li and Liang, Prefix-Tuning: Optimizing Continuous Prompts for Generation: trainable per-layer prefixes for generation.
- Lester et al., The Power of Scale for Parameter-Efficient Prompt Tuning: input-level soft prompts and the scoped T5 scale study.
- Ben-Zaken et al., BitFit: Simple Parameter-Efficient Fine-Tuning for Transformer-Based Masked Language-Models: bias-term updates plus the task-specific classifier in the original experiments.
- Hu et al., LoRA: Low-Rank Adaptation of Large Language Models: the low-rank update parameterization, initialization, and merge.
- Wei et al., Chain-of-Thought Prompting Elicits Reasoning in Large Language Models: scoped gains from prompting with intermediate reasoning steps.
- Turpin et al., Language Models Don’t Always Say What They Think: Unfaithful Explanations in Chain-of-Thought Prompting: controlled evidence that visible explanations can omit causal influences on predictions.
- Greshake et al., Not What You’ve Signed Up For: Compromising Real-World LLM-Integrated Applications with Indirect Prompt Injection: the attack surface created when untrusted retrieved data can act as instructions.
- Jacob et al., Quantization and Training of Neural Networks for Efficient Integer-Arithmetic-Only Inference: affine quantization and the fake-quantization training pattern.
- Dettmers et al., LLM.int8(): 8-bit Matrix Multiplication for Transformers at Scale: mixed-precision decomposition for activation outliers—and measured overhead caveats.
- Frantar et al., GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformers: calibration-output-aware, layerwise weight-only PTQ.
- Lin et al., AWQ: Activation-aware Weight Quantization for LLM Compression and Acceleration: activation evidence for protecting salient weights.
- Dettmers et al., QLoRA: Efficient Finetuning of Quantized LLMs: NF4, double quantization, paged optimizers, and LoRA over a frozen quantized base.
Exercises
- (Pencil.) A model has 24 matrices of shape \(4096\times4096\). LoRA targets each with rank 16. Compute full and adapter value counts, the adapter fraction, and FP16 adapter bytes. Then compute the ideal 4-bit base payload. Add one FP32 scale per block of 64 weights and explain which deployment costs remain absent.
- (Code.) Reproduce the frozen-context experiment, then randomize demonstration order, repeat one group with the same label, and finally insert a contradictory label for a repeated group. Predeclare which interventions preserve Equation 17.2 and which make it inapplicable. Report all seeds and verify bitwise parameter identity before interpreting a difference.
- (Pencil.) (a) Prove \(\operatorname{rank}(\matr{B}\matr{A})\le r\). (Code.) (b) Extend the rank-capacity audit with a planted rank-10 correction and ranks \(2,6,10,14\). Audit gradients after the first backward pass for (a) random \(\matr{A}\) and zero \(\matr{B}\), and (b) both factors zero. Verify merged–unmerged equality.
- (Pencil.) (a) Quantize \([-1.2,-0.3,0.0,0.5,1.7]\) onto a signed 4-bit symmetric grid, showing \(Q,s,q,\) and \(\widehat w\). Then construct an affine 8-bit grid for a nonnegative interval. (Code.) (b) On the chapter’s unequal-row matrix, compare calibration and held-out output error after inserting one new outlier row.
- (Audit.) For each of three cases—fresh facts with citations, a stable output format, and a target domain absent from the source data—choose a first experiment among prompting/RAG, PEFT, full tuning, and quantization. State where task state lives, which bill it reduces, what it cannot repair, and one retained behavior you would evaluate. Finally explain why none of those choices answers the Chapter 18 question of which behavior deserves reward.