Appendix D — Notation
Notation should shorten an argument, not turn it into a guessing game. This book therefore treats a symbol’s declared shape and local definition as its contract. The typeface gives a useful first clue—the dimensions settle the question.
This appendix is a decoding key for Chapters 1–20—not a replacement for definitions at first use. Appendix A develops the linear algebra behind the symbols (Appendix A), while Appendix B develops their tensor and PyTorch contracts (Appendix B).
D.1 Typeface tells you what kind of object to expect
The book uses the following visual conventions.
| Object | Typical notation | How to read it |
|---|---|---|
| Scalar | \(x\), \(x_i\), \(b\), \(\eta\) | One number; italic unless it names a standard operator |
| Vector | \(\vect{x}\), \(\vect{w}\), \(\vect{\mu}\) | A one-axis array, normally a column vector in mathematical expressions |
| Matrix or batched array | \(\matr{X}\), \(\matr{W}\), \(\matr{\Sigma}\) | A two-axis linear map or a named multi-axis array; its declared shape identifies which |
| Function | \(f\), \(g_\theta\), \(p_\theta\) | A map, model, probability mass function, or density, as defined locally |
| Set or collection | \(\mathcal{B}\), \(\mathcal{D}\), \(\R^d\) | A minibatch, dataset, or ambient space |
| Objective | \(\loss\), \(\ell_i\) | An aggregate loss or one local/per-example loss term |
Lowercase bold symbols usually denote vectors, and uppercase bold symbols usually denote matrices or batched arrays. Both are bold—the source helpers \vect{} and \matr{} intentionally share one visual style. Boldface alone does not distinguish a matrix from a rank-four image tensor—the shape does that work.
Plain uppercase letters can still be scalars. For example, \(B\) is a batch size, \(T\) is a sequence length, and \(V\) is a vocabulary size—all three are single numbers. By contrast, a bold uppercase symbol such as \(\matr{V}\) is an array. The identity matrix and zero vector carry their size when ambiguity would matter: \(\matr{I}_d\in\R^{d\times d}\) and \(\vect{0}\in\R^d\).
A mathematical example is normally a column vector \(\vect{x}\in\R^{D_{\mathrm{in}}}\), so a dense layer is \(\vect{y}=\matr{W}\vect{x}+\vect{b}\). PyTorch stores a batch as rows: \(\matr{X}\in\R^{B\times D_{\mathrm{in}}}\). With \(\matr{W}\in\R^{D_{\mathrm{out}}\times D_{\mathrm{in}}}\), the same layer becomes \(\matr{Y}=\matr{X}\matr{W}^{\top}+\vect{1}\vect{b}^{\top}\). The transpose is a consequence of the row-batch convention—it is not a different model.
D.2 Decorations carry a second message
Once the base object is clear, decorations tell us what happened to it.
| Decoration | Example | Meaning in this book |
|---|---|---|
| Subscript | \(x_i\), \(h_t\), \(W_\ell\) | A coordinate, example, time step, or layer; the surrounding text names the axis |
| Hat | \(\widehat{y}\), \(\widehat{\vect{x}}\) | A prediction, estimate, or reconstruction |
| Star | \(\vect{w}^*\), \(D^*\) | An optimizer or theoretically optimal object |
| Tilde | \(\widetilde{\vect{x}}\) | A perturbed, sampled, or alternative version of an object |
| Bar | \(\bar\alpha_t\) | An average or accumulated quantity; in diffusion it is a cumulative product |
| Transpose | \(\matr{W}^{\top}\) | Rows and columns exchange roles |
| Inverse | \(\matr{A}^{-1}\) | The inverse map, when it exists; it does not prescribe how to compute a solution |
| Parenthesized superscript | \(\vect{w}^{(t)}\) | An optimizer iterate or other numbered stage, not exponentiation |
The operators follow standard roles:
- \(\norm{\vect{x}}_2\) is the Euclidean norm, while a subscript such as \(F\) names a different norm—for example, the Frobenius norm of a matrix.
- \(\E[Z]\), \(\var(Z)\), and \(\cov(X,Y)\) are expectation, variance, and covariance.
- \(\argmin_x f(x)\) and \(\argmax_x f(x)\) return an optimizing argument; \(\min_x f(x)\) and \(\max_x f(x)\) return the attained value.
- \(\nabla_{\vect{w}}\loss\) is a gradient with respect to a vector or parameter collection. \(\partial \loss/\partial w_j\) selects one scalar derivative.
- \(\dd x\) is a differential inside an integral, and \(\R\) denotes the real numbers.
Equality, definition, and update are also distinct. The sign \(=\) asserts equality; \(:=\) introduces a definition; \(\approx\) marks an approximation; and \(\vect{w}\leftarrow\vect{w}-\eta\vect{g}\) describes an algorithmic update—an instruction rather than an equality. In prose, an arrow may instead summarize a deduction or a flow, so read its immediate context.
D.3 Indices and dimensions: a compact dictionary
An index answers “which entry?” A dimension answers “how many entries?” We define both near first use—the same letter can legitimately change roles across topics.
| Symbol | Usual role |
|---|---|
| \(i,j\) | Example, coordinate, class, query position, or key position |
| \(b\) | Batch item; bold \(\vect{b}\) instead denotes a bias vector |
| \(\ell\) | Layer or feature coordinate |
| \(t\) | Optimizer iteration, sequence position, or diffusion step |
| \(h\) | Attention-head index; bold \(\vect{h}_t\) is a hidden state |
| \(c\) | Channel index or class label, according to the section |
| \(a,b\) | Spatial offsets inside a convolution sum |
| Dimension | Usual role |
|---|---|
| \(n,N\) | Number of examples or another locally counted collection |
| \(d,D\) | Feature dimension or model width |
| \(B\) | Minibatch size |
| \(D_{\mathrm{in}},D_{\mathrm{out}}\) | Input and output widths of a dense map |
| \(C,H,W\) | Channels, image height, and image width |
| \(K_h,K_w\) | Kernel height and kernel width |
| \(T,S\) | Query/target length and source/key length when both are needed |
| \(N_h,D_h\) | Number of attention heads and width per head |
| \(V\) | Vocabulary size; \(K\) often counts classes when declared that way |
| \(k,r\) | Bottleneck dimension, retained rank, or low-rank update size |
Equations use one-based sums unless stated otherwise, for example \(\sum_{i=1}^{n}\). Python and PyTorch index from zero—translating an equation into code changes labels, not the number of terms.
There is no global semantic difference between \(d\) and \(D\), or between \(n\) and \(N\). Likewise, \(t\) can mean an optimization step in Chapter 4 and a diffusion step in Chapter 19. The dictionary records recurring habits—it does not license skipping a section’s definitions.
D.4 The book’s recurring shape contracts
These shapes are the fastest way to audit a derivation or a code cell. Code comments write them with parentheses; equations write the same contract as a product of dimensions—the axis order is identical.
| Object | Mathematical shape | PyTorch axis order |
|---|---|---|
| Dense batch \(\matr{X}\) | \(B\times D\) | (B, D) |
| Dense weight \(\matr{W}\) | \(D_{\mathrm{out}}\times D_{\mathrm{in}}\) | (D_out, D_in) |
| Image batch | \(B\times C\times H\times W\) | (B, C, H, W) |
| Convolution weight | \(C_{\mathrm{out}}\times C_{\mathrm{in}}\times K_h\times K_w\) | (C_out, C_in, K_h, K_w) |
| Sequence states | \(B\times T\times D\) | (B, T, D) |
| Multi-head states | \(B\times N_h\times T\times D_h\) | (B, N_h, T, D_h) |
| Attention scores | \(B\times N_h\times T\times S\) | (B, N_h, T, S) |
| Token logits | \(B\times T\times V\) | (B, T, V) |
In attention, \(\matr{Q}\) carries \(T\) query positions and \(\matr{K},\matr{V}\) carry \(S\) source positions. The score product contracts only the shared head width \(D_h\), leaving \((B,N_h,T,S)\). Notice the typography: plain \(V\) is vocabulary size, while bold \(\matr{V}\) is the value array. Plain \(H\) is image height, while a bold \(\matr{H}\) can be a hidden-state array.
An external library may choose another order—time-first sequences or NHWC images, for example. Convert once at the boundary, state the permutation, and keep the internal contract stable. Table B.1 gives Appendix B’s fuller shape dictionary; the rest of that appendix supplies the broadcasting, stride, masking, and contraction audit.
D.5 Probability and optimization symbols are role labels
Probability sections commonly use uppercase \(X,Y\) for random variables and lowercase \(x,y\) for realized values, but the distinction is not imposed where boldface and shape already make the object clear. A conditional such as \(p(y\mid x)\) reads “the distribution of \(y\) given \(x\),” and \(\mathcal{N}(\vect{\mu},\matr{\Sigma})\) is a Gaussian with mean vector \(\vect{\mu}\) and covariance matrix \(\matr{\Sigma}\).
Subscripts on a distribution identify its parameters or role. Thus \(p_\theta\) is a model parameterized by \(\theta\). The symbol \(q\) often denotes an auxiliary, variational, or forward process—its precise role is still local. A sampling statement such as \(\vect{z}\sim p(\vect{z})\) means that \(\vect{z}\) is drawn from that distribution.
Parameter letters are similarly local. The collections \(\theta\) and \(\phi\) can name two networks or two sides of one objective. The symbols \(\eta\) or \(\alpha\) often denote a learning rate, \(\beta\) can denote momentum or a diffusion schedule, and \(\lambda\) often weights a penalty. Each chapter declares the role before using the symbol in a calculation.
For losses, \(\ell_i\) usually names a term associated with example \(i\), while \(\loss\) names the aggregate objective. A reported metric is not automatically the training loss: accuracy, reconstruction error, perplexity, and distributional diagnostics answer different questions even when they are computed from the same predictions.
D.6 A quick notation audit
When an expression feels opaque, ask four questions in order:
- What does each symbol represent here?
- What are its axes and dimensions?
- Which indices are free, and which are summed or otherwise reduced?
- Does the claimed output retain exactly the free axes?
That audit catches more than illegal products. It also catches legal calculations that contract the wrong axis—the notation equivalent of code that runs while solving the wrong problem.
For contributors, the canonical source helpers are \vect, \matr, \E, \Ex, \var, \cov, \norm, \argmin, \argmax, \imp, \R, \dd, and \loss. The helpers \Ex and \imp remain available for compatibility; current chapters prefer \(\E\) for expectation and choose the arrow appropriate to the local statement. Every helper is defined for both the PDF and HTML renderers.