Beyond This Volume

The Coda closes this book’s promise: a training failure becomes diagnosable when geometry, dynamics, arithmetic, state, and estimator controls are read together. DS 6210 continues beyond that endpoint. Those later topics are not missing chapters from this volume; they are branches that reuse its trunk.

Similarity-weighted aggregation and kernel geometry

An all-pairs similarity matrix raises three questions already owned here: which high-dimensional alignments are typical, which normalization keeps weights in a usable numerical range, and which Jacobian paths preserve signal. C05–C10 supply concentration and spectral baselines; C14–C15 supply criticality and normalization diagnostics.

The standard literature calls this primitive attention; the canonical self-attention architecture is Vaswani et al. (2017). The sibling volume supplies its assembly mechanics; the continuation of DS 6210 asks which kernel geometry, estimator, and derivative-path claims survive at finite width and precision.

IO-aware exact algorithms

An exact all-pairs result need not materialize the full intermediate matrix. C01 returns through arithmetic intensity and traffic lower bounds. C02 returns through bounded mergeable state. C03 and C15 return through stable online normalizers. The central audit is whether tiling and recomputation preserve the mathematical output while changing the bytes moved.

FlashAttention (Dao et al. 2022) is the standard named case. It belongs after the traffic model and online-normalizer invariant, not before them.

Parallel recurrence and associativity

A recurrence can sometimes be represented by an associative summary and evaluated by a parallel prefix scan. C02 supplies the state-and-merge question; C11 supplies retained-state and recomputation accounting; C03 supplies the warning that algebraic associativity does not imply floating-point order independence. The paper-level question is therefore both algebraic and physical: what summary composes, and what numerical result does a different reduction tree produce?

Lazy and feature-learning regimes

C04’s fixed quadratic, C08’s adaptive directions, C12’s curvature objects, and C13’s stochastic regimes form the control for asking whether training mostly changes coefficients in a nearly fixed feature map or changes the features themselves. C16 warns that the relevant operator moves with the trajectory. C17 asks whether the update geometry encourages that movement or suppresses it.

Under a declared tangent-kernel scaling and time horizon, the limit that makes training linearized also suppresses feature movement; finite-width corrections restore it at the order carried by the expansion (Jacot et al. 2018; Lee et al. 2019; Chizat et al. 2019; Roberts and Yaida 2022). Different parameterizations and simultaneous depth–width limits can retain movement, so “infinite width” is not itself a regime diagnosis.

These topics remain course synthesis and research-facing Paper audits. Keeping them outside the numbered core preserves the book’s contract: build the instruments once, then recognize their return quickly in a new method.

Chizat, Lénaïc, Edouard Oyallon, and Francis Bach. 2019. “On Lazy Training in Differentiable Programming.” Advances in Neural Information Processing Systems 32. https://papers.nips.cc/paper/2019/hash/ae614c557843b1df326cb29c57225459-Abstract.html.

Dao, Tri, Daniel Y. Fu, Stefano Ermon, Atri Rudra, and Christopher Ré. 2022. “FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness.” Advances in Neural Information Processing Systems 35, 16344–59. https://proceedings.neurips.cc/paper_files/paper/2022/hash/67d57c32e20fd0a7a302cb81d36e40d5-Abstract-Conference.html.

Jacot, Arthur, Franck Gabriel, and Clément Hongler. 2018. “Neural Tangent Kernel: Convergence and Generalization in Neural Networks.” Advances in Neural Information Processing Systems 31. https://papers.nips.cc/paper/2018/hash/5a4be1fa34e62bb8a6ec6b7ae4572128-Abstract.html.

Lee, Jaehoon, Lechao Xiao, Samuel S. Schoenholz, et al. 2019. “Wide Neural Networks of Any Depth Evolve as Linear Models Under Gradient Descent.” Advances in Neural Information Processing Systems 32. https://papers.nips.cc/paper/2019/hash/0d1a9651497a38d8b1c3871c84528bd4-Abstract.html.

Roberts, Daniel A., and Sho Yaida. 2022. The Principles of Deep Learning Theory: An Effective Theory Approach to Understanding Neural Networks. Cambridge University Press. https://doi.org/10.1017/9781009023405.

Vaswani, Ashish, Noam Shazeer, Niki Parmar, et al. 2017. “Attention Is All You Need.” Advances in Neural Information Processing Systems 30. https://proceedings.neurips.cc/paper/2017/hash/3f5ee243547dee91fbd053c1c4a845aa-Abstract.html.