Deep Learning: Making It Trainable
Geometry, Dynamics, and the Machine
Deep Learning: Making It Trainable
Training fails in ways that look deceptively alike.
A loss spike may be caused by a step that crosses a curvature boundary. It may be caused by a value that leaves the representable range. It may be caused by a noisy direction that only looks like signal. It may be caused by an implementation that moves the right mathematical object through the machine in the wrong way.
The trace alone does not name the cause. We need instruments.
This book develops those instruments through three lenses:
- geometry, which describes the high-dimensional arena;
- dynamics, which describes how the update moves through it;
- algorithms and systems, which describe the arithmetic, state, and data movement that make the update physically executable.
The lenses are coupled. A repair can move the failure instead of removing it: a larger step can accelerate a flat direction while destabilizing a sharp one; clipping can bound an extreme update while biasing the estimator; normalizing scale can change both the Jacobian and the curvature. The book therefore audits what each repair changes, what it preserves, and which new boundary it creates.
The governing question is:
What mathematical and physical conditions make a modern learning system trainable, and which diagnostic tells us which condition failed?
How the book moves
Chapters begin with something that breaks. You will predict a cause before the diagnostic is revealed. The mathematics then arrives as the smallest language that separates the competing explanations, and the chapter closes with a repair and an audit of its boundary.
The recurring order is:
Problem → Prediction → Instrumentation → Diagnosis → Theory → Solution → Audit
The book is inductive on purpose. A finished term can make an object feel settled before its assumptions are visible. We therefore build the primitive first and supply the conventional literature name when that name becomes useful for search, comparison, and primary sources.
Each chapter closes its ledger under the heading “Okay, so —”: what was inherited, what changed, what was instrumented, what is now established, and what remains unresolved. The unresolved entry is always a question; the next chapter opens with the phenomenon that forces it.
The route at a glance
| Stage | Question the reader gains the instrument to answer |
|---|---|
| On-ramp | Do I have the minimum tensor, loss, reverse-accumulation, mini-batch, and update mechanics needed to begin? |
| Act 0 · C01–C04 | What did the machine actually execute, retain, round, and move? |
| Act I · C05–C10 | Which directions and spectra should high dimension make typical, and what finite evidence turns that expectation into a verdict? |
| Act II · C11–C17 | How do derivatives, stochastic estimators, curvature, normalization, and matrix-valued updates change along a trajectory? |
| Coda | Which control identifies the cause of one incident rather than merely detecting it? |
| Beyond this volume | Where do these instruments return in IO-aware interaction, parallel recurrence, and feature-learning research? |
The detailed contents locates every theorem and subsection. This table is the argument to remember before opening that index.
Why this book
Goodfellow et al. (2016) owns the broad mechanics and modeling vocabulary of modern deep learning; this volume assumes only the small on-ramp above and does not rebuild that survey. Vershynin (2026) owns the general high-dimensional probability theory, while Act I turns selected results into finite diagnostic controls. Bottou et al. (2018) and Schmidt (2026) map optimization theory and practice; this book follows quantities across an actual trajectory instead of cataloging methods. Roberts and Yaida (2022) is the derivational companion for wide and finite-width statistical models, whereas the witnesses here measure where a declared approximation succeeds or fails. Austin et al. (2025) and Tazi et al. (2025) are systems-side companions for accelerator topology and distributed execution. The division of labor is deliberate: this book connects arithmetic, geometry, and dynamics through differential diagnosis, typed evidence, and executable controls; it does not claim new research theorems in place of those sources.
How to read this book
As a student. Read in order, and read each chapter as an investigation rather than a summary target. Stop at the Prediction and commit to an answer specific enough to fail; the gap between your answer and the diagnostic is the chapter’s actual content. Work the three exercise kinds as three different muscles: Pencil is derivation, Code is implementation, and Audit is judgment. Let none substitute for another. Keep your own closing ledger as you go: the object you can now name, the failure you can now detect, the rival you can now acquit, and the question still open. By the Coda, that notebook should read as an instrument panel, not a collection of formulas.
As a researcher or practitioner. Enter through your incident, not through Chapter 1. Complete the first fields of the Incident Card: symptom, ranked suspects, and declared contract. Do that before shopping for a remedy, then let the route table above send you to the missing instrument. A result from a clean example transfers exactly as far as its contract survives: the same object, the same estimator, the same axes and denominator, a comparable numerical contract, and a control that still separates the same rivals. When one of these breaks, the conclusion has quietly moved from identification to detection. Say so, and measure again.
As an instructor. The chapter structure is a lesson plan. Assign the opening phenomenon and require predictions before class; spend the class deriving the minimum theory that separates the rivals; put the deliberate error in the computational session and let it produce a reasonable-looking answer; grade the declared contract and the control, not the polish of the final artifact. A companion instructor note on assessment patterns, broken implementations, and assistance policy lives with the course materials rather than in this volume.
With a language model beside you. This book assumes you may study it with a model open in the next window, and treats that neither as cheating nor as a substitute for the work. What has changed is the price of fluency: a restatement, a derivation, an implementation, or a plausible diagnosis now costs seconds. What has not changed is the price of judgment: deciding what was actually computed, which object it is, which assumptions carry the conclusion, and what the evidence permits you to say. The reading contract below was written for exactly that judgment, and every clause doubles as a rule for model-assisted work.
Predict before you ask, and keep your prediction distinct from the model’s. Make generated code declare its object, shapes, reduction axes, denominator, and dtype before you accept it. Treat it as a proposed experiment, and add the assertion that would fail if the contract were violated. Use the model as tutor, translator, and implementer where translation is the bottleneck; use it above all as an adversary. Ask it for the rival mechanism, the hidden assumption, the counterexample, and the control under which your preferred explanation should fail. Do not let it serve as an authority: fluency is not evidence, and a polished explanation does not enlarge the permission granted by the proof or experiment beneath it. The test of any session is whether you could defend the result after the conversation disappeared: name the object, state the contract, run the control, and say what remains uncertain. That test, not possession of an answer, is what this book trains.
Proof modes and reading paths
Three proof labels carry different permissions. A Proof supplies the logical argument under the displayed assumptions. A Diagnostic proof sketch exposes the mechanism and the exact usable consequence while marking the technical steps it omits. A Proof with pointer states the theorem in the form used here and locates a complete external proof; the pointer is part of the contract, not permission to drop assumptions.
The acts deliberately change the lens balance. Act 0 is algorithmic-primary: formulas meet representation, state, and traffic. Act I is geometric-primary by construction. Act II turns the dynamic lens forward and uses geometry and systems as controls. The three lenses therefore recur across the book without occupying equal space in every act.
Read in chapter order for the full course argument. For practitioner triage, begin with C01, C03, C13, and C16 before returning to the supporting theory. For a theory-first route, read Act I after C04, then enter Act II through C12. All routes should end with C17 and the Coda, where the separate instruments must diagnose one event together.
The computational harness
One codebase grows with the argument. The first capability is deliberately small: a state that can carry stable moments through a stream and merge partial states without revisiting the data. Later chapters add precision diagnostics, spectral instruments, curvature products, and update audits.
Visible code is divided into:
- the equation;
- a 10–25 line mechanism kernel whenever the object permits one;
- two or three critical assertions that protect the interpretation;
- a compact provenance strip naming the claim, seed, dtype, device, estimator, and artifact;
- the full harness, linked rather than repeatedly printed.
Every published numerical result is either produced on the page or linked to a machine-readable provenance record. Hash verification, manifest activation, path handling, and JSON orchestration still execute, but their centralized implementation lives in the Executable Claim Contract. Printed code emphasizes the semantic change from one chapter to the next.
Reading contract
The book assumes graduate linear algebra and probability, plus working Python and NumPy. It does not assume that you have completed a deep-learning course. Before Chapter 1, the 20–30 minute on-ramp diagnoses the few mechanics used here and provides one minimal training-loop notebook. Any gap is repaired through three selected stops in the sibling volume, Deep Learning: Making It Learnable, not by requiring the entire earlier course.
The destination is not recall of a catalog. It is the ability to read a current optimization or systems paper and identify:
- the assumption carrying the claim;
- the quantity that was actually measured;
- the estimator, reduction axes, and denominator behind that quantity;
- the numerical and hardware contract;
- the regime in which the conclusion survives;
- the control that would distinguish the proposed mechanism from a plausible rival.
Colophon and edition note
The HTML edition is canonical; the PDF is a derived, content-equivalent print conversion. Differences are limited to pagination, float placement, line breaking, and other presentation requirements.
Two PDF layouts carry the same content:
- print PDF, with recto chapter and act openings;
- continuous screen PDF, without intentionally blank versos.
This volume is developed for DS 6210 — Computation II: Numerical Analysis & Optimization, also known as Algorithms for Deep Learning, in the School of Data Science at the University of Virginia. It is the advanced companion to Deep Learning: Making It Learnable.
© Heman Shakeri · Text CC BY-NC-SA 4.0 · Code MIT
This is a public working v0.x edition. A numbered chapter enters the public draft only after its mathematics, code, provenance, HTML, and PDF pass the repository audits. The present C01–C17 core, Coda, interludes, On-Ramp, Incident Card, and continuation map are author accepted; later revisions must pass the same gates again.
Suggested citation
Shakeri, Heman. 2026. Deep Learning: Making It Trainable. Public draft 0.2. School of Data Science, University of Virginia. https://shakeri-lab.github.io/opt-book/.