1  The Count Is Not the Cost: Arithmetic Intensity and the Memory Hierarchy

Geometric quiet · Dynamic supporting · Algorithmic primary

A copy performs no arithmetic. A fused affine transform performs \(2N\) operations. Their compulsory traffic — and therefore their bandwidth time bound — is the same.

Both computations act on \(N\) stored FP32 values. The copy moves at least \(8N\) bytes: one read and one write. The fused transform \(y_i=ax_i+b\) has the same dominant traffic when \(a\) and \(b\) remain in a fast store. One operation count is zero; that count does not identify elapsed time.

Now compare a square matrix product of order \(n=512\). The conventional count is \(2n^3\) operations. Its compulsory traffic under an idealized one-read, one-write model is \(12n^2\) bytes, so its arithmetic intensity is \(n/6=85.33\) operations per byte. The affine transform has intensity \(2/8=0.25\). On a hypothetical machine whose peak arithmetic-to-bandwidth ratio is 20, one falls below the balance point and the other lies above it. The larger operation count can be the computation with more opportunity to reuse data.

ImportantPrediction

On one fixed machine, a proposed method halves the multiplications and doubles compulsory traffic. Choose a prediction before continuing: faster, slower, or undetermined. Which traffic boundary and machine balance would let you defend that choice?

1.1 Two resource clocks

Let \(F\) be the required operations, \(Q\) the bytes transferred through a declared boundary, \(P_{\max}\) peak arithmetic rate, and \(B\) peak bandwidth. The two unavoidable time lower bounds are

\[ T\geq \frac{F}{P_{\max}},\qquad T\geq\frac{Q}{B}. \tag{1.1}\]

With arithmetic intensity \(I=F/Q\), achieved performance \(P=F/T\) obeys the following deliberately small theorem.

Theorem 1.1 (Two-resource performance ceiling) Under the declared operation count, traffic boundary, peak arithmetic rate, and peak bandwidth,

\[ P\leq \min(P_{\max},BI). \tag{1.2}\]

Proof

Invert each inequality in Equation 1.1 and multiply by \(F\). The arithmetic bound gives \(P\leq P_{\max}\); the traffic bound gives \(P\leq BF/Q=BI\). Both must hold.

The bend occurs at the machine balance \(I_*=P_{\max}/B\). This diagram is the roofline model (Williams et al. 2009). It is an upper envelope, not a prediction that a kernel reaches either roof.

  1. Read the committed C01 model calculation.
  2. Declare one hypothetical machine balance.
  3. Plot the bandwidth and arithmetic ceilings.
  4. Place the vector transform and matrix product on the same axes.
  5. Check the registered arithmetic intensities before interpretation.
import matplotlib.pyplot as plt
import numpy as np

# [1]
claim = verify_claim("c01-roofline-001")
# [2]
balance = claim["result"]["machine_balance_flop_per_byte"]
# [3]
intensity = np.logspace(-2, 3, 300)
ceiling = np.minimum(intensity / balance, 1.0)
fig, axis = plt.subplots(figsize=(7.2, 3.8))
axis.loglog(intensity, ceiling, color="#232D4B", linewidth=2)
# [4]
points = {
    "fused affine": claim["result"]["vector_affine_intensity"],
    "512×512 product": claim["result"]["gemm_intensity"],
}
for label, value in points.items():
    axis.scatter(value, min(value / balance, 1.0), s=55)
    axis.annotate(
        label,
        (value, min(value / balance, 1.0)),
        xytext=(5, 8),
        textcoords="offset points",
    )
axis.set(
    xlabel="arithmetic intensity (operation/byte)",
    ylabel="fraction of arithmetic peak",
)
axis.grid(True, which="both", alpha=0.25)
fig.tight_layout()
# [5]
assert np.isclose(points["fused affine"], 0.25)
assert np.isclose(points["512×512 product"], 512 / 6)
print("claim c01-roofline-001: verified")
claim c01-roofline-001: verified
Log-log roofline chart with a sloped bandwidth ceiling meeting a flat arithmetic ceiling at intensity 20. The vector transform is at intensity 0.25 on the sloped branch; the order-512 matrix product is at intensity about 85 on the flat branch.
Figure 1.1: One operation count is not one cost. Under the declared two-resource model, the vector transform lies below the balance point while the reused matrix product lies above it.

1.2 A zero-count kernel still consumes a clock

The model now needs a measured control. The committed observation times a 64 MiB FP32 payload copy and an order-512 FP32 matrix product on one Apple M4 Max CPU. It also sweeps nearby sizes to establish two local empirical ceilings in the same process. Those ceilings are not vendor peaks. They are reference rates that make the achieved fractions inspectable without claiming that a laptop CPU represents an accelerator.

  1. Read the pinned local measurement and its protocol.
  2. Extract the selected copy and matrix-product rates.
  3. Normalize each rate by its own local empirical ceiling.
  4. Plot resource-specific fractions rather than a false common unit.
  5. Check that the displayed values match the committed observation.
# [1]
measurement = verify_claim("c01-local-measurement-001")
protocol = measurement["measurement_contract"]

# [2]
copy_result = measurement["result"]["copy_by_mib"]["64"]
matmul_result = measurement["result"]["matmul_by_order"]["512"]

# [3]
fractions = np.array([
    measurement["result"]["copy_64_fraction_of_local_bandwidth_ceiling"],
    measurement["result"]["matmul_512_fraction_of_local_arithmetic_ceiling"],
])

# [4]
fig, axis = plt.subplots(figsize=(7.2, 2.8))
axis.barh(
    ["64 MiB copy", "order-512 product"],
    fractions,
    color=["#9C2F2F", "#232D4B"],
)
axis.set(
    xlim=(0, 1.08),
    xlabel="fraction of the matching local empirical ceiling",
)
axis.grid(axis="x", alpha=0.22)
fig.tight_layout()

# [5]
assert protocol["statistic"] == "median elapsed time"
assert np.isclose(copy_result["median_seconds"], 0.000947375)
assert np.isclose(matmul_result["median_seconds"], 0.0001039795)
print(
    f"copy: {1e3 * copy_result['median_seconds']:.3f} ms; "
    f"{copy_result['achieved_gbyte_per_second']:.1f} GB/s"
)
print(
    f"matmul: {1e3 * matmul_result['median_seconds']:.3f} ms; "
    f"{matmul_result['achieved_gflop_per_second']:.1f} GFLOP/s"
)
copy: 0.947 ms; 141.7 GB/s
matmul: 0.104 ms; 2581.6 GFLOP/s
Two horizontal bars. The 64 MiB copy reaches about 0.875 of the local empirical bandwidth ceiling, while the order-512 matrix product reaches the local empirical arithmetic ceiling.
Figure 1.2: A zero-FLOP copy has a positive measured cost. Each bar is normalized by a separately measured local ceiling for its own resource, so the figure supports a resource diagnosis rather than a cross-kernel speed ranking.

The copy moved a 64 MiB payload across the declared read-plus-write boundary, so the traffic count was 128 MiB. Its median time was about \(0.947\) ms and its achieved bandwidth was \(141.7\) GB/s, or \(87.5\%\) of the best copy rate in the declared size sweep. The matrix product reached \(2581.6\) GFLOP/s in the same software process. These observations do not say that one kernel is generally faster: they show that zero counted arithmetic does not mean zero elapsed time, and that each kernel must be compared with the roof for the resource it actually demands.

1.3 The denominator is an algorithmic choice

Traffic has no meaning until its boundary is named. Bytes between registers and a first-level cache, between cache and main memory, and between host and accelerator are different denominators. Tiling changes \(Q\) by retaining a working set near arithmetic units. Fusion changes \(Q\) by keeping an intermediate value from making a round trip. Recompute can deliberately increase \(F\) to reduce \(Q\). C11 will return to exactly that trade when it checkpoints derivative state.

This also foreshadows the numerical-stability thread that C02 seeds. A fused operation can change both traffic and rounding: eliminating an intermediate store may eliminate one rounding step. A performance transformation is therefore part of the numerical contract, not merely a scheduling detail.

WarningNamed wrong answer: fewer operations means faster

Fewer operations is evidence only after the limiting resource is identified. It may improve a compute-limited kernel, do nothing to a bandwidth-limited kernel, or make the kernel slower by destroying reuse or fusion.

1.4 Hardware-claim boundary

The roofline is useful because it makes omissions visible. Peak values may be unreachable; compulsory traffic can underestimate actual traffic; latency, launch overhead, synchronization, occupancy, instruction mix, and access regularity remain outside Equation 1.2. CPU simulation of FP16 behavior is not evidence of accelerator timing. Any measured claim later in this book must name the device, software stack, warmup, repetitions, statistic, and traffic boundary—or be labeled a model calculation or pinned artifact.

TipField note: read a speedup claim in three passes

First locate the mathematical work and the denominator behind its reported rate. Then locate the resource boundary: bytes, arithmetic, latency, communication, or some mixture. Only then compare measured elapsed time, including baseline, device, precision, warmup, repetitions, statistic, and failure accounting. A speedup is an observation about that full comparison, not a synonym for a smaller operation count.

1.5 Check yourself

A kernel performs \(10^{10}\) operations and moves \(10^9\) bytes. On a device with \(P_{\max}=10^{13}\) operation/s and \(B=5\times10^{11}\) byte/s, its intensity is what? What are the machine balance, active roofline branch, and model ceiling? If \(F\) is halved while \(Q\) is unchanged, which lower bound changes and which one does not?

1.6 Okay, so —

  • Inherited: asymptotic operation counts remain indispensable scaling summaries.
  • Changed: elapsed-time reasoning now requires a declared traffic denominator and machine balance.
  • Instrumented: one exact roofline claim and one pinned laptop measurement separate modeled ceilings from achieved rates.
  • Established: \(P\leq\min(P_{\max},BI)\) follows from two resource lower bounds.
  • Unresolved: Which state must an online computation retain when real executions move data through multiple memory levels?

1.7 Sources and further reading

The roofline formulation and its machine-balance interpretation follow Williams et al. (2009). For systems-scale companions that carry rooflines into accelerator topology, collectives, and large-model execution, see Austin et al. (2025) and Tazi et al. (2025).

Reading order. Start with Williams et al. (2009) for the two-resource diagram, then return to this chapter’s denominator and measurement sections before using Austin et al. (2025) or Tazi et al. (2025) for multi-device scale.

1.8 Exercises

  1. (Pencil.) Recompute the matrix-product intensity if one input matrix is reread four times. State the traffic boundary.
  2. (Code.) For three hypothetical machine balances, redraw the figure and report which classifications change. Do not time the plotting code.
  3. (Pencil.) A checkpointing rule adds \(30\%\) arithmetic and removes half the retained-state traffic. Derive the intensities before and after, then state the machine-balance range in which the trade can improve the roofline bound.
  4. (Code.) Re-run the local protocol with three payload sizes and three matrix orders. Declare device, stack, warmup, repetitions, statistic, and traffic boundary. Compare medians and dispersion; do not call the best observed rate a vendor peak.
  5. (Audit.) Paper audit: Find one reported speedup and complete the Claim, Mathematical object, Evidence culture and interface, Numerical and hardware contract, and Transfer verdict fields of the Paper Autopsy Protocol. Separate operation count, modeled traffic, measured elapsed time, hardware identity, and baseline. Which claim is actually supported?