Answer: Transformers lack some useful CNN inductive biases, especially locality and translation equivariance, which can help CNNs generalize with limited data. With enough data and compute, a Transformer's weaker spatial prior can instead offer useful flexibility.

CNN locality and translation equivariance can improve sample efficiency, while a weaker spatial prior can become flexible at scale. Dataset size is one important axis, but compute, augmentation, pretraining, architecture, and optimization recipe also affect the comparison; no single threshold determines the winner.

Answer: BERT uses masked language modeling (predicting masked tokens with bidirectional context), allowing it to use bidirectional attention. GPT uses autoregressive language modeling (predicting the next token), requiring causal masking to prevent attending to future tokens.

The pretraining objective determines the permissible attention pattern. Masking creates a denoising problem where context can flow both directions, while autoregressive generation requires left-to-right causality. This architectural choice directly follows from the training objective.

Answer: In both architectures, `<cls>` is prepended to the input (patches in ViT, tokens in BERT), and its representation after the Transformer is used for classification. Through self-attention, it aggregates information from all inputs and serves as a learnable global representation.

A dedicated pooling token gives the objective an explicit location whose representation can learn to aggregate the sequence. It has direct attention paths to all inputs, but it is a design choice rather than inherently superior: mean pooling, attention pooling, or a designated final token can also work when the architecture and objective support them.

Answer: T5 converts all tasks to text-to-text format by adding task description prefixes (e.g., "Summarize:", "Translate English to German:") and generating the target as text. This enables a single model and objective to handle multiple task types and allows multitask learning.

The unified framework is powerful because it eliminates task-specific architectures and loss functions. The same encoder-decoder model with the same objective handles generation, classification, and structured prediction. This also enables zero-shot transfer and efficient multitask training.

Answer: In controlled empirical studies, loss often follows an approximately linear trend on log-log plots as model size, data, or compute is varied. Because compute couples model size and training tokens rather than forming an independent multiplicative factor, fitted scaling laws can be used to estimate a compute-efficient frontier instead of simply maximizing one quantity.

These are empirical regularities over measured regimes, not a universal product formula. Their practical value is comparative: under an explicit compute budget and training recipe, they help estimate how to divide resources between parameters and data and where extrapolation becomes risky.

Answer: It gives the label and teacher objectives separate readout pathways that can specialize, while both still train the shared trunk

The classification head reads a token trained toward ground-truth labels, while the distillation head reads a token trained toward teacher targets. Separate readouts reduce direct competition for one summary vector and let the two tokens specialize. They do not make the objectives independent: both losses backpropagate through shared Transformer parameters, where their gradients can still agree or conflict.

Answer: Scale parameters and training tokens together under a compute budget; its fitted regime suggested roughly 20 training tokens per parameter

The study found that several earlier large models were undertrained for their size. In its measured regime, allocating compute to both model size and more training tokens produced a smaller, more thoroughly trained Chinchilla model that outperformed the larger Gopher on many evaluations. The roughly 20:1 figure is an empirical rule from that setup, not a universal constant.

Answer: Attention is a query-local nonparametric regressor that retains its observed key/value pairs as the dataset; a fixed-state SSM compresses the prefix into bounded state

For exact softmax attention, the projected keys and values are the data traversed by each new query. Caching them avoids recomputing earlier projections, but the dataset still grows with the prefix. A fixed-state SSM chooses a different statistical contract: it updates a bounded summary, gaining constant state size while generally accepting lossy compression. Keeping every K/V row does not by itself guarantee exact numeric recall; the query-dependent weighted average still matters.

Answer: Token-dependent retention and input injection, interpretable as a learned forgetting and update schedule

Selectivity lets the recurrence retain or inject information differently for different tokens. In regression language, that plays a role analogous to learning which history to weight and how strongly to update. This is an interpretive bridge, not an exact derivation of Mamba from one least-squares objective, and it says nothing about selective-scan implementation details.

Answer: The claim that unrestricted $M$ with $w=1$ and no regularizer directly yields the softmax weighted average, because such a function can interpolate the observed pairs instead

The exact statement fixes one constant prediction $c$ at query $q$ and minimizes $\frac12\sum_\tau\kappa(q,k_\tau)\lVert c-v_\tau\rVert^2$. Stationarity gives $\sum_\tau\kappa(q,k_\tau)(c-v_\tau)=0$, so $c$ is the normalized kernel-weighted average. The kernel supplies the weights; the local-constant fit supplies the average.

Answer: The pair $(S_t,z_t)$, where $S_t=\sum_{\tau\le t}\phi(k_\tau)v_\tau^\top$ and $z_t=\sum_{\tau\le t}\phi(k_\tau)$

$S_t$ supplies the weighted numerator and $z_t$ supplies the normalizing denominator: $\phi(q)^\top S_t/(\phi(q)^\top z_t)$. Both are sufficient statistics for that factorized kernel. This exact recurrence does not make ordinary softmax attention a fixed-state method unless its kernel is replaced by, or represented with, an appropriate factorization.

Answer: Some architecture and prompting confounds are controlled, but horizon-dependent internal planning work is not matched by counting generated tokens alone

Comparable added parameter counts and a shared prompting/sampling recipe are useful controls, but they do not equalize total per-example inference work. A compute-matched rematch would fix the backbone, data, prompt, sampling, hardware, and output cap; include the planner's internal horizon work; allocate equal measured FLOPs or wall time; and report accuracy versus compute with uncertainty and a memory-only baseline. That would be a rematch, not a referendum.