Every model you use is a small number of linear-algebra moves, applied a huge number of times.

Four ideas the rest of the page is built on.

If you've done linear algebra before, skim this section — it's short on purpose. Everything here is used again, by name, in a later section.

What can a handful of vectors actually reach?

Every embedding table, every hidden layer, every weight matrix defines a space of vectors it can produce. Before asking what a network learns, it's worth asking what it could possibly reach in the first place — that's a vector-space question, and it has a precise answer.

Given a small set of vectors, the span is every point reachable by scaling and adding them. Two vectors in the plane usually span the whole plane. But if one is just a scaled copy of the other — they're linearly dependent — every combination lands on a single line, no matter how you scale them. The set is linearly independent exactly when this collapse doesn't happen, and dropping any vector genuinely shrinks the span.

A basis is a linearly independent set whose span is the whole space — the smallest ruler that reaches everywhere, first met in §0.4. The dimension of a space is simply how many vectors a basis for it needs: 2 for the plane, 768 for a typical sentence embedding, and, for a neural network layer, however many independent directions its weight matrix's columns actually point in — a number that can be smaller than the layer's width. A layer with 512 output units whose columns are secretly dependent isn't using 512 dimensions of capacity; it's using fewer, and every dependent column is wasted parameters. §3 gives this a single number.

A matrix is a machine for warping space — consistently.

§0.3 showed $Ax$ landing on a single output point. Slide that same matrix over every point in the plane at once, and a matrix stops looking like arithmetic and starts looking like what it is: a picture of space getting stretched, rotated, sheared, or flattened, in a way that keeps straight lines straight and the origin fixed. That picture is exactly what a neural network layer does to its input, before any nonlinearity gets involved.

Two transformations applied back to back — first $B$, then $A$ — combine into a single matrix, their product $AB$. This is why "stack two layers" and "multiply two matrices" are the same sentence in a neural network. Matrix multiplication is not commutative: $AB \neq BA$ in general, because doing the shear first and the rotation second lands you somewhere different than doing them in the other order.

One number that says whether a transformation destroys information.

§2's warped grid changed the shape of a unit square. The determinant $\det(A)$ is by how much it changed the square's area — a single signed number that captures how a transformation scales space, in any number of dimensions (area in 2-D, volume in 3-D, and so on).

Three facts follow directly from "determinant = area-scale-factor," and all three matter for what a layer of a network can and can't do:

• $\det(A) = 0$ means singular. The transformation flattens the plane onto a line (or the line onto a point) — a whole direction of input is thrown away. Two different inputs can now map to the exact same output, so $A$ has no inverse: there's no way to undo the operation and recover what went in. In a network, a weight matrix that becomes singular during training has permanently lost representational capacity along some direction — a hidden form of the "dead neuron" problem.
• The sign of $\det(A)$ says whether orientation flips. Positive keeps a clockwise triangle clockwise; negative mirrors it. Magnitude and sign are independent facts about the same transformation.
• Determinants multiply under composition: $\det(AB) = \det(A)\det(B)$. Stack a volume-preserving rotation ($\det{=}1$) with a volume-preserving shear ($\det{=}1$) from §2, and the combined transform still preserves area exactly — even though the shape it produces looks nothing like either step alone.

The directions a transformation doesn't rotate — only stretches.

§2 showed that most vectors change direction under a matrix. A special few don't: they come back out pointing exactly the same way (or exactly opposite), just longer or shorter. Those are eigenvectors, and the amount they stretch by is their eigenvalue: $Av = \lambda v$.

Not every real matrix has real eigenvectors — try a pure rotation (revisit the hero demo's "rotate" preset). Rotating every vector by a fixed angle never leaves any direction unchanged, so the characteristic equation's discriminant goes negative and the eigenvalues come out complex. A negative discriminant in the readout above is exactly that case.

Why applying a matrix 50 times is either boring or catastrophic.

If $A$ has a full set of independent eigenvectors, it factors as $A = PDP^{-1}$: $P$'s columns are the eigenvectors, and $D$ is diagonal with the eigenvalues down the middle. This is diagonalization, and it turns a painful question — what does $A$ applied 50 times look like? — into an easy one, because $A^n = PD^nP^{-1}$, and raising a diagonal matrix to the $n$-th power just means raising each eigenvalue to the $n$-th power.

The three presets above are the three futures a repeated linear update can have, and they map directly onto training pathologies with names you've likely already seen:

• Exploding. $|\lambda_{\max}| > 1$: every application stretches the dominant direction further. Hidden states and gradients grow exponentially in the number of steps — the exploding-gradient problem.
• Vanishing. $|\lambda_{\max}| \lt 1$: everything shrinks toward zero. Gradients from many steps back become too small to update anything — the reason plain RNNs struggle to remember long-range context, and part of the motivation for LSTMs, GRUs, and careful weight initialization (Glorot & Bengio, linked in the hero).
• Stable. $|\lambda| = 1$ for every eigenvalue — the orthogonal case, where $A$ only rotates, never stretches. Lengths are preserved exactly, no matter how many times $A$ is applied. Keeping weight matrices close to orthogonal is a real, deliberate training technique for exactly this reason.

The directions your data actually varies along are eigenvectors.

Take a cloud of data points, centre it at the origin, and build its covariance matrix $\Sigma$ — a matrix that records how much each pair of coordinates varies together. It's always symmetric ($\Sigma_{xy} = \Sigma_{yx}$ by construction), and the spectral theorem guarantees every real symmetric matrix has real eigenvalues and perpendicular eigenvectors — no complex-eigenvalue surprises like §4's pure rotation. Those perpendicular eigenvectors are the principal components: the axes the data spreads out along the most, sorted by their eigenvalues, which are exactly the variance along each axis.

With only two coordinates this is a nice picture; the payoff shows up in the dimensions you can't draw. A dataset of 768-dimensional sentence embeddings still has a $768\times768$ covariance matrix with 768 eigenvectors — but if the first two or three eigenvalues capture most of the total variance, projecting onto just those eigenvectors keeps most of what the data was actually saying while throwing away the rest. That's how a 768-dimensional embedding space gets compressed down to the 2-D scatter plot on a slide, and it's the same operation — on a much larger scale — behind whitening layers and dimensionality reduction in real training pipelines.

Meaning, encoded as a direction in space.

An embedding is a vector a model has learned to assign to a word, sentence, or image, chosen so that things with similar meaning end up as vectors that point similar ways. "Similar way" is not a metaphor here — it's §0.2's cosine similarity, computed directly on the embedding vectors, and it's how every semantic search, recommendation system, and retrieval-augmented model on the market ranks results.

Because embeddings are ordinary vectors, they inherit ordinary vector arithmetic (§1's closure again). The most-quoted demonstration of this, from the original word2vec work linked in the hero, is analogy-by-subtraction: the offset from "man" to "woman" is, roughly, the same offset as "king" to "queen" — because both offsets point in the direction a model has learned to associate with gender, independent of the royalty concept riding along with it.

A layer is $Wx+b$. Depth is what happens when you interleave a bend.

A single neural-network layer computes $y = \phi(Wx + b)$: a linear transformation (§2) with a bias offset, followed by a nonlinearity $\phi$ like $\tanh$ or ReLU applied number-by-number. The matrix part alone can rotate, scale, and shear. It can never bend a straight decision boundary into a curved one — composing linear maps just gives another linear map, $W_2(W_1x) = (W_2W_1)x$, one matrix again, exactly as in §2. The nonlinearity is the only thing standing between "one big matrix" and "deep network."

Scale this picture up and you have the recipe for everything from a small classifier to a large language model: alternating linear transforms and nonlinearities, with attention layers adding one more linear-algebra primitive on top — each token's relevance to every other token, computed as a scaled dot product $QK^\top$ between query and key vectors, exactly the inner product from §0.2, just computed for every pair of tokens in the sequence at once via one matrix multiplication.

Represent. Transform. Decompose. That's the whole page.

Back in the hero: almost anything a machine-learning system does to data sorts into one of three moves. Here is every section of this page, filed under the move it belongs to, next to the real system that leans on it.

None of the individual ideas on this page are exotic — every one of them is standard, decades- or centuries-old linear algebra. What's changed is scale: instead of a $2\times2$ matrix, a transformer layer's weight matrix might be thousands of rows and columns wide, and instead of one matrix multiplication, a modern model runs hundreds of them per token, per layer, per forward pass. The mathematics underneath does not change. Only the size of the arrows does.