word2vec learning dynamics: theory shows PCA features

Berkeley AI Research provides a closed-form theory showing that, in realistic practical regimes, word2vec learning reduces to unweighted least-squares matrix factorization and that the final learned representations are given by PCA. The authors solve the gradient flow dynamics in closed form and show embeddings trained from small initialization learn one concept at a time in discrete, sequential steps.

What the theory shows

The theory proves that when embedding vectors are initialized randomly and very close to the origin, and under four mild approximations, word2vec learns orthogonal linear concepts one at a time. The four approximations are: a quartic approximation of the objective around the origin, a particular constraint on algorithmic hyperparameters, sufficiently small initial embedding weights, and vanishingly small gradient descent steps. None of these approximations involve assumptions on the data distribution.

The learned latent features are the top eigenvectors of a corpus-defined matrix M* with entries

M*_{ij} = (P(i,j) - P(i)P(j)) / (1/2 (P(i,j) + P(i)P(j)))

where P(i,j) is the co-occurrence probability for words i and j, and P(i) is the unigram probability for word i. Constructing and diagonalizing M* from Wikipedia statistics, the authors find interpretable topic-level directions: the top eigenvector selects words associated with celebrity biographies, the second selects words associated with government and municipal administration, and the third is associated with geographical and cartographical descriptors.

Because the theory reduces the learning problem to unweighted least-squares matrix factorization, the model’s training trajectory corresponds to computing a sequence of optimal low-rank approximations of M*. Concretely, with small initialization the embeddings increment their effective rank in discrete steps, and once a linear subspace is learned it does not rotate, so these subspaces serve as the model’s learned features. The closed-form solution therefore identifies those features as PCA eigenvectors of M*.

Learning dynamics and empirical checks

The blog presents visual evidence and experiments showing rank-incrementing steps in the weight matrix and three time slices of the latent embedding space where embedding vectors expand into subspaces of increasing dimension at each learning step. By inspecting words that align with each learned direction, each discrete piece of knowledge corresponds to an interpretable topic-level concept.

The authors compare true word2vec, their approximate model, and a classical alternative on the standard analogy completion benchmark as a coarse indicator of agreement. The published numbers are: word2vec achieves 68% accuracy, the approximate model achieves 66%, and the classical PPMI method gets 51%.

The theory is applied to study the emergence of abstract linear representations that correspond to binary concepts such as masculine/feminine or past/future. The authors find these linear representations build up over learning as a sequence of noisy learning steps and that their geometry is well described by a spiked random matrix model. Early in training, semantic signal dominates, but later in training noise may begin to dominate, causing degradation of the model’s ability to resolve the linear representation.

Why it matters

This is one of the first closed-form, distribution-agnostic descriptions of feature learning for a practical natural language task. By giving an explicit matrix M* defined solely in terms of corpus statistics and hyperparameters, the theory makes testable predictions about which features word2vec will learn and in what order. That connection explains why word2vec embeddings show striking linear structure and supports why embedding directions enable tasks such as analogy completion.

What to watch

Compare the theory’s predictions to full word2vec across other corpora and hyperparameter settings to test how the four approximations affect fidelity. Also monitor when and how noise begins to dominate learned linear representations, which the authors identify as a mechanism for late-stage degradation of some semantic directions.

Learn more in the paper linked from the original Berkeley AI Research post.