Knots atlas

A language model reads a token through 13 planes: its embedding (plane 0), then the stream after each of its 12 layers. The first atlas mapped one of them, plane 6. This one maps all thirteen.

At every plane the GPU found the regions the model's activity clusters into (k-means at 4, 16 and 64 regions, on held-out text) and tied a knot at each. Separate AI agents read where each knot fires and wrote a label; different agents, given only the label, predicted where it fires on text they had not seen. Two controls ran through the same pipeline, blind: the same model before training, and random directions set to fire as often as each plane's knots. Every number below is measured; hover for detail, and each chart has its numbers underneath.

MEASURED How much more an agent's label explains than naming the knot's commonest word, plane by plane (median over the plane's knots; the label's F1 minus the one-word label's, both on the same blind test). Above zero: the plane holds more than word identity, and the agents found it.

MEASURED Every region of the 64-region dictionary at every plane (height: its share of 4,096 held-out rows), joined to the next plane's regions by the rows they share (8 rows or more). The same rows at every plane: one text, one frozen model. Hover a region to follow its rows up and down the model; click it for its label.

Hover a region.

MEASURED What happens to each plane's 64 regions at the next plane: carried (80% or more of its rows stay together), split (two or more regions take a fifth or more each), or scattered. The trained model, then the untrained one.

MEASURED How alike any two planes' 64-region partitions of the same rows are (normalized mutual information: 1 the same partition, about 0.13 chance). The trained model on the left, the untrained one on the right.

The cube at each plane

Each plane's knots in that plane's own principal axes, on the main page: