How Knots works

1 The short version

A language model is a black box: billions of multiplications turn text into a guess about the next word, and nobody wrote down what any of the numbers inside mean. Knots is an attempt to label the inside of that box, place by place, with labels that are checked.

We trained a small language model (GPT-2 small's shape: 12 layers, 163 million numbers) from scratch on our own GPU operating system, AlphaOS. Then, with the model frozen, we:

found places inside it where its activity clusters (84 regions, in its 6th layer);
tied a knot at each place: a detector that fires whenever the model's activity points that way;
had AI agents read where each knot fires and write a one-line label for it;
had different agents, given only the label, predict where the knot fires on text they had never seen;
scored those predictions against what the knot really did, and against a dumb rival label that only names the knot's most common word.

ILLUSTRATION The whole loop. Press play to follow one knot through it.

The labels beat the dumb rival by a wide margin on the trained model, and, importantly, not at all on an untrained model, which is how we know they are reading something the model learned rather than something about words. The rest of this page explains every step, then shows what we measured and what we have not shown.

2 Inside a language model

The model reads text as tokens: whole words or pieces of words (" cat", " sat", "ing"). It turns each token into a list of 768 numbers, called its row. Then 12 blocks (layers) each read every row, let each row look back at the rows before it (that is how " sat" learns there is a cat), and add a correction to it. The running total that each block adds to is called the residual stream.

So every token has 13 versions of its row: before block 1, between each pair of blocks, and after the last. We call these planes 0 to 12. Plane 0 is just "which word is this, at which position". By plane 12 the row has to encode a guess about the next word. Everything interesting happens in between.

ILLUSTRATION Six tokens moving up through the planes. Each little bar chart stands for a row (real rows have 768 numbers, not 8, and these are made up). Press play: each block adds its correction, and arrows show a row looking back at earlier tokens.

Training is the slow part: show the model a stretch of text, measure how surprised it was by each next token (the loss), and nudge every one of its numbers to be less surprised next time. We did that 30,000 times, on 123 million tokens of web text (FineWeb).

MEASURED The model's loss over its 30,000 training updates (every 100th shown). Random guessing over GPT-2's 50,257 tokens scores 10.8; the finished model scores 4.46 on its training text and 4.77 on text it never saw. Training stopped at 30,000 updates (123 million tokens), where the curve had flattened: averaged over each block of 5,000 updates, the loss went from 4.79 to 4.78 across the last two. For scale: llm.c's reproduction of GPT-2 small, trained on 10 billion tokens of the same FineWeb text, reaches about 3.3, so ours is a young model.

3 What the weights are

Everything the model learned is stored in its weights: 163,306,496 numbers, fixed once training ends. They are not one big cube. They are about 150 separate flat grids (matrices) and short lists, each with a job. Almost every grid has 768 on one side, because each of them reads or writes a row of the stream.

The token table (50,432 × 768) holds one row per token: the row a word starts with.
Each of the 12 blocks holds six grids. Q, K and V (768 × 768 each) turn a row into a question, a key and a value: attention compares each token's question with earlier tokens' keys and takes in their values. The attention output (768 × 768) writes the result back into the stream. Then the feed-forward pair widens the row to 3,072 numbers (expansion, 3,072 × 768) and narrows it back (reduction, 768 × 3,072).
The output head (50,432 × 768) turns the last row into a score for every token: the next-word guess.
The small lists: each block's two layer norms and its biases (about 10,000 numbers a block), and five skip weights.

Below is every matrix of our trained model, drawn from its weights file. One pixel is one tile of 8 × 8 weights (the two big tables: 64 rows × 8 columns), so the shapes are true to scale.

MEASURED The trained model's weights (update 30,000). The bar shows where the 163 million numbers sit; pick a part to see its matrices. Size: how large the weights in each tile are (their root mean square). Sign: whether they lean positive or negative (their mean). Each picture is scaled to its own range; hover for the real values.

A few things you can see in the real weights:

The token table fades from left to right. GPT-2's tokenizer numbered its pieces roughly in the order it found them common, so low numbers are common pieces, and common tokens grew larger rows: the median row length is 0.81 for tokens 256 to 999 and 0.42 past token 20,000. The first 256 tokens are single bytes, rarely used on their own (0.53). The last 175 rows are padding no text ever uses; they keep their shrunken starting values (0.27).
Heads show as stripes, early on. Attention works in 12 heads of 64 numbers. In the attention output of blocks 1 to 4, the columns come in 12 bands of 64, and some heads write more than others (in block 2, the loudest head's weights are 2.1 times the quietest's). From block 5 on, every head's band is the same size to within 3% (block 12: 9%).
Feed-forward units show as lines. Each row of an expansion is one of 3,072 hidden units (the "neurons"); a bright row is a unit with large weights. In block 1 the top 1% of units have rows three times as long as the bottom 1%'s; deeper blocks are more even.
The output head leans on a few stream numbers. The head's long horizontal lines are single numbers of the 768 that every token's score reads heavily: number 131's weights are 28 times the typical number's, 154's 18 times, 48's 14 times.
The position table is all zeros. GPT-2 learns a row for each position. Our model instead turns its questions and keys by their position (rotary positions), so this table stays zero by design: 786,432 numbers that are always 0.

Our model is GPT-2 small's shape, not GPT-2 itself. It has GPT-2 small's sizes (12 blocks, width 768, 12 heads of 64, feed-forward 3,072, GPT-2's 50,257-token vocabulary, 1,024-token context) and is trained from scratch; OpenAI's released weights are never used. It differs in ways taken from recent practice: rotary positions with normalized questions and keys; relu(x)² in the feed-forward instead of GELU; an output head of its own (GPT-2 reuses the token table there, which is why GPT-2 small counts 124 million and ours 163 million); attention outputs, reductions and the head start at zero; logits gently capped at 30; and five skips that carry the stream at planes 5, 4, 3, 2 and 1 forward and add it in at planes 7, 8, 9, 10 and 11. Training learned these skip weights (each started at 1): ….

Where the knots are. Not in the weights. The weights are the machinery; the knots watch what the machinery produces. A knot is a direction in the 768-number space of the stream, the rows flowing between the blocks, and it is read at plane 6. The knots never change a weight.

ILLUSTRATION The weights (fixed grids, inside each block) and the stream (one token's row, flowing up). A knot reads the row at plane 6; the cube on the live page shows the knots' directions in that 768-number space, flattened to 3 axes.

4 A knot: a direction and a line

Picture each row as an arrow in space. In 768 dimensions we cannot draw it, but the idea works the same in 2. A knot is:

a direction (an arrow of its own),
a plane (which layer it watches),
and a threshold: how far along its direction a row must reach.

On every token, the GPU measures how far that token's row reaches along the knot's direction (its projection). If it reaches past the threshold, the knot fires on that token. That is all a knot is: a named detector. It reads the model; it does not change it.

ILLUSTRATION Drag the arrowhead to point the knot, and slide the threshold. Dots are rows (toy data in 2 dimensions); the ones past the dashed line fire. Try pointing it at one cluster.

Where does a knot's direction come from? Earlier experiments tied knots at words (the direction of a word's own embedding). The knots on the live page come from the next step: directions the model's activity itself clusters around.

5 Finding regions nobody named

We did not tell the model what to look for. With its weights frozen, we ran it on text it had never seen and collected every row at plane 6 (the middle of the model). Then the GPU ran k-means clustering: put down a few centres, assign every row to its nearest centre, move each centre to the middle of its rows, and repeat until they settle. Each centre marks a region: a direction the model's activity keeps returning to.

ILLUSTRATION k-means on toy rows. Step through it: assign each dot to the nearest centre (×), then move each centre to the middle of its dots. It settles in a few steps.

We ran it three times, with 4, 16 and 64 centres, and tied a knot at each region with enough rows: 84 knots. The coarse and fine regions nest: most of each fine region's rows fall inside one coarse region, far more often than chance. That is the tree you see as lines on the live page.

MEASURED How cleanly the 64 fine regions sit inside the 16 coarser ones (purity: the share of a fine region's rows inside its best coarse region), against what random assignment would give. The trained model's regions nest much more than the untrained model's.

Each region gets a provisional name from its most common words (like "I · it · you · that · she"). That name is just a list; the real description comes next.

6 Agents label, other agents test

For every knot we recorded every token where it fired on held-out text (65,536 positions), each firing checked against the GPU's own exact counts. Then two separate groups of AI agents took over:

ILLUSTRATION The labelling test. The explainer sees examples with answers; the tester sees only the label and new examples, without answers.

Try it yourself. Below is a real knot from the trained model, with real text it was measured on. First read where it fires and where it stays quiet. Then the label an agent wrote for it. Then predict, from the label alone, whether it fires on new examples.

MEASURED Real knots, real held-out text, real answers from the GPU. The marked token is highlighted; the model only sees text up to it.

How a label is scored. The tester's guesses are compared with what the knot really did. Precision: of the tokens it said would fire, how many did. Recall: of the tokens that fired, how many it caught. F1 combines them (their harmonic mean): 1 is perfect, and a label that just guesses scores about 0.5 on our tests, which are half firing and half not.

ILLUSTRATION Change the four counts and watch precision, recall and F1. Notice that saying "fires" everywhere gets perfect recall and poor precision.

The rival label. Many regions are mostly one word. A label like "the word the" might score well without understanding anything. So every knot's test also scores a rival that just names its commonest word, and what matters is how much the agent's label beats it. Weak labels got one revision from their mistakes and a fresh test. Over all 84 knots, the final labels score a median F1 of 0.79 against the rival's 0.18, ahead on 78 of 84.

7 The control that changed the headline

A known trap in this kind of work: automatic labels can score just as well on a model that learned nothing. So we ran the whole pipeline, unchanged and blind, on the same model after one training update (essentially untrained, loss 10.82).

The untrained model's labels scored a higher F1 (0.79 against 0.71 in the first round). Its regions are words in disguise: at initialization, plane 6 is little more than "which word is this", so naming the word is enough. The rival one-word label scores 0.65 there, and the agents' labels gain nothing over it. On the trained model the rival scores 0.14 and the labels gain +0.46.

MEASURED Each dot is one knot: how much its agent label beats the one-word rival, on the same blind test (first round, before any revision). Hover a dot. Trained: a median gain of +0.46, ahead on 74 of 84. Untrained: a median gain of +0.00, ahead on 35 of 84.

So the number to quote is the gain over the one-word label, not F1. It says the trained model's regions hold more than word identity (roles like "the subject of a new clause", "a preposition opening a phrase"), and that the agents capture it.

8 Reading the cube

The live page draws each knot as a dot in a cube. Each knot's direction is a list of 768 numbers, so we flatten 768 dimensions to 3 with principal component analysis: find the 3 axes along which the 84 knots' directions spread out most, and place each knot by its position on those axes. Knots close together point the same way inside the model. Nothing about the positions is made up.

ILLUSTRATION Projection, in miniature: points in 3 dimensions cast onto the 2 axes that keep them most spread out. It is like the best-angle shadow of an object. Drag to turn it.

The catch: 3 axes cannot hold everything. They capture 28% of how the knots' directions differ; the other 72% is invisible in the cube. Two knots that look close may still differ in directions the cube cannot show.

MEASURED How much of the 84 directions' spread each axis holds.

MEASURED The real knots on the first two axes (the cube seen from one side). Hover for each knot's label. Subject pronouns gather at one end of the first axis; names and places at the other.

9 Emits: asking a direction what it means

A label is one way to describe a knot. An emit asks the model directly. We add the knot's direction to every token's row at its plane (at one typical row's length), run the same text again, and see which next-word guesses rise. To know what "rise" means, we do the same with 16 random directions of the same length, and report how far above them each word rose (in standard deviations, σ).

ILLUSTRATION An emit: the knot's direction is added into the stream at its layer; the blocks above carry it into the guesses.

MEASURED Recorded emits on the trained model. Choose a knot; flip it to push the other way. Bars show how far above random directions each word rose. These are real runs; on the live page you can run them yourself.

10 Knots sending messages

A knot can also speak: wherever it fires, add a chosen vector (a carrier) to that row. Another knot, a listener, watches for that carrier further up the model. Can a message cross the model's 12 blocks? On a trained model: only if it is loud.

MEASURED A carrier spoken at plane 0 on the rows of " and", read back at each plane (how much of the carrier each spoken row carries). The grey band is what the model's own activity reads on rows the message cannot reach. Loud (32) crosses all 12 blocks; quiet (8) fades into the background by plane 10; whisper (1) is gone after plane 0. A pilot: 4 rows, one carrier.

Messages also crossed between two separately trained models: the GPU relayed what model A's listener heard to model B's speaker, and "KNOT" (32 bits) arrived bit for bit, with nothing arriving when the relay was off.

11 A knot that runs a program

So far a knot only detects, or speaks where it fires. A knot can also carry a small program that the GPU runs inside the model, as the model reads. Our first program tracks a simple state: are we inside a quotation? It opens on a quote mark after a space ( ") and closes on ", ." or ,". The knot speaks only where its program says, for example on the token that closes a quotation.

ILLUSTRATION The same rule the GPU runs, on a made-up sentence. Step through the tokens: the state opens and closes, and the knot speaks on the highlighted tokens. Switch between "inside a quotation" and "at the closing quote".

On the trained model, the GPU's program picked exactly the tokens the rule picks on real text it had never seen (8 of 8 closing quotes, 122 of 122 tokens inside quotations). Then we made it do something: at each closing quote, it pushes the model toward the word said, as in '…," said'.

MEASURED How often " said" becomes the model's first guess at the 8 closing quotes, by how hard the program pushes, against a random push and an unconditional push that lands almost everywhere. The program changes those 8 places and 3 others; the unconditional push changes 1,128. One stretch of text, 8 closing quotes: a pilot.

12 Looking for compersion

Compersion is joy at a partner's joy, especially with someone else. The word never appears in the model's training text (0 times in 300 million tokens), and no passage there even describes it. Can the model still hold the idea? We wrote matched scenarios, sets of four where only the situation or the feeling changes:

We fed them through the model and looked for a direction that separates compersion from the other three. Fit on one author's scenarios, it works fairly well. Tested on scenarios by authors it never saw, mostly the situation carries over (someone else, a relationship), and the feeling only weakly.

MEASURED How well a direction tells compersion apart from each group on held-out scenarios (0.5 is chance; with the scenario labels shuffled, directions score 0.39 to 0.62), fit on one author against tested across authors.

The honest summary: the model holds the parts of compersion, not a clean compersion detector. It never read the concept.

13 AlphaOS: the operating system underneath

None of this runs on a usual machine-learning framework. AlphaOS is our own operating system for the GPU, written in our own language and compiler. It starts one program on the GPU that never exits. Each of the GPU's processors runs a node: a loop that claims the next ready task (a piece of the model's computation, a knot's measurement) from a task graph, runs it, and marks it done. Training, knots, labels, emits: all tasks.

ILLUSTRATION Nodes claiming tasks from a graph: a task becomes ready when the tasks it needs are done.

Because every task does its arithmetic in a fixed order, training is bit-exact: the same model, to the last bit, on an RTX 3070 and an RTX 3090. A rerun reproduces every number.

The bug that blocked long runs. The rented machines' host software interrupts the GPU every 15 minutes. A GPU can only switch to another program at the edge of a piece of work, and AlphaOS's never-ending program had no edges, so the switch timed out and the driver killed the run. The fix: resumable nodes. Every 5 ms, between tasks, each node saves its small state and hands itself on to a fresh copy, so the GPU always finds an edge within milliseconds.

ILLUSTRATION Before: the switch request waits forever and times out. After: nodes hand themselves on every few milliseconds, and the switch slips in.

14 What is measured, and what is not

One model, one layer. The labels are of plane 6 of one GPT-2-small-shaped model, trained 30,000 updates (a short run: 123 million tokens, a held-out loss of 4.77).
The tests are half firing, half not. F1 measures how well a label separates the two, not how often a knot fires in normal text.
The agents are AI models (Claude). Explainers and testers were different agents, and testers never saw answers. The one-word rival and the untrained model are the controls; no random-direction control has been run for the labels.
The cube shows 28% of how the knots differ.
Messages, the program and compersion are pilots: few rows, one carrier, one stretch of text, one set of scenarios.
The "parse state" reading of plane 6 (tokens grouped by their job in the phrase) is our interpretation of the labels, not a separate test.
Every number links to its evidence in the project's repository (tests/observability/import/rtx3090/), with the raw outputs.

15 Glossary

Weights: The 163 million numbers training sets and then fixes: flat matrices and short lists inside the token table, the 12 blocks and the output head. Knots read the stream, not the weights.
Token: A word or piece of a word, as the model reads text.
Row: The 768 numbers the model keeps for one token at one layer.
Plane: One layer's version of the rows: plane 0 is the input, plane 12 the last.
Residual stream: The running total each block adds its correction to.
Knot: A named detector: a direction, a plane and a threshold. It fires on a token whose row reaches past the threshold along its direction.
Region: A direction the model's activity clusters around, found by k-means; a knot is tied at each.
Label: An agent's one-line description of where a knot fires.
F1: How well a label predicts firings: the harmonic mean of precision and recall.
One-word label: The rival: a label naming only the knot's commonest word. A label's gain over it is the measure that matters.
Emit: Adding a knot's direction into the model to see which guesses it pushes up, against random directions.
Carrier: A vector a speaking knot adds; a listener knot watches for it.
Program (gate): A small state machine a knot carries; the GPU runs it to decide where the knot speaks.
AlphaOS: Our GPU operating system: one never-ending GPU program whose nodes run a task graph.
Node: A loop on one GPU processor that claims and runs tasks.
Bit-exact: The same computation gives the same bits, every run and on every card.