Tiny Transformer

A small GPT-style transformer is running in this page, writing text one character at a time. The bars show the probabilities it assigns to the next character, and the heatmap rows show where each of its attention heads is looking in the recent text.

Click a probability bar to force that character on the model. Click any character it has written to see the attention from the moment it was chosen. Low temperature plays the favorite every time; high temperature gambles.

A tiny character-level transformer. Trained on the Tiny Shakespeare dataset from Andrej Karpathy's char-rnn. The full forward pass runs here in plain JavaScript, no ML library. At this size it produces Shakespeare-shaped text, not meaning. The point is that you can watch the machinery choose every character.