Teach a machine its first words

What does an LLM actually eat?

The whole world, as one long string

A large language model is not taught grammar, facts, or manners. It is handed text — trillions of characters of it — and one instruction: predict what comes next.

Nobody labels anything. The text is its own answer key, because the next character is always sitting right there. That trick is called self-supervision, and it is why language models could be trained on the whole internet: the labelling is free.

Your machine gets a smaller world: two hundred characters about the taxi from lesson 1. Same game, same maths, same loop.

How does text become numbers?

The machine has never seen a letter

Models compute with numbers, so the first step of every language model is a swap: each piece of text gets a number, and from then on the machine only ever sees the numbers.

Real models chop text into tokens — chunks of a few characters. Yours uses single letters, which is the same idea at its smallest.

Look at the mapping. This is the entire interface between human language and everything the machine does. When you type into a chat box, this substitution is the first thing that happens.

What is the model actually predicting?

One question, asked forever

Here is the whole game. Show the model a character. It must give a probability to every possible next character — not one answer, a full spread of belief.

Pick a letter below and look at the amber bars: that is your model's current belief about what follows. The white ticks are what the corpus actually does.

Right now the model believes nothing — every option equally likely, because all its dials start at zero. Perfect ignorance. Training is the process of dragging those bars onto the ticks.

How do you score a guess that is a spread?

Surprise is the loss

The truth arrives — say the next character really was h. How wrong was the model? Look at the probability it had given h, and take the negative logarithm of it.

That number is the model's surprise. Gave h a 90% chance? Barely surprised, tiny loss. Gave it 1%? Enormous surprise, enormous loss. This scoring rule is called cross-entropy, and it is the loss function behind every language model ever shipped.

The maths panel shows it live, with your model's real numbers in it. Note the beautiful part: the gradient of this whole apparatus collapses to probability − truth. The nudge for each dial is just how much belief it wrongly placed.

So what does training an LLM look like?

Watch it learn to talk

Press play. Every step: guess the next character everywhere in the corpus at once, measure the average surprise, nudge every dial by probability − truth. The same loop as the taxi fare — just aimed at text.

Watch the speech panel. At step zero the machine babbles uniform noise. Within a hundred steps it has discovered spaces, then th, then the . Nobody taught it a single one of those — they fell out of shrinking surprise.

This is genuinely how the big ones are trained. GPT ran this loop on trillions of tokens for months across thousands of GPUs. You are running it on two hundred characters in a browser tab. The loop does not change — only the bill.

Where does everything it learned live?

The brain is a file. Save it.

Everything your machine now knows — that q is rare, that the is everywhere, the shape of every habit it picked up — lives in one place: the grid of dials. There is no other storage. The weights are the knowledge.

Which means saving a trained model is just writing the numbers down. Labs training for months write these numbers to disk every few hours — a checkpoint — so a crash costs hours, not months. When a lab 'releases a model', the release is this file, grown large.

Press the button. That file is your model's entire mind — every number it learned, readable in any text editor. Llama's weights are the same object at 140 gigabytes.

Is ChatGPT really just this, bigger?

The honest gap between your model and GPT

Yes and no, and both halves matter.

Yes: the objective (next token), the loss (cross-entropy), the update (gradient descent), the state (a weight file) — identical. You have now personally operated every one of them.

No: your model remembers one character. GPT's transformer attends over a hundred thousand tokens, and between input and output sit hundreds of layers that build meaning before predicting. That architecture is the story's 2017 scene — and where this course goes next.

But hold the true thing you built: when a chatbot answers you, it is doing nothing you have not done today — reading numbers, running them through learned dials, and betting on what comes next, one token at a time.