How an LLM really works

When you type a question into ChatGPT, Claude or Gemini it feels like talking to someone who “knows”. In reality there is a mechanism that is surprisingly simple in its core idea and incredibly complex in its scale. Understanding it changes how you use it: you stop trusting it where you shouldn’t and start using it where it shines.

A machine that predicts the next word

At the heart of every language model is a completion game. Given some text, the model computes how likely every possible continuation is and picks one. Then it appends the chosen word and starts again. A 500-word answer is the result of roughly 650 of these steps in a row.

Try it below: you can pick the next word yourself or let the “model” choose. The probabilities are illustrative, but the mechanism is exactly this.

Two things to notice:

1. The model does not plan the whole sentence in the human sense. Every choice shapes the next ones. That is why, once it heads down the wrong path, it tends to continue consistently along it (one source of hallucinations).
2. The answer is not fixed. Ask the same question twice and you may get different answers, because each choice is “drawn” from the options according to their probabilities. Temperature controls how daring the model is: low, it almost always picks the most likely; high, it picks unlikely ones more often.

Tokens: the machine’s words

The model sees neither letters nor whole words: it sees tokens, pieces of text that an algorithm (the tokenizer) learned to split based on how often they appear. Common words are a single token; long or rare words get split.

• “house” → 1 token
• “intelligence” → probably 1 or 2 tokens
• “antidisestablishmentarianism” → 5 or 6 tokens

Useful rules of thumb:

• In English 100 words are about 130 tokens; in Italian about 140-160, because tokenizers were built mostly on English text. Other languages cost slightly more.
• You pay per token, both reading (input) and writing (output). A model like GPT-6.1 Sol costs $2 per million tokens read and $10 per million written.
• Numbers are split oddly: “2026” may be one or two tokens. That’s one reason LLMs get arithmetic wrong unless they use a calculator (today they do: they run code).

How it grasps meaning: embeddings and attention

Each token is turned into an embedding: a list of thousands of numbers that represents its meaning as a point in space. Words with similar meanings end up close together: “king” and “queen”, “Rome” and “Milan”, “invoice” and “receipt”.

But a word alone is ambiguous. Is a “bank” a lender or the side of a river? Here comes the idea that changed everything in 2017, introduced by eight Google researchers in Attention Is All You Need: the Transformer and its attention mechanism.

With attention, every token “looks at” all the other tokens in the text and decides how relevant each is. In “I deposited my salary at the bank”, the token “bank” pays a lot of attention to “deposited” and “salary”, and its meaning shifts towards the lender. This happens across dozens of stacked layers: the first ones capture grammar, the last ones concepts, intentions, tone.

The three stages of training

A model like Claude Opus 5.5 or GPT-6 isn’t born able to converse. It goes through three very different stages.

1. Pre-training: reading almost everything

The model reads a huge amount of text (web pages, books, scientific papers, code) and trains on the usual task: predicting the next token. Each mistake nudges its parameters, the numbers that govern its behaviour. Frontier models have from hundreds of billions to several trillion.

This stage needs tens of thousands of GPUs for months and costs hundreds of millions of dollars. The result is a “base model” that knows a lot but can’t answer: if you write “How do I register as a freelancer?”, it might continue with “How do I deregister? How do I change tax regime?”, because online that question often appears in a list of questions.

2. Instruction tuning: learning to answer

The base model is trained on hundreds of thousands of examples of good conversations written by people: question and ideal answer. It learns the dialogue format, to follow instructions, to sound like an assistant.

3. Reinforcement: learning what is better

Finally the model learns from a quality signal. In the classic form (RLHF, reinforcement learning from human feedback) people compare two answers and choose the better one; the model learns to produce more answers of the preferred kind. Since 2024, reinforcement on verifiable problems (maths, code, tasks with a checkable answer) is used more and more: the model tries, is rewarded when right, and learns reasoning strategies. That’s how models that “think” before answering were born.

What’s inside: compression, not an archive

A common mistake is to picture the model as an archive of texts from which it “retrieves” answers. It isn’t. The parameters are a statistical compression of what it read: regularities, concepts, relationships, styles. The model doesn’t have Leonardo’s Wikipedia page: it has a distributed representation of “things people say about Leonardo”.

Practical consequences:

• Well-known facts are reliable, rare ones aren’t. It won’t get Dante’s birth year wrong; it may invent last quarter’s court ruling, confidently.
• It has a knowledge cut-off. It knows little or nothing after its training data ends, unless it uses web search. That’s why Lumen’s chatbot checks Artificial Analysis, Arena and TestingCatalog in real time for current-events questions.
• It doesn’t know what it doesn’t know. There’s no reliable “certainty meter”: the same confident tone accompanies right and wrong answers.

Why writing costs more than reading

When you send a message, the model works in two phases:

1. Reading (prefill). It processes all the input text in parallel, in one go. Fast and efficient.
2. Writing (decode). It generates the answer one token at a time: for each token it has to run through the whole model. Slow, and it ties up the GPU much longer.

That’s why output costs 4 to 5 times input in price lists (GPT-6.1 Sol: $2 vs $10; Claude Opus 5.5: $4 vs $20). And why models that reason at length, generating thousands of “thinking” tokens before the answer, cost more and are slower.

On Artificial Analysis, top models typically write between 45 and 240 tokens per second: Claude Haiku 5.5 about 240, GPT-6 Astra about 44.

What an LLM is not

• Not a search engine. Without web search it consults nothing: it remembers, in the statistical sense above.
• Not a calculator. Modern models know when to use a tool, though: for maths they write and run code. Ask for it explicitly when numbers matter.
• It doesn’t remember other chats unless a memory feature is on, which actually inserts saved notes into the new conversation’s context.
• Not conscious. The philosophical debate is open and interesting, but for daily use the useful thing to remember is different: it produces plausible text, which is often also true. “Often” isn’t “always”.

What it means for you, in practice

1. Give context. Attention works on what you write: more relevant information, more relevant answers.
2. Check rare facts and numbers. The more specific, recent or niche the information, the more it needs checking.
3. Ask for tools when needed. “Use web search and cite sources”, “compute by running code”.
4. Repeat important questions. If two answers to the same question diverge, the model isn’t sure.
5. Pick the model for the job. A fast, cheap model is enough for an email; a complex analysis needs one that reasons. See the guide on choosing a model and plan.

Sources

• Vaswani et al., "Attention Is All You Need" (2017)
• 3Blue1Brown — Large Language Models explained briefly
• Andrej Karpathy — Deep Dive into LLMs like ChatGPT
• Tradeoff Layer — Why Output Tokens Cost 5x More
• Artificial Analysis — LLM Leaderboard

Informational content, current as of the date shown. Prices, models and rules change often: always check official sources. Written with the support of AI tools and reviewed by the author.