Quick answer: A large language model (LLM) works by predicting the next token. It splits your text into tokens, turns each token into a list of numbers (an embedding), passes those numbers through many transformer layers where an attention mechanism lets each token use information from the tokens before it, and then scores every token in its vocabulary to pick the next one. It adds that token to the text and repeats the loop until the answer is complete. Its built-in knowledge comes from training on huge amounts of text with this same prediction task, followed by extra tuning that teaches it to follow instructions.
This guide shows how large language models work by following one prompt through a model. Along the way it explains why chatbots stream their answers, why they sometimes invent facts with full confidence, and why long prompts are slower. The prompt is:
The capital of Pakistan is
We will go through it in six steps, then look at how the model learned to do this and where its limits come from. To keep things clear, we look at the raw model. A chat app wraps your message in a hidden template with a system prompt and special tokens, so the model actually sees more text than you typed, but the mechanics are the same.
Table of Contents
What is a large language model?
A large language model is a neural network trained to predict the next piece of text given the text that came before it. “Large” refers to the number of parameters, the adjustable numbers the model learns during training, which runs into the billions for modern models. GPT, Claude, Gemini and Llama are all families of LLMs.
Almost every current LLM is built on the transformer architecture, introduced by Google researchers in the 2017 paper Attention Is All You Need. Chat models use the “decoder-only” version of it, which only reads text left to right and is built to generate text one token at a time. It is a stack of layers with the same structure, each one refining the model’s picture of the text.
Three terms come up again and again, so here they are in one place:
| Term | What it means |
|---|---|
| Token | A chunk of text the model works with: a word, part of a word, a number or a punctuation mark |
| Parameters | The learned numbers (weights) inside the network; they store what the model picked up in training |
| Context window | The maximum number of tokens the model can handle in one request, counting your prompt and its reply |
How large language models work, in six steps
Here is the whole journey at a glance. Each step is explained below.

Step 1: Text becomes tokens
The model never sees letters or words directly. A tokenizer first cuts the text into tokens from a fixed vocabulary and replaces each one with an ID number. Common words are usually a single token; rare words get split into pieces.
You can see this yourself with OpenAI’s open-source tiktoken library:
# pip install tiktoken
import tiktoken
enc = tiktoken.get_encoding("o200k_base") # tokenizer used by recent OpenAI models
text = "The capital of Pakistan is"
ids = enc.encode(text)
print(ids)
print([enc.decode([i]) for i in ids])
print(len(text), "characters ->", len(ids), "tokens")
Output:
[976, 9029, 328, 27950, 382]
['The', ' capital', ' of', ' Pakistan', ' is']
26 characters -> 5 tokens
Each space is attached to the start of the next word. That matters: ” capital” with a leading space is token 9029, while “capital” without one is token 128780. Longer or rarer words get split. ” tokenization”, for example, becomes two tokens: ” token” and “ization”. ” Islamabad” happens to be a single token in this vocabulary.
For English text, OpenAI’s token guide gives a rule of thumb of about 4 characters, or three-quarters of a word, per token. Other languages often need more tokens for the same meaning, and each model family has its own tokenizer, so the same sentence can cost a different number of tokens on different models.
Why it matters for you: API prices and context limits are counted in tokens, not words. A 1,500-word article is roughly 2,000 tokens.
Step 2: Tokens become embeddings
An ID like 976 means nothing by itself, so the model looks up each ID in a large table and pulls out an embedding: a long list of numbers (thousands of them in big models) that represents the token. During training, tokens used in similar ways end up with similar embeddings, so “Lahore” and “Karachi” sit closer together than “Lahore” and “banana”.
The model also needs word order, because “Ali called Sara” and “Sara called Ali” contain the same tokens but mean different things. The original transformer added a position signal to each embedding. Most modern LLMs add position inside the attention step instead, using a method called rotary position embeddings (RoPE). Either way, the model knows where each token sits.
At this point our prompt is five tokens, each turned into a vector of numbers.
Step 3: Attention lets tokens share information
In each layer, self-attention lets every token gather information from itself and the tokens before it.
A simplified way to picture it is that each token produces three vectors:
- A query, which describes the kind of information this token needs.
- A key, which describes the kind of information this token can provide.
- A value, which is the information it actually hands over.
The model multiplies one token’s query with each earlier token’s key (a dot product) to get a relevance score. A softmax turns those scores into weights that add up to 1, and the token’s new vector becomes a weighted mix of the values. In a simplified picture, the token ” is” would draw heavily on ” capital” and ” Pakistan”, because those two decide what should come next. Real models spread this work across many heads and layers in ways that are harder to read.
Three details make this work at scale:
- Causal masking. A token can only look backwards. When working on ” is”, the model cannot see anything that comes after it.
- Multiple heads. Each layer runs many attention “heads” in parallel, and each head can learn a different kind of relationship, such as grammar or which noun a pronoun refers to.
- Many layers. After attention, each token passes through a feed-forward network, a set of dense layers where most of the model’s parameters live. The attention-plus-feed-forward block repeats dozens of times, and the representations become more abstract as they move up the stack.
One cost to remember: in standard attention, each token is compared with every earlier token, so doubling the prompt roughly quadruples the attention work, while the rest of the network grows only in step with length. At very long lengths attention dominates, which is one reason long-context requests are slower. What you pay through an API, though, is simply priced per token.
Step 4: The model scores every possible next token
After the last layer, the model takes the final vector of the last token (” is”) and converts it into one score for every token in its vocabulary, often well over 100,000 scores. Using only the last token works because, thanks to attention, its vector already carries information from the whole prompt. (During training, every position predicts its own next token at once.)
A softmax function then turns these raw scores (called logits) into probabilities that add up to 1. For our prompt, the result looks something like this:
| Candidate next token | Probability |
|---|---|
| ” Islamabad” | 0.91 |
| ” a” | 0.03 |
| ” located” | 0.02 |
| ” the” | 0.01 |
| everything else | 0.03 |
These numbers are for illustration; each model gives its own values, and most chat APIs hide them by default. Some APIs can return the top candidates with their log-probabilities if you ask for them.
The model has no lookup table of facts. ” Islamabad” scores highest because training made it the likeliest continuation of this pattern.
Step 5: One token is chosen
Now a decoding strategy picks a single token from that distribution:
- Greedy decoding always takes the top token. It is predictable, but it can sound repetitive in long answers.
- Temperature reshapes the probabilities before sampling. At 1, the model’s probabilities are used as they are. Values below 1 sharpen them, so the top choice wins more often; values above 1 flatten them, so less likely tokens get a real chance. Low values suit facts and code, higher ones suit brainstorming.
- Top-k and top-p cut off the long tail first. Top-k keeps only the k most likely tokens (for example, 40). Top-p, also called nucleus sampling, keeps the smallest group of tokens whose probabilities add up to a set value such as 0.9. The remaining probabilities are rescaled to add up to 1, and one token is sampled.
This is why the same prompt can come back with different wording: the model samples from the probabilities. A temperature near 0 makes answers much more consistent, though not always identical, and some models don’t let you change it at all.
Step 6: Append and repeat
The chosen token, ” Islamabad”, is added to the end of the text, and the whole process runs again on “The capital of Pakistan is Islamabad” to pick the next token, perhaps “.”. This loop is called autoregressive generation. It continues until the model produces a special end-of-text token or hits a length limit.
One loop explains several things you notice when using chatbots:
- Answers stream in small chunks because they are produced one token at a time.
- The prompt is fast, the reply is slow. Your whole prompt is processed in one parallel pass. Only the reply is generated token by token, and each new token needs another pass through the network. Servers keep the keys and values already computed (the KV cache) so earlier tokens are not reprocessed. This sequential work is part of why output tokens usually cost more than input tokens.
- Early mistakes carry forward. Once a wrong token is written, every later token is predicted on top of it. The model has no built-in step that goes back and edits what it already wrote.
How an LLM learns to do this
The six steps describe a model that is already trained. Training is how the billions of parameters get their values, and for a modern chat assistant it happens in stages.
| Stage | Training data | What the model learns |
|---|---|---|
| 1. Pre-training | Trillions of tokens of web text, books, code and other sources, including licensed and synthetic data | Language, facts and patterns, all by predicting the next token |
| 2. Instruction tuning (supervised fine-tuning) | Tens of thousands to millions of example prompts with high-quality answers, written by people or generated and checked by models | To respond like an assistant instead of just continuing text |
| 3. Learning from feedback (e.g. RLHF) | People, or trained models, ranking several answers to the same prompt | Which answers are helpful, honest and safe |

Pre-training: learning from next-token prediction
In pre-training, the model reads text, predicts each next token, compares its guess with the real token, and adjusts its parameters slightly so the right token becomes more likely next time. Repeated over a vast amount of text, this simple task pushes the model to absorb grammar, facts and patterns of reasoning, because all of them help it predict text better.
No one labels this data by hand; the correct answer is simply the next token already in the text. That is why it is called self-supervised learning.
A pre-trained “base” model is a strong text completer but not a helpful assistant. Ask it a question and it may reply with more questions, because on many web pages a question is followed by another question.
Instruction tuning and human feedback
To turn a base model into an assistant, developers fine-tune it on examples of instructions paired with good answers. Then they collect preferences: people compare two or more model replies and pick the better one. A separate reward model learns to predict those choices, and the LLM is tuned with reinforcement learning to produce answers the reward model scores highly. This is reinforcement learning from human feedback, or RLHF.
OpenAI’s InstructGPT paper (2022) described this three-step recipe. On prompts submitted to OpenAI’s API, human labelers preferred answers from a 1.3-billion-parameter tuned model over the original 175-billion-parameter GPT-3, a model about 100 times larger. In that study, tuning mattered more to raters than raw size.
Newer methods change parts of this recipe. Direct preference optimization (DPO) learns from the same kind of preference pairs but skips the separate reward model and the reinforcement-learning step. Many recent “reasoning” models also get an extra round of reinforcement learning on tasks with checkable answers, such as math problems and code that must pass tests.
Why LLMs make things up
Because an LLM generates the most plausible next token rather than retrieving a stored fact, it can produce fluent, confident text that is wrong. This is called a hallucination. It happens most often when:
- the fact was rare or missing in the training data, such as a small local company or a recent event;
- the question contains a false premise and the model goes along with it;
- the answer needs exact details such as citations, URLs, statistics or dates.
The model’s built-in knowledge also stops at its training cutoff. It does not know what happened afterwards unless that information is placed in the prompt.
The most practical way to reduce this is to give the model the right information at question time instead of relying on its memory. That is the idea behind retrieval-augmented generation (RAG) and the broader practice of context engineering: you fetch trusted documents and put them in the context window, so the model writes from real text in front of it.
How the context window works
The context window covers everything the model can see in one request: system instructions, the conversation so far, any documents you pasted, and the reply being written. Frontier models now accept hundreds of thousands of tokens, and in some cases more than a million, but a larger window doesn’t mean the model uses every part of it equally well.
In the Lost in the Middle study (2023), the models tested used information placed at the beginning or end of a long input more reliably than information buried in the middle. Newer models show this effect less strongly, but the practical lessons still help:
- Keep key instructions at the start or end of a long prompt, not buried in the middle.
- With long documents, put the documents first and your question last.
- Send only the documents that matter, and summarize older turns in long chats.
The model also does not remember you between separate sessions. Any “memory” a product offers is the app saving notes and inserting them into the context window later, which is how AI agent memory across sessions is built.
Common misconceptions about LLMs
| Belief | What actually happens |
|---|---|
| “It searches the internet for answers.” | A plain LLM only uses its trained parameters and the text in its context window. Web search is a separate tool an app may add. |
| “It understands words the way we do.” | It processes numeric representations of tokens and predicts likely continuations. Whether that counts as understanding is debated; it does not check facts against the world. |
| “It learns from my chats as I talk.” | The parameters do not change during a conversation. Some providers may use chats to train future versions, depending on their settings and policies. |
| “Bigger models are always better.” | Size helps, but tuning matters too: in the InstructGPT study, raters preferred a tuned 1.3B model over the untuned 175B GPT-3. |
| “Same prompt, same answer.” | With sampling on, wording changes from run to run. A low temperature makes answers more consistent, not guaranteed identical. |
How this connects to AI agents and tools
Coding assistants and AI agents still run on this same loop. An AI agent is an LLM placed in a loop where it can call tools, read the results and decide on its next step, and the Model Context Protocol (MCP) standardizes how those tools connect. Because every step is still next-token prediction, one bad tool result or a misleading document can steer every step after it, which is why agents need step limits and human checks before risky actions.
Frequently asked questions
How do LLMs generate text?
They generate text one token at a time. At each step, the model scores every token in its vocabulary, picks one (using settings such as temperature), adds it to the text and repeats until it outputs an end token or reaches a length limit.
What is a token in an LLM?
A token is the unit of text a model reads and writes. It can be a whole word, part of a word, a number or a punctuation mark. In English, one token averages about four characters, or three-quarters of a word.
Are LLMs trained with supervised learning?
Pre-training is self-supervised: the next token in the text serves as the label, so no human labeling is needed. Later stages use supervised fine-tuning on example answers and reinforcement learning from human or AI feedback.
Why do LLMs hallucinate?
They are trained to produce plausible text, not to verify facts. When the training data lacked a fact or a prompt is ambiguous, the most likely-sounding continuation can be false. Supplying trusted documents in the prompt (RAG) reduces this, though it does not remove it completely.
What is the difference between an LLM and ChatGPT?
An LLM is the underlying model, such as GPT-5. ChatGPT is a product built around one or more LLMs, adding a chat interface, safety systems, memory features and tools such as web search and file reading.
Key points to remember
- An LLM is a next-token predictor built from a stack of transformer layers.
- Text is split into tokens, turned into embeddings, mixed by attention across many layers, and scored to choose the next token, one token at a time.
- Pre-training on huge text teaches knowledge and patterns; instruction tuning and feedback make the model a helpful assistant.
- Hallucinations, the knowledge cutoff, context-window limits and varied wording all follow from this design.
- For reliable answers, control the context: give the model the facts it needs, keep prompts focused and use a low temperature for factual work.
Sources
- Vaswani et al., Attention Is All You Need, 2017
- Ouyang et al., Training language models to follow instructions with human feedback, 2022
- Liu et al., Lost in the Middle: How Language Models Use Long Contexts, 2023
- OpenAI Help Center, What are tokens and how to count them?






