An LLM does one thing: given some text, it predicts what text should come next. Everything else, answering questions, writing code, following instructions, grows out of doing that well at huge scale.
Tokens
Models do not read letters or whole words. A tokenizer splits text into tokens, which can be a word, part of a word or punctuation. "Resume" might be one token and a rare name might be several. Prices, limits and speed are all measured in tokens.
Next-token prediction
The model reads your tokens and produces a probability for every token in its vocabulary. It picks one, adds it to the text, and repeats. The answer is built one token at a time.
Sampling: temperature, top-k and top-p
- Temperature controls randomness. Low values make the model choose the most likely token (more consistent). High values allow more variety.
- Top-k limits the choice to the k most likely tokens.
- Top-p limits it to the smallest group of tokens whose probabilities add up to p.
Use low settings for extraction, classification and anything that must be repeatable. Use higher settings for brainstorming.
Context window
The context window is how much text (input plus output) the model can consider at once. Anything outside it is not seen. Long documents may need to be shortened or searched instead of pasted in.
Messages and roles
Chat models receive a list of messages with roles: a system message sets the rules and persona, user messages carry the request, and assistant messages hold earlier replies. Chat templates format these for the model.
Hallucinations and the knowledge cutoff
A model produces text that sounds right, not text that is checked against a database. When it lacks the answer it can still produce a confident, false one. This is a hallucination. Models also stop learning at a knowledge cutoff date, so recent events need web search or your own data.
Reasoning models
Some models spend extra tokens "thinking" before they answer. That helps on multi-step problems, at the cost of time and tokens.