LLM fundamentals

Tokenization

Tokenization is the process of splitting text into tokens, the units the model works with: word fragments, whole words or punctuation marks. In English, a token is on average about three quarters of a word; in languages such as Spanish or Catalan, each word usually takes up more tokens.

It matters for three reasons: providers charge per token (input and output, often at different prices), model limits are measured in tokens, and the model "sees" tokens, not letters. That is why it struggles to count characters or to handle rare words letter by letter.

Embeddings

An embedding is a vector of numbers (often hundreds or thousands of dimensions) that represents the meaning of a text. Texts with similar meanings have nearby vectors, even if they share no words: "I want to send the product back" and "how do I make a return?" end up close together.

They are the basis of semantic search, classification and RAG (Module 3). They are generated with dedicated embedding models, which are smaller and cheaper than chat models. One important rule: to compare vectors, they must all have been generated with the same model and the same dimensions.

Context window

The context window is the maximum number of tokens the model can take into account in a single call, adding up the instructions, the conversation history, the documents you pass it and the response it generates.

A large window doesn't mean you have to fill it. More context means more cost and more latency, and models tend to pay less attention to information buried in the middle of a long context. Deciding what goes in and what stays out is precisely context engineering (Module 2).

Temperature, top-p and top-k

These are parameters that control how the model picks the next token among the possible candidates. Temperature adjusts randomness: low (close to 0) makes responses more predictable and repeatable; high makes them more varied and creative. Top-k limits the candidates to the k most likely ones, and top-p to those that add up to a cumulative probability p.

In practice, tuning the temperature is enough: low for extracting data, classifying or following a format, and higher for writing or generating ideas. Some models, especially reasoning models, ignore or restrict these parameters.

System prompt and user prompt

The system prompt defines the model's role and rules for the whole conversation ("you are the support assistant for a shop; always answer in a friendly tone and never promise a refund"). The user prompt is the specific request of each turn: the ticket to summarise or the customer's question.

Separating them has practical consequences: the system prompt is stable and can be versioned and tested, while the user prompt carries variable data, often written by third parties. This separation is also the first line of defence against prompt injection (Module 5).

Zero-shot, few-shot and chain-of-thought

These are three ways of asking for a task. Zero-shot: just the instructions, with no examples. Few-shot: the instructions plus a few examples of input and expected output, so the model imitates the pattern. Chain-of-thought: asking the model to reason step by step before giving its final answer.

Few-shot works very well for fixing a format or a tone, but examples also "contaminate": the model tends to copy details you didn't want. Chain-of-thought improves multi-step problems at the cost of more tokens and more time; today's reasoning models already do it internally (Module 11).

Grounding

Grounding means anchoring the model's response in specific information you provide (documents, database records, the text of a ticket) instead of letting it answer only with what it learned during training.

It is the main way to get correct, up-to-date answers about your domain: the model doesn't know your return policy until you give it to it. RAG (Module 3) is, in fact, a technique for grounding automatically.

Hallucinations

A hallucination is a response that sounds convincing but is false: a made-up figure, a reference that doesn't exist or a company policy nobody ever wrote. It is not an occasional bug but a consequence of how models work: they generate the most plausible text, which is not always the true one.

They can't be eliminated completely, but they can be reduced and contained: grounding with reliable sources, explicit instructions to say "I don't know", structured output that can be validated in code, and human review before the response has any effect. The design should assume the model will be wrong from time to time.