Advanced model concepts
It is not a prerequisite for any other module: it is background knowledge to understand how the models you already use work under the hood. You can read it at any time.
Reading time: 3 min
Mixture of Experts (MoE)
Mixture of Experts is a model architecture in which the network has many "experts" (sub-networks) and, for each token, a routing mechanism activates only a few of them. The model has a large number of parameters in total, but only uses a fraction of them at each step.
The result is a model with the capacity of a very large one and an inference cost closer to that of a medium-sized one. Many current models, both commercial and open, use this architecture. For people building applications it doesn't change how you use them, but it helps explain why some large models are surprisingly fast and cheap.
Context distillation
Context distillation is a training technique that makes a model internalise the behaviour it would have with a long prompt (detailed instructions, examples) so that it reproduces it without needing to receive that prompt. Responses are generated with the full context and the model is trained to give them without it.
It is a form of fine-tuning aimed at saving context: fewer tokens per call and therefore lower cost and latency. It makes sense when the same long prompt is repeated at very high volume. For most applications, prompt caching achieves similar savings without training anything.
Multimodality
A multimodal model can process (and, in some cases, generate) more than one type of data: text, images, audio, video or PDF documents. You can send it a photo of a broken product, a screenshot of an error or a scanned invoice and ask it directly about the content.
It opens up use cases that used to require dedicated systems, such as extracting data from documents, understanding screenshots or transcribing and summarising calls. Images and audio consume a lot of tokens, and accuracy on small details (the figures in a blurry document, for example) is not perfect: whatever is extracted should be validated like any other model output.
Reasoning models
Reasoning models are models trained to reason before answering: they generate an internal chain of thought (sometimes visible, sometimes not) and spend more compute on difficult problems. Many let you adjust how much they think with a reasoning token budget or an effort level.
They are clearly better at multi-step tasks (planning, maths, code, complex analysis), but they are slower and more expensive, because reasoning tokens are billed too. For simple tasks such as classifying, extracting or rephrasing, they add little. They are a good candidate for model routing (Module 7): reasoning only where it makes a difference.
Test yourself on this module
Copy this prompt and paste it into your AI (ChatGPT, Claude, Gemini…). It will give you a 20-question multiple-choice test on the module's concepts and then suggest a hands-on exercise.
Act as the examiner for Dani Pérez's "AI Engineering Guide". Examine me on the module "Advanced model concepts" (https://daniperez.pro/en/resources/ai-engineering-guide/advanced-model-concepts). Concepts covered by the exam: - Mixture of Experts (MoE) - Context distillation - Multimodality - Reasoning models Exam: 1. 20 multiple-choice questions, each with 4 options (a, b, c, d) and a single correct answer. 2. Ask about understanding and judgement (what each thing is for and when NOT to use it), not about memorising definitions. 3. Spread the position of the correct answer evenly across a, b, c and d. 4. Ask me the questions in 4 rounds of 5. Don't give any example answer (like "1a 2b 3c 4d 5a"): I already know to answer with the letters. Don't tell me whether I got them right until I have answered all 20. 5. At the end, mark them all: for each question, my answer, the correct one and a short explanation. Give me my score out of 20 and tell me which concepts I should review. Hands-on exercise (after marking): 6. Ask me what application I have or want to build, and which language and framework I work with. If I don't have one, use this: a customer support application for an online shop, with tickets, customers, orders and a knowledge base (FAQ and return policies). In that case, focus the exercise on: tickets that arrive with a screenshot of the error, to apply multimodality; if I prefer, give me a reflection exercise on when a reasoning model is worth it. 7. Suggest an exercise that applies this module's concepts to that application: goal, requirements, criteria to consider it done and common mistakes to avoid. 8. Don't solve it for me. When I bring you my solution, review it against those criteria.