Architecture and production
With the system validated and measured, it is time to make it efficient and scalable in a real production environment.
Reading time: 2 min
Streaming
Streaming means receiving the model's response token by token as it is generated, instead of waiting for it to be complete. The user starts reading after a few hundred milliseconds, even if the full response takes ten seconds.
It changes the perception of speed more than any other optimisation, and it is almost mandatory in chat interfaces. It has an implementation cost: you need a channel that supports it (Server-Sent Events or WebSockets), you need to handle errors halfway through a response, and structured output can't be validated until the end.
Prompt caching
Prompt caching is a provider feature that reuses the processing of the part of the prompt that repeats across calls, such as a long system prompt, a set of fixed documents or a conversation history. Calls that reuse that prefix are cheaper and respond faster.
To take advantage of it, the stable part must go at the start of the prompt and the variable part (the question, the specific ticket) at the end, because the cache works on exact prefixes. It shouldn't be confused with storing responses in your own cache, which only helps when exactly the same question is repeated.
Batch processing
Batch processing means sending many requests together for the provider to process asynchronously, usually within a window of hours, in exchange for a significant discount (often around 50%).
It is ideal for work that isn't urgent: classifying all historical tickets, generating overnight summaries or running a whole eval set. It is no use for anything a user is waiting for. The decision is simple: if the result isn't needed in the next few minutes, it should probably go in a batch.
Model routing
Model routing means dynamically choosing which model handles each request based on difficulty, cost or latency: a small, cheap model for simple tasks and a large one for complex ones.
In a customer support application, classifying a ticket by category doesn't need the same model as drafting the reply to a sensitive complaint. Routing can be a fixed rule per task (the most common approach and the easiest to maintain) or a dynamic decision, even one made by a small model. It also works as a plan B: if one provider fails, the request moves to another.
Test yourself on this module
Copy this prompt and paste it into your AI (ChatGPT, Claude, Gemini…). It will give you a 20-question multiple-choice test on the module's concepts and then suggest a hands-on exercise.
Act as the examiner for Dani Pérez's "AI Engineering Guide". Examine me on the module "Architecture and production" (https://daniperez.pro/en/resources/ai-engineering-guide/production-architecture). Concepts covered by the exam: - Streaming - Prompt caching - Batch processing - Model routing Exam: 1. 20 multiple-choice questions, each with 4 options (a, b, c, d) and a single correct answer. 2. Ask about understanding and judgement (what each thing is for and when NOT to use it), not about memorising definitions. 3. Spread the position of the correct answer evenly across a, b, c and d. 4. Ask me the questions in 4 rounds of 5. Don't give any example answer (like "1a 2b 3c 4d 5a"): I already know to answer with the letters. Don't tell me whether I got them right until I have answered all 20. 5. At the end, mark them all: for each question, my answer, the correct one and a short explanation. Give me my score out of 20 and tell me which concepts I should review. Hands-on exercise (after marking): 6. Ask me what application I have or want to build, and which language and framework I work with. If I don't have one, use this: a customer support application for an online shop, with tickets, customers, orders and a knowledge base (FAQ and return policies). In that case, focus the exercise on: moving heavy operations to a queue, streaming the suggested reply, making use of prompt caching and choosing the model per task. 7. Suggest an exercise that applies this module's concepts to that application: goal, requirements, criteria to consider it done and common mistakes to avoid. 8. Don't solve it for me. When I bring you my solution, review it against those criteria.