Evaluation and observability

Evals

An eval (evaluation) is a set of test cases with criteria for measuring whether the AI system responds well: representative inputs and, for each one, the expected answer or the conditions it must meet. It is the equivalent of automated tests, adapted to outputs that are not deterministic.

Without evals, every change of prompt, model or parameters is a gamble: what improves one case may break ten. With a set of a few dozen real cases (including the ones that have already failed at some point), you can compare versions with numbers before deploying them. The best cases come from production: every error you spot becomes a new case.

LLM-as-a-judge

LLM-as-a-judge means using one model to evaluate the responses of another (or of itself) against a set of criteria: whether the answer is correct, whether it cites its source, whether the tone is appropriate or whether it follows company policy. It is useful when the response is free text and can't be compared with an exact value.

It is practical but has known biases: it tends to prefer longer answers, answers from its own model and the first option when comparing two. It works better with concrete criteria and simple scores (yes or no, 1 to 3) than with marks from 1 to 10, and it should be calibrated against human reviews from time to time.

Tracing and logging

Tracing records every step of an AI request as a trace: the prompt sent, the response, the tool calls, the retrieved documents, the tokens and the time of each step. In a system with RAG and agents, a single answer can involve ten calls, and without a trace it is impossible to know where it went wrong.

There are dedicated tools (Langfuse, LangSmith, Arize Phoenix or the OpenTelemetry standard), but you can also start with a database table. What matters is being able to answer, days later, "why did the assistant tell this customer that?". Watch out for personal data: traces are data too and must be protected.

Latency

Latency is the time the system takes to respond. For LLMs it is measured mainly with two metrics: time to first token (when the user starts to see something) and total generation time, which grows with the length of the response.

It depends on the model (large and reasoning models are slower), on the size of the context and on how many calls you chain. The main levers are choosing a smaller model for simple tasks, reducing the context, running independent calls in parallel and using streaming (Module 7) so the wait is less noticeable.

Cost per token

Providers charge per input token and per output token, usually with output several times more expensive. The cost of a feature is therefore tokens per call multiplied by calls per use and by the number of uses, and each of these factors can grow without anyone noticing.

It should be measured per feature, not just as a monthly total, so you know what each thing costs and whether it is worth it. The levers are cheaper models for simple tasks, shorter contexts, prompt caching (Module 7), batch processing when there is no rush and avoiding unnecessary calls with upfront filters.

Throughput

Throughput is the amount of work the system can process per unit of time: requests per minute or tokens per minute. In AI, the limit is often not your server but the rate limits the provider sets on your account.

It matters when you process volume: classifying thousands of tickets, indexing a knowledge base or handling traffic spikes. The tools for managing it are queues with controlled concurrency, batch processing, spreading the load across models or providers and, when necessary, asking the provider for higher limits.