What Changed Recently
Mar 18, 2025
vLLM 0.8.0: the V1 engine is on by default.
Jun 5, 2025
Gateway API Inference Extension announced for model-aware routing.
Jun 11, 2025
MLflow 3.0: tracing and LoggedModel for GenAI.
Aug 2, 2025
EU AI Act: rules for general-purpose AI models apply.
Aug 2025
Kubernetes 1.34: Dynamic Resource Allocation is GA.
Dec 9, 2025
Model Context Protocol moves to the Linux Foundation's Agentic AI Foundation.
26. Why is LLM serving different from classic model serving?
Because generation is sequential and memory-bound. Each output token needs a pass through the model, and the attention key-value (KV) cache for every active request grows with its length.
GPU memory, not compute, usually limits how many requests you can serve at once.
The vLLM paper addressed this with PagedAttention, which stores the KV cache in fixed-size blocks like virtual memory pages, and reports 2 to 4 times the throughput of earlier systems at the same latency. vLLM 0.8.0 then made its rewritten V1 engine the default.
Candidates should also know speculative decoding, where a small draft model proposes tokens the large model verifies, which its authors measured at 2 to 3 times faster with identical outputs.
27. TorchServe is no longer maintained. What do you serve PyTorch models with now?
The TorchServe repository is archived, and its README says there are no planned updates, bug fixes or security patches. Its last release was v0.12.0 in September 2024.
An answer that still proposes TorchServe for a new system is out of date.
Current choices depend on the model. For LLMs, vLLM or another server with continuous batching.
For general models, NVIDIA Triton Inference Server (which serves PyTorch, ONNX and TensorRT models with dynamic batching), Ray Serve, or KServe on Kubernetes as the control layer.
A simple model can also run in a plain FastAPI service behind an autoscaler. The good answer names the trade-off, not a favorite tool.
28. How do Kubernetes changes affect GPU scheduling and inference routing?
Two changes matter. Dynamic Resource Allocation graduated to GA in Kubernetes v1.34, giving a richer way than the old device plugin count to request and share GPUs and other accelerators.
And the Gateway API Inference Extension adds an InferencePool resource and model-aware routing, so a gateway can send each request to the model server replica with the most free capacity rather than using round robin.
Round robin works poorly for LLMs, because one replica can be stuck on long generations while another is idle. Candidates who mention load balancing on queue length or KV cache use show real serving experience.
29. How do you evaluate and monitor an LLM application in production?
Trace every request end to end (prompt, retrieved context, tool calls, output, latency and token cost), keep a versioned evaluation set, and score outputs with a mix of automated checks, LLM-as-judge graders calibrated against human labels, and human review of samples.
Run the evaluation set on every prompt or model change, the same way unit tests run on code.
Tooling caught up in 2025. MLflow 3, released in June 2025, added tracing, a LoggedModel entity and links between traces, prompts and evaluation results.
The interview point is the method, though: without a fixed evaluation set, "the new prompt seems better" is not evidence.
30. Design an AI agent that can take actions in company systems.
Treat the LLM as an untrusted planner and put the controls in the system around it.
Expose each capability as a narrow tool with a schema, authenticate as the end user rather than a shared service account, require confirmation for actions that write or spend money, and log every tool call.
Limit steps and cost per task so a loop cannot run away.
The Model Context Protocol has become a common way to connect agents to tools and data, and Anthropic donated it to the Linux Foundation's Agentic AI Foundation on December 9, 2025.
Strong candidates also raise prompt injection: text in a retrieved document or email must never be able to trigger a tool call on its own.
For teams that build or ship general-purpose models in the EU, the AI Act timeline adds documentation duties from August 2, 2025.