MLOps and LLMOps interview questions, answered the way you’d say them.

Time to first token (TTFT) is the wait before output starts: queueing, prefill and network, so it grows with prompt length and server load. Inter-token latency (ITL) is the average gap between later tokens, set by decode speed. End-to-end latency is roughly TTFT plus ITL for each further token, so 0.5 seconds plus 20 milliseconds × 299 gives about 6.5 seconds for a 300-token answer.

With streaming, users feel TTFT most; for batch jobs, report throughput in tokens per second instead.

NVIDIA NIM: LLM inference benchmarking metricsKwon et al., Efficient Memory Management for Large Language Model Serving with PagedAttention (2023)

Practise it with Prism

Cost per request is input tokens times the input price plus output tokens times the output price, and output usually costs more: on Anthropic’s November 2026 price list, every model charges five times as much per output token. So 3,000 input and 500 output tokens cost the same as 5,500 input tokens, with output making up 45% of the bill from 14% of the tokens. Multiply by requests per day and days, and allow for retries.

Tool definitions, system prompts and chat history are input tokens too, and they are sent again with every call.

Claude API docs: Pricing (read 1 Nov 2026)Claude API docs: Tool use with Claude

Practise it with Prism

Match the metric to the task. BLEU and ROUGE count n-gram overlap with reference texts: cheap and repeatable, but they reward matching the reference’s wording and can miss a correct paraphrase. BERTScore matches tokens by contextual embeddings and correlates better with human judgments. For open-ended answers, an LLM judge with a rubric scales best once checked against human labels, and anything verifiable, such as a number or a JSON field, should be checked by code.

BLEU was designed for machine translation and ROUGE for summaries, so say why either fits before using it on chatbot answers.

Papineni et al., BLEU: a Method for Automatic Evaluation of Machine Translation (ACL 2002)Lin, ROUGE: A Package for Automatic Evaluation of Summaries (2004)Zhang et al., BERTScore: Evaluating Text Generation with BERT (2019)

Practise it with Prism

The provider stores the processed prefix of a prompt and reuses it when a later request starts with exactly the same content, cutting cost and time to first token. So put stable content first (tools, system instructions, documents, examples) and the changing part, such as the user’s question, last; any change early on invalidates everything after it. Anthropic, for example, bills cache reads at a tenth of the input price or less and keeps entries for five minutes by default, refreshed on each use.

Don’t confuse it with semantic caching, which reuses whole answers for similar questions and can return a wrong answer to a subtly different one.

Claude API docs: Prompt cachingvLLM documentation: Automatic prefix caching

Practise it with Prism

Guardrails are checks around the model, not instructions inside it. Input checks catch off-topic requests, personal data and likely injections; output checks validate the format, screen for unsafe or leaked content and test grounding; action checks gate tools behind permissions and approval. They can be rules, classifiers such as Llama Guard, which labels both prompts and responses against a risk taxonomy, or another model. Treat model output as untrusted input to anything downstream.

Each check adds latency and false positives, so measure both and run independent checks in parallel.

Inan et al., Llama Guard: LLM-based Input-Output Safeguard for Human-AI Conversations (2023)OWASP Top 10 for LLM Applications 2025: LLM05 Improper Output Handling

Practise it with Prism

Covariate shift changes the inputs, P(x), while P(y | x) holds; label shift changes how common each outcome is, P(y), such as a rise in fraud; concept drift changes P(y | x) itself, so the same inputs now mean something else. Detect the first two by comparing live data with training data, using Kolmogorov–Smirnov tests on numeric features or the model’s outputs and chi-squared tests on categories, corrected for multiple tests. Concept drift needs fresh labels, so watch live accuracy; respond by retraining on recent data, reweighting or rolling back.

In a large comparison, Rabanser and colleagues found the best detector was a two-sample test on a pretrained classifier’s outputs rather than on the raw features.

Rabanser, Günnemann and Lipton, Failing Loudly: An Empirical Study of Methods for Detecting Dataset Shift (2018)Lipton, Wang and Smola, Detecting and Correcting for Label Shift with Black Box Predictors (2018)

Practise it with Prism

A semantic cache stores past questions as embeddings, with their answers. When a new question lands within a similarity threshold of a stored one, it returns the stored answer without calling the model, saving cost and latency, as GPTCache does. The risk is a wrong hit: “cancel my order” and “don’t cancel my order” can embed close together. So tune the threshold on labelled pairs, keep personal answers out of shared caches, expire entries when facts change, and track wrong hits as well as the hit rate.

Contrast it with prompt caching, which reuses the processed prefix of an identical prompt, so the model still writes a fresh answer every time.

GPTCache: READMEClaude API docs: Prompt caching

Practise it with Prism

Agree the bar before training: the business outcome, an offline metric that tracks it, and the baseline to beat, usually the current system or a simple heuristic. Then test the candidate on recent held-out data and on important slices, such as new users or smaller cities, and check latency, cost and that serving gives the same scores as training. Finally, run an A/B test with guardrail metrics; ship if the primary metric improves and nothing important gets worse, and keep a way to roll back.

Google’s Rules of ML puts it bluntly: the only easy launch decisions are when all metrics get better, or at least don’t get worse.

Google for Developers: Rules of Machine Learning

Practise it with Prism

Log the model with MLflow, register it, and point an alias such as champion at the version to serve. A FastAPI service loads that model once at start-up and validates every request against a Pydantic schema, so bad input gets a 422 error with details instead of a crash. Package it in a container, set timeouts on downstream calls and retry only safe ones, and monitor latency, errors and the distribution of inputs and predictions. Moving the alias rolls a new version forward, or back.

import fastapi, mlflow.pyfunc, pandas as pd
from pydantic import BaseModel
model = mlflow.pyfunc.load_model("models:/churn@champion")   # loaded once, by alias
app = fastapi.FastAPI()
class Customer(BaseModel):                                     # a wrong type gets a 422
    tenure_months: int
    monthly_charge: float
@app.post("/predict")
def predict(c: Customer):
    return {"prediction": model.predict(pd.DataFrame([c.model_dump()])).tolist()[0]}

Load the model at start-up, not inside the request handler, or every call pays the loading time.

MLflow: Model Registry (aliases)FastAPI: Handling errors (request validation)

Practise it with Prism

Each run should record what’s needed to reproduce and compare it: code version, data version, parameters, metrics over time, environment and artifacts such as model files and plots, plus tags for search. Store metadata in a database and large artifacts in object storage, the split MLflow makes between its backend store and artifact store. Provide a light logging SDK, an API and a UI to filter, sort and compare runs, and link registered models back to the runs that produced them. Add access control, quotas and retention rules.

Log asynchronously and in batches, so tracking never slows training down.

MLflow: TrackingMLflow: Model Registry

Practise it with Prism

Have people label a sample and compare the judge with Cohen’s kappa, not raw agreement: if judge and humans each pass 70% of answers, agreeing on 8 in 10 gives a kappa of only 0.52, since 58% agreement is expected by chance. Then probe the biases the MT-Bench study found: swap answer order to catch position bias, compare long and short answers for verbosity bias, and don’t let a model grade its own outputs.

MT-Bench found strong judges agreed with humans over 80% of the time, as often as humans agree with each other, but that was their setup; measure yours, and re-check when the judge’s prompt or model changes.

scikit-learn: cohen_kappa_scoreZheng et al., Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena (2023)

Practise it with Prism

Track operations (latency percentiles, errors, timeouts, tokens and cost per request) and quality (user feedback, escalations, refusal rate and groundedness on a sample). Drift has three sources: users ask new things, the documents behind retrieval change or go stale, and the model itself changes. Chen and colleagues found GPT-4’s accuracy at telling prime from composite numbers fell from 84% to 51% between its March and June 2023 versions, so pin model versions and re-run your evaluation before switching.

To spot new topics, cluster the embeddings of incoming questions each week and look for clusters your test set doesn’t cover.

Chen, Zaharia and Zou, How is ChatGPT's behavior changing over time? (2023)

Practise it with Prism

A small draft model proposes several tokens, and the large model checks them all in one parallel forward pass, keeping the accepted run and resampling at the first rejection. The acceptance rule is designed so that tokens follow the large model’s own distribution, so outputs are the same as decoding with the large model alone, with no retraining. Leviathan and colleagues reported 2 to 3 times faster T5-XXL inference; the gain depends on how often the draft is accepted.

It targets decode, where the GPU waits on memory: checking several tokens per pass puts otherwise idle compute to work.

Leviathan, Kalman and Matias, Fast Inference from Transformers via Speculative Decoding (2022)Kwon et al., Efficient Memory Management for Large Language Model Serving with PagedAttention (2023)

Practise it with Prism

I’d measure before changing anything: cost per request, time to first token and tokens per second. Then I’d serve it on an engine that does continuous batching and paged KV caching, vLLM for example, quantise only if quality holds up, limit output length, cache prefixes that repeat, and send the easy requests to a smaller model.

Product-company rounds go deep on serving metrics, so open with what you’d measure.

Kwon et al. 2023: Efficient Memory Management for Large Language Model Serving with PagedAttention

The full answer, in the guidePractise it with Prism

First I’d measure requests per user and tokens per request. Then I’d trim the system prompt and the retrieved context, cache answers to questions that repeat, let a smaller model take the simple requests, run offline work in batches, and cap output length and per-user spend. Runaway usage is also a security issue: OWASP lists it as its own risk, unbounded consumption.

OWASP Top 10 for LLM Applications 2025

The full answer, in the guidePractise it with Prism

Prompt injection is input that hijacks the model away from your instructions, either typed by the user, which is direct, or hidden inside a web page, an email or a retrieved document, which is indirect. It’s number one on OWASP’s 2025 list. Since no prompt can fully stop it, I’d contain the damage: give tools the least privilege they need, require human approval before actions, keep instructions apart from data, and check outputs before anything runs.

Don’t promise that a better system prompt will stop it: no prompt fully prevents injection, so talk about limiting the damage.

OWASP Top 10 for LLM Applications 2025

The full answer, in the guidePractise it with Prism

All ten on the 2025 list: prompt injection, sensitive information disclosure, supply chain, data and model poisoning, improper output handling, excessive agency, system prompt leakage, vector and embedding weaknesses, misinformation, and unbounded consumption. Which ones I’d stress depends on the system. For a RAG assistant, vector and embedding weaknesses plus misinformation; for an agent, excessive agency.

Excessive agency, no. 6 on the list, is the risk that agents add.

OWASP Top 10 for LLM Applications 2025

The full answer, in the guidePractise it with Prism

I collect as little as possible, then detect and mask personal data, such as names, phone numbers and account numbers, before text is indexed, logged or sent to an external model. Presidio, which is open source, provides an analyzer and an anonymizer, but its own docs say it can’t guarantee catching everything. I’d also set retention limits and access controls, and have the legal team review it against India’s Digital Personal Data Protection Act.

Presidio: data protection and de-identification

The full answer, in the guidePractise it with Prism

I’d use it for scale, but with checks around it. The MT-Bench study found that strong judge models matched human preferences in over 80% of cases, roughly as often as humans agree with one another, but it also identified biases: position, verbosity and self-enhancement. So I swap the order of the answers, give the judge a rubric plus reference answers, and compare a sample of its verdicts with human labels.

Zheng et al. 2023: Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena

The full answer, in the guidePractise it with Prism

In data parallelism, every GPU holds a full copy of the model and processes its own slice of the batch, and an all-reduce averages the gradients before each update, as PyTorch’s DistributedDataParallel does; when the model won’t fit, FSDP or ZeRO shard parameters, gradients and optimiser states across the GPUs. Tensor parallelism splits each layer’s weight matrices across GPUs, as Megatron-LM does. Pipeline parallelism puts groups of layers on different GPUs and streams micro-batches through them, as GPipe does. Large models combine all three.

GPipe found the pipeline’s idle “bubble” negligible once there were at least four times as many micro-batches as pipeline stages.

PyTorch: Distributed Data ParallelPyTorch: FullyShardedDataParallelShoeybi et al., Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelism (2019)Huang et al., GPipe: Efficient Training of Giant Neural Networks using Pipeline Parallelism (2018)

Practise it with Prism

Production traffic becomes the new data. Log inputs, outputs and signals such as thumbs-downs, edits, retries and escalations; read a sample of failures, group them by cause, and fix the biggest group first, whether it’s the prompt, retrieval or a tool. Add each fixed failure to the golden set, so every later change is tested against all of them, and fill gaps with edge cases written by a model and checked by people. To grade at scale, use an LLM judge you’ve checked against human grades.

Expect your criteria to change as you read outputs: Shankar and colleagues call this criteria drift, since grading outputs is how people work out what good means.

Shankar et al., Who Validates the Validators? Aligning LLM-Assisted Evaluation of LLM Outputs with Human Preferences (2024)Claude API docs: Define success criteria and build evaluations

Practise it with Prism

Detect it with health checks, heartbeats, timeouts on collective operations, GPU error counters, and step-time monitoring to catch slow stragglers. Synchronous training can’t carry on without a rank, so stop the job, cordon or replace the node, and restart every rank from the latest checkpoint; torchrun can restart workers automatically, and its elastic mode can continue with a different number of nodes. Checkpoint often enough that little work is lost. At scale this is routine: Meta logged 466 job interruptions in a 54-day Llama 3 pre-training snapshot.

The Llama 3 team still kept effective training time above 90%, using tools such as PyTorch’s NCCL flight recorder to find hangs and stragglers quickly.

Llama Team, The Llama 3 Herd of Models (2024), section 3.3.4PyTorch: torchrun (Elastic Launch)

Practise it with Prism

Each of n ranks starts with one chunk and must end with all n. In a ring, every rank sends to its right-hand neighbour and receives from its left; at each step it forwards the chunk it received last, so after n − 1 steps every chunk has reached every rank. Each rank receives exactly the n − 1 chunks it lacks, the least possible, and every link is busy at once. Post each send and receive together, batched, or ranks can block on each other and deadlock.

import torch, torch.distributed as dist
def ring_all_gather(chunk):                        # this rank's piece; returns all, in rank order
    n, r = dist.get_world_size(), dist.get_rank()
    out = [torch.empty_like(chunk) for _ in range(n)]
    out[r].copy_(chunk)
    for s in range(n - 1):                         # n - 1 steps around the ring
        send, recv = (r - s) % n, (r - s - 1) % n
        ops = [dist.P2POp(dist.isend, out[send], (r + 1) % n), dist.P2POp(dist.irecv, out[recv], (r - 1) % n)]
        for req in dist.batch_isend_irecv(ops): req.wait()
    return torch.cat(out)

In real code you’d call dist.all_gather; the exercise checks that you understand the communication pattern behind it.

PyTorch: torch.distributed (P2POp, batch_isend_irecv, all_gather)

Practise it with Prism