LLM and GenAI interview questions, answered the way you’d say them.

Because tokenisers learn their vocabulary from training text, and most were built mainly for English: common English words become single tokens, while Hindi words split into several pieces. Petrov and colleagues measured Hindi at 4.79 times English’s token count for the same translated sentences with GPT-4’s cl100k_base tokeniser, against 1.25 times with the multilingual XLM-RoBERTa tokeniser. More tokens mean a higher bill, slower replies and less room in the context window.

Always count with your model’s own tokeniser: newer ones narrow the gap, for example Llama 3 added about 28,000 tokens for non-English languages to the 100,000 it took from tiktoken.

Petrov et al., Language Model Tokenizers Introduce Unfairness Between Languages (2023), Appendix CSennrich, Haddow and Birch, Neural Machine Translation of Rare Words with Subword Units (2016)Llama Team, The Llama 3 Herd of Models (2024), Table 3

Try it in the token counterPractise it with Prism

Multiply the parameters by the bytes each takes: 7 billion × 2 bytes is 14 GB in 16-bit, 7 GB in 8-bit and 3.5 GB in 4-bit, plus a little for quantisation scales. That covers the weights only; the KV cache and activations come on top. LLM.int8() showed 8-bit inference can halve memory while keeping full-precision performance, and QLoRA’s 4-bit NormalFloat goes further, but measure quality on your own task after quantising.

With long contexts and big batches the KV cache can need more memory than the weights, so size both.

Dettmers et al., LLM.int8(): 8-bit Matrix Multiplication for Transformers at Scale (2022)Dettmers et al., QLoRA: Efficient Finetuning of Quantized LLMs (2023)

Practise it with Prism

Because training and evaluation reward a confident guess over an honest “I don’t know”. Kalai and colleagues argue that errors arise in pretraining from ordinary statistical pressure whenever true and false statements can’t be told apart from the data. Most benchmarks then mark answers simply right or wrong, giving no credit for abstaining, so models learn to guess, like students facing a hard exam.

Their proposed fix is to change how existing leaderboard benchmarks score answers, rather than to add separate hallucination tests.

Kalai et al., Why Language Models Hallucinate (2025)

Practise it with Prism

A jailbreak is one kind of prompt injection: input that makes the model ignore its safety rules altogether. Prompt injection is the wider class of inputs that change a model’s behaviour in unintended ways, either typed directly by a user or hidden indirectly in a web page, file or email the model reads. OWASP says it’s unclear whether fool-proof prevention exists, so mark untrusted content, limit privileges and require human approval for risky actions.

Spotlighting, which transforms untrusted text so the model can tell where it came from, cut attack success from over 50% to under 2% in its authors’ tests on GPT-family models.

OWASP Top 10 for LLM Applications 2025: LLM01 Prompt InjectionHines et al., Defending Against Indirect Prompt Injection Attacks With Spotlighting (2024)

Practise it with Prism

It’s a neural network, typically a transformer, that learned to predict the next token by training on an enormous amount of text. It writes by predicting one token, adding it to the input and going again, so a reply is just one prediction after another. Instruction tuning and preference training are what teach it to follow requests, but none of that makes it a store of facts.

The full answer, in the guidePractise it with Prism

Tokens are the units a model actually reads and writes, which can be whole words, pieces of words or single characters. Popular tokenisers like byte-pair encoding begin with characters and keep merging the most frequent pair. They matter because context limits, speed and cost are all measured in tokens, and some languages break into more tokens than others, so I’d always measure my own text.

Hugging Face LLM Course: Byte-Pair Encoding tokenization

The full answer, in the guideTry it in the token counterPractise it with Prism

Self-attention lets every token decide how much weight to give each other token when it builds its own representation. Each token gets its own query, key and value vectors; the weights are softmax(QKᵀ/√d), and the output is a weighted sum of the values. With multi-head attention, several heads work side by side, positional encodings supply word order, and the attention-only transformer comes from Vaswani and colleagues in 2017.

Be ready to write attention as a four-line NumPy function.

Vaswani et al. 2017: Attention Is All You Need

The full answer, in the guidePractise it with Prism

Encoder-only models like BERT take in the whole input together, which makes them good for classification and embeddings. Decoder-only models like GPT produce text left to right and are what chat assistants run on. Encoder–decoder models like T5 map one sequence to another, so they fit translation and summarisation.

The full answer, in the guidePractise it with Prism

All three control how the next token is sampled from the model’s probabilities. Temperature rescales those probabilities, so a low value gives more predictable text and a high one more variety. Top-k samples only from the k most likely tokens, and top-p from the smallest group whose probabilities sum to p; in Hugging Face Transformers the defaults are temperature 1.0, top-k 50 and top-p 1.0 unless the model’s config sets them.

Hugging Face Transformers: Generation

The full answer, in the guideTry it in the temperature playgroundPractise it with Prism

The context window is the maximum number of tokens a model can handle in one request, counting the prompt and the answer together. A bigger one doesn’t solve everything: longer inputs cost more and take longer, and models make the best use of information near the start or the end of a long input, and the worst use of what’s in the middle. So a handful of well-chosen chunks often does better than pasting in everything.

Liu et al.: Lost in the Middle

The full answer, in the guideTry it in the token counterPractise it with Prism

An embedding is a vector that positions a piece of text by its meaning, so texts that mean similar things end up near each other. Two embeddings are compared with cosine similarity. For text in Indian languages I’d choose a multilingual model, such as Qwen3-Embedding-0.6B, which supports over 100 languages.

Sentence Transformers: QuickstartQwen3-Embedding-0.6B model card

The full answer, in the guidePractise it with Prism

Hallucination is output that sounds fluent but isn’t supported by the input or by the facts. To reduce it, I base answers on retrieved documents and cite them, and tell the model to admit when the documents don’t cover a question. I’d also lower the temperature for factual tasks, ask for output that can be checked, and keep a human in the loop for high-stakes answers.

The full answer, in the guidePractise it with Prism

Zero-shot means the prompt carries only the instruction. Few-shot includes a few solved examples, which helps the model get the format and the edge cases right. Chain-of-thought asks it to reason through the steps before answering, which helps on multi-step problems. Whichever I use, I version prompts like code and test them on the same fixed examples every time.

The full answer, in the guidePractise it with Prism

Greedy decoding takes the most likely token at every step; it’s the default in Hugging Face Transformers, but it tends to repeat itself on long outputs. Beam search keeps several candidate sequences and returns the one with the highest overall probability, which suits tasks grounded in an input, such as translation, captioning or speech recognition. Sampling draws from the distribution, shaped by temperature, top-k and top-p, giving open-ended text its variety; Holtzman and colleagues found that likelihood-maximising decoding makes such text bland and repetitive.

In Hugging Face, num_beams above 1 turns on beam search, and do_sample=True with num_beams=1 turns on sampling.

Hugging Face Transformers: Generation strategiesHoltzman et al., The Curious Case of Neural Text Degeneration (2019)

Try it in the temperature playgroundPractise it with Prism

Stateless: the model keeps nothing between calls, and Anthropic’s Messages API, for example, expects the full conversation history in every request. So the app stores each conversation’s messages under a conversation ID and resends the relevant history with every turn. When the history outgrows the context budget, it trims or summarises older turns, and long-term memory, such as a user’s preferences, lives in a database and is retrieved into the prompt when relevant. KV and prompt caches only speed this up; they aren’t memory.

For consistent behaviour across turns, keep the system prompt fixed and versioned, and log which version each conversation used.

Claude API docs: Using the Messages API (multiple conversational turns)Anthropic Engineering, Effective context engineering for AI agents (2025)

Practise it with Prism

Special tokens are reserved IDs that mark structure rather than text: the start and end of a sequence, padding, and the role markers a chat template puts around system, user and assistant turns. Each chat model expects its own template, and Hugging Face warns that the wrong control tokens make performance drastically worse, as does duplicating special tokens. They are also a security boundary: tiktoken’s encode raises an error on user text containing a special token unless you allow it.

Let the tokeniser’s apply_chat_template method build the prompt instead of concatenating strings by hand.

Hugging Face Transformers: Chat templatesOpenAI, tiktoken source: Encoding.encode and special tokens

Practise it with Prism

When the vectors are normalised to unit length. Then the dot product equals the cosine, and the squared Euclidean distance equals 2 minus twice the cosine, so all three put neighbours in the same order. That’s why FAISS tells you to normalise vectors and use inner product for cosine search. Without normalisation, the dot product also grows with a vector’s length, so a long vector can outrank one that points closer.

FAISS reports squared L2 distances, where smaller means closer, so don’t sort them like similarities.

FAISS wiki: MetricType and distances

Practise it with Prism

To keep the softmax out of its flat regions. If query and key components are independent with mean 0 and variance 1, their dot product has variance d_k, so raw scores grow with the head size and push the softmax towards near one-hot outputs with extremely small gradients. Dividing by √d_k brings the variance back to 1. The original transformer used d_k = 64 per head, so it divided every score by 8.

The paper’s motivation: without the scaling, additive attention beat dot-product attention for larger d_k.

Vaswani et al., Attention Is All You Need (2017)

Practise it with Prism

It lets each position attend only to itself and earlier positions, so a next-token prediction never sees the future, exactly as in generation. Scores for future positions are set to minus infinity before the softmax, which gives them zero weight. Thanks to the mask, training can score every position of a sequence in one parallel pass while matching how the model generates one token at a time.

import torch
import torch.nn.functional as F
q = k = v = torch.randn(1, 4, 8)                         # batch 1, 4 tokens, 8 dimensions
scores = q @ k.transpose(-2, -1) / 8 ** 0.5
mask = torch.ones(4, 4, dtype=torch.bool).tril()          # True on and below the diagonal
weights = scores.masked_fill(~mask, float("-inf")).softmax(dim=-1)
manual = weights @ v
fast = F.scaled_dot_product_attention(q, k, v, is_causal=True)
print(torch.allclose(manual, fast, atol=1e-6))            # True

Encoder models such as BERT use no causal mask, since every token may see the whole input.

Vaswani et al., Attention Is All You Need (2017)PyTorch: torch.nn.functional.scaled_dot_product_attention

Practise it with Prism

They inject position information. The original transformer added fixed sine and cosine patterns to the embeddings, and learned position embeddings gave nearly identical results. Llama 3 uses rotary embeddings (RoPE), which rotate queries and keys by position-dependent angles so that their dot product depends on relative distance. ALiBi adds no embeddings at all: it penalises attention scores in proportion to distance, and a model trained on 1,024 tokens worked on 2,048.

A common follow-up is length extrapolation: the original authors chose sinusoids partly in the hope of handling longer sequences than seen in training, and ALiBi was built to do exactly that.

Vaswani et al., Attention Is All You Need (2017)Su et al., RoFormer: Enhanced Transformer with Rotary Position Embedding (2021)Press, Smith and Lewis, Train Short, Test Long: Attention with Linear Biases Enables Input Length Extrapolation (2021)Llama Team, The Llama 3 Herd of Models (2024), Table 3

Practise it with Prism

Divide the logits by the temperature and apply softmax; then sort tokens by probability, keep the smallest set whose probabilities add up to at least p, renormalise and sample. Temperature reshapes the whole distribution, while top-p cuts off the unreliable tail, keeping more tokens when the model is unsure and fewer when it is confident. Holtzman and colleagues proposed it because always choosing the likeliest text tends to be bland and repetitive.

import numpy as np
def sample_top_p(logits, p=0.9, temperature=0.7, rng=np.random.default_rng()):
    z = np.asarray(logits) / temperature
    probs = np.exp(z - z.max()); probs /= probs.sum()         # stable softmax
    order = np.argsort(probs)[::-1]                           # likeliest first
    cum = np.cumsum(probs[order])
    keep = order[: np.searchsorted(cum, p) + 1]               # smallest set reaching p
    return rng.choice(keep, p=probs[keep] / probs[keep].sum())
print(sample_top_p([2.0, 1.0, 0.5, -1.0]))                    # token 0 or 1 here

Top-k keeps a fixed number of tokens however confident the model is; that contrast with top-p is the usual follow-up.

Holtzman et al., The Curious Case of Neural Text Degeneration (2019)

Try it in the temperature playgroundPractise it with Prism

Temperature 0 means taking the top token at each step, but the probabilities themselves can differ slightly between runs. A 2025 analysis by Thinking Machines traced most of this in LLM services to batch size: GPU kernels return slightly different numbers depending on how many requests share a batch, and that depends on server load. One flipped near-tie changes every token after it, so pin model versions and evaluate on repeated runs.

PyTorch’s own documentation says fully reproducible results aren’t guaranteed across releases or platforms, or between CPU and GPU, even with the same seed.

Horace He and Thinking Machines Lab, Defeating Nondeterminism in LLM Inference (2025)PyTorch: Reproducibility

Try it in the temperature playgroundPractise it with Prism

Pick examples similar to the input, for instance retrieved by embedding similarity, which beat random examples for GPT-3, and cover the labels and formats you expect. Then test the order: Lu and colleagues found order alone could swing results between near state-of-the-art and random guessing, and a good order for one model didn’t transfer to another. Models also favour answers near the end of the prompt, so balance and shuffle the labels.

Contextual calibration, which measures this bias with a content-free input such as “N/A” and corrects for it, improved GPT-3 accuracy by up to 30.0% absolute in Zhao and colleagues’ tests.

Liu et al., What Makes Good In-Context Examples for GPT-3? (2021)Lu et al., Fantastically Ordered Prompts and Where to Find Them (2021)Zhao et al., Calibrate Before Use: Improving Few-Shot Performance of Language Models (2021)

Practise it with Prism

Use retrieval for the catalogue and keep fine-tuning for behaviour. Ovadia and colleagues found retrieval-augmented generation consistently beat unsupervised fine-tuning at injecting knowledge, old and new, and Gekhman and colleagues found new facts are learned slowly through fine-tuning and, once learned, make hallucination more likely. A catalogue also changes daily, and updating an index is far easier than retraining. Fine-tune for format, tone or a narrow task, on top of retrieval if needed.

Ask how they would remove a discontinued product from the model’s weights; with retrieval, you delete it from the index.

Ovadia et al., Fine-Tuning or Retrieval? Comparing Knowledge Injection in LLMs (2023)Gekhman et al., Does Fine-Tuning LLMs on New Knowledge Encourage Hallucinations? (2024)Lewis et al., Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks (2020)

Practise it with Prism

Distillation trains a small student model to imitate a large teacher. Soft targets, the teacher’s probabilities at a raised softmax temperature, carry more than the right answer: they show which wrong answers are close, so each example teaches more. Hinton and colleagues mixed this loss with a lower-weighted hard-label loss and scaled the soft part by T². DistilBERT used distillation to make BERT 40% smaller and 60% faster while keeping 97% of its language understanding.

import torch.nn.functional as F
def distill_loss(student_logits, teacher_logits, labels, T=2.0, alpha=0.9):
    soft = F.kl_div(F.log_softmax(student_logits / T, dim=-1),
                    F.softmax(teacher_logits / T, dim=-1),
                    reduction="batchmean") * T * T     # scale by T squared
    hard = F.cross_entropy(student_logits, labels)    # temperature 1
    return alpha * soft + (1 - alpha) * hard

In PyTorch’s kl_div, pass log-probabilities first and use reduction="batchmean"; the default "mean" doesn’t give the true KL divergence.

Hinton, Vinyals and Dean, Distilling the Knowledge in a Neural Network (2015)Sanh et al., DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lighter (2019)PyTorch: torch.nn.functional.kl_div

Practise it with Prism

I start with prompting, because it’s the cheapest thing to change. If the model needs knowledge it doesn’t have, or knowledge that changes, like policies or product data, I add RAG. I fine-tune for behaviour that prompting can’t deliver reliably, such as a fixed format, a tone, a narrow classification task, or a small model that handles one job cheaply, and the approaches combine, since fine-tuned models can use RAG too.

If asked for the difference in one line: RAG supplies knowledge at question time, and fine-tuning changes the weights to change behaviour.

The full answer, in the guidePractise it with Prism

LoRA, Low-Rank Adaptation, keeps the pretrained weights frozen and trains small low-rank matrices attached to selected layers. It’s popular because it’s so efficient: on GPT-3 175B, the paper reported trainable parameters down 10,000 times and GPU memory down 3 times versus full fine-tuning with Adam, and inference no slower. In practice I’d use it through Hugging Face’s PEFT library.

Hu et al. 2021: LoRAHugging Face PEFT: LoRA

The full answer, in the guidePractise it with Prism

Quantisation keeps the weights in fewer bits, 8 or 4 for example, which saves memory and cost but gives up some accuracy that you have to measure. QLoRA keeps the base model frozen in 4 bits and trains LoRA adapters through it, using paged optimisers, double quantisation and a 4-bit NormalFloat data type. With it, the paper’s authors managed to fine-tune a 65-billion-parameter model on just one 48 GB GPU.

Dettmers et al. 2023: QLoRA

The full answer, in the guidePractise it with Prism

Both align a model with human preferences. RLHF first learns a reward model from human comparisons, then uses reinforcement learning to optimise the model against that reward. DPO needs neither a separate reward model nor sampling while fine-tuning; it reaches the same goal with a plain classification loss that favours the preferred answer over the rejected one.

Rafailov et al. 2023: Direct Preference Optimization

The full answer, in the guidePractise it with Prism

Each pair of dimensions is a sine and a cosine at its own frequency, with wavelengths in a geometric progression from 2π to 10,000 · 2π, so the encoding behaves like a smooth binary counter: low dimensions change quickly, high ones slowly. Unlike 0/1 bits, the values are continuous and bounded between −1 and 1. And for any fixed offset k, PE(pos + k) is a linear function of PE(pos), which the authors hypothesised would make relative positions easy to attend to.

They also tried learned position embeddings, with nearly identical results, and kept sinusoids in the hope they’d extrapolate to longer sequences.

Vaswani et al., Attention Is All You Need (2017)

Practise it with Prism

PEFT freezes the pretrained weights and trains a small number of new ones, so each task needs only a small add-on. Adapters insert small trainable layers inside each transformer block; Houlsby and colleagues came within 0.4% of full fine-tuning on GLUE while adding 3.6% parameters per task. LoRA adds trainable low-rank matrices beside existing weights and, unlike adapters, adds no inference latency. QLoRA trains LoRA adapters through a base model quantised to 4 bits, enough to fine-tune a 65B model on one 48 GB GPU.

Don’t mix up the axes: instruction tuning is about the data, instructions paired with good answers, and can be done with full fine-tuning or with PEFT.

Houlsby et al., Parameter-Efficient Transfer Learning for NLP (2019)Hu et al., LoRA: Low-Rank Adaptation of Large Language Models (2021)Dettmers et al., QLoRA: Efficient Finetuning of Quantized LLMs (2023)

Practise it with Prism

Each token’s vector is projected three ways: a query, what it’s looking for; a key, what it offers to be matched on; and a value, the content it passes on. softmax(QKᵀ / √d_k) sets how much of each value each token takes. BERT is trained to recover masked words, so every token attends to both sides. GPT is trained to predict the next token, so each token attends only to earlier ones: otherwise training would see the answer, and in generation later tokens don’t exist yet.

BERT’s two-way view suits classification and embeddings, while GPT’s left-to-right view is what lets it generate text one token at a time.

Vaswani et al., Attention Is All You Need (2017)Devlin et al., BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding (2018)

Practise it with Prism

In masked language modelling, 15% of the input tokens are chosen; of those, 80% become [MASK], 10% a random token and 10% stay unchanged, and the model predicts the originals using context from both sides. In next-sentence prediction, it sees two segments and predicts whether the second really followed the first, which was true half the time. BERT was pretrained on BooksCorpus and English Wikipedia, about 3.3 billion words. RoBERTa later found that dropping next-sentence prediction matched or slightly improved results.

The random and unchanged tokens are there because [MASK] never appears during fine-tuning, so the model mustn’t learn to depend on it.

Devlin et al., BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding (2018)Liu et al., RoBERTa: A Robustly Optimized BERT Pretraining Approach (2019)

Practise it with Prism

Perplexity is the exponential of the average negative log-likelihood per token on held-out text: lower means the model predicts it better. It depends on the tokeniser, so compare only like with like, and it isn’t well defined for masked models such as BERT. MMLU tests knowledge across 57 subjects; HumanEval checks whether generated code passes unit tests; BIG-bench gathers 204 tasks from 450 authors; HELM scores models on 7 metrics, from accuracy and calibration to robustness, fairness and efficiency.

Public benchmarks can leak into training data, so also test on your own task.

Hugging Face Transformers: Perplexity of fixed-length modelsHendrycks et al., Measuring Massive Multitask Language Understanding (MMLU, 2020)Chen et al., Evaluating Large Language Models Trained on Code (HumanEval, 2021)Srivastava et al., Beyond the Imitation Game (BIG-bench, 2022)Liang et al., Holistic Evaluation of Language Models (HELM, 2022)

Practise it with Prism

Mostly simplicity at scale. One stack trained to predict the next token learns from every token of raw text, with no need to split data into inputs and targets, and any task can be posed as continuing a prompt, as GPT-3’s few-shot results showed. Generation needs just one model and a KV cache. It isn’t strictly better: Wang and colleagues found causal decoders generalised best zero-shot straight after pretraining, while models that see their whole input, trained with masked objectives, did best after multitask fine-tuning.

Encoder–decoders such as T5 remain a good fit for fixed input-to-output jobs like translation.

Wang et al., What Language Model Architecture and Pretraining Objective Work Best for Zero-Shot Generalization? (2022)Brown et al., Language Models are Few-Shot Learners (GPT-3, 2020)

Practise it with Prism

They shrink the KV cache by letting several query heads share key and value heads. Multi-query attention keeps a single key-value head, which makes decoding much faster for a small quality loss; grouped-query attention keeps a few, reaching quality close to multi-head attention at speed close to multi-query. Llama 3 8B, for example, has 32 query heads but 8 key-value heads, so its KV cache is a quarter of the multi-head size.

Interviewers like the arithmetic: KV cache size scales with the number of key-value heads, not query heads.

Shazeer, Fast Transformer Decoding: One Write-Head is All You Need (2019)Ainslie et al., GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpoints (2023)Llama Team, The Llama 3 Herd of Models (2024), Table 3

Practise it with Prism

About 1 GiB in 16-bit precision. Each token stores a key and a value for every layer and key-value head: 2 × 32 layers × 8 heads × 128 dimensions (4,096 / 32) × 2 bytes = 131,072 bytes, or 128 KiB. Over 8,192 tokens that is 1 GiB for one sequence, so 16 such requests need 16 GiB on top of the weights, which is why KV cache memory limits how many requests a server can batch.

The follow-up is how to cut it: fewer key-value heads (grouped-query attention), shorter contexts, and paged allocation such as vLLM’s PagedAttention, which wastes less of the cache.

Llama Team, The Llama 3 Herd of Models (2024), Table 3Kwon et al., Efficient Memory Management for Large Language Model Serving with PagedAttention (2023)

Practise it with Prism

Prefill processes the whole prompt at once with matrix-matrix operations that keep the GPU busy, and it ends with the first output token. Decode then generates one token per step; each step does matrix-vector work while reading the weights and the KV cache from memory, so it is memory-bound and leaves compute idle. That’s why long prompts raise time to first token, while batching many requests raises decode throughput.

Because decode is memory-bound, serving engines batch many sequences so that each read of the weights serves all of them.

Kwon et al., Efficient Memory Management for Large Language Model Serving with PagedAttention (2023)NVIDIA NIM: LLM inference benchmarking metrics

Practise it with Prism

Training keeps much more than weights. For mixed-precision Adam, the ZeRO paper counts 16 bytes per parameter: 16-bit weights and gradients (2 + 2) plus 32-bit master weights, momentum and variance (12), so a 7B model needs about 112 GB before activations, against 14 GB of 16-bit weights to serve it. LoRA freezes the base model, so gradients and optimiser states exist only for small adapters; QLoRA also stores the frozen base in 4 bits.

ZeRO’s own example: a 1.5-billion-parameter GPT-2 needs at least 24 GB for these states, against 3 GB for its 16-bit weights alone.

Rajbhandari et al., ZeRO: Memory Optimizations Toward Training Trillion Parameter Models (2019), section 3.1Hu et al., LoRA: Low-Rank Adaptation of Large Language Models (2021)Dettmers et al., QLoRA: Efficient Finetuning of Quantized LLMs (2023)

Practise it with Prism

With constrained decoding. The JSON Schema or grammar is compiled into a state machine, and at each step the decoder masks every token that would break it, so only valid continuations can be sampled; Willard and Louf’s Outlines builds an index over the vocabulary so this adds little overhead. It guarantees the format, not the facts, and strict formats have been shown to hurt reasoning, so validate the values and compare quality with and without the constraint.

Providers offer this as a setting, such as strict tool use in Anthropic’s API, which guarantees that tool calls match your schema.

Willard and Louf, Efficient Guided Generation for Large Language Models (2023)Tam et al., Let Me Speak Freely? A Study on the Impact of Format Restrictions on Performance of Large Language Models (2024)Claude API docs: Tool use with Claude

Practise it with Prism

LoRA adds B (4,096 × 8) and A (8 × 4,096): 8 × (4,096 + 4,096) = 65,536 parameters, about 0.4% of the matrix’s 16,777,216. A starts random and B at zero, so the update BA is zero and training begins exactly from the pretrained model; the update is scaled by alpha divided by r. After training, BA can be merged into W, so serving adds no latency.

import torch, torch.nn as nn
class LoRALinear(nn.Module):
    def __init__(self, base: nn.Linear, r=8, alpha=16):
        super().__init__()
        self.base = base.requires_grad_(False)                         # frozen W
        self.A = nn.Parameter(torch.randn(r, base.in_features) * 0.01)  # random
        self.B = nn.Parameter(torch.zeros(base.out_features, r))        # zero: starts as W
        self.scale = alpha / r
    def forward(self, x):
        return self.base(x) + (x @ self.A.T @ self.B.T) * self.scale

A common follow-up is which matrices to adapt: the paper’s GPT-3 experiments adapted only the query and value projections.

Hu et al., LoRA: Low-Rank Adaptation of Large Language Models (2021)

Practise it with Prism

A mixture-of-experts layer replaces one feed-forward block with several experts and a router that sends each token to only a few. Mixtral 8x7B has 8 experts per layer and picks 2 per token, so each token uses about 13B of its 47B parameters: compute per token is like a 13B model’s, but memory must hold all 47B. Training adds a load-balancing loss so the router doesn’t overload a few experts.

Switch Transformers simplified routing to a single expert per token and kept model quality, a useful contrast with Mixtral’s two.

Jiang et al., Mixtral of Experts (2024)Fedus, Zoph and Shazeer, Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsity (2021)Shazeer et al., Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layer (2017)

Practise it with Prism

To attend, each new token needs the keys and values of all the tokens before it, and the KV cache keeps those instead of recomputing them at every step. The limit is memory: the cache gets bigger as sequences lengthen and batches grow, so long contexts end up memory-bound. PagedAttention in vLLM manages the cache in pages, like virtual memory, and its paper reports 2–4 times higher throughput at the same latency.

Hugging Face Transformers: Cache strategiesKwon et al. 2023: Efficient Memory Management for Large Language Model Serving with PagedAttention

The full answer, in the guidePractise it with Prism

Most of the learning happened before your data arrived: pretraining on vast amounts of text built general representations, so the task needs little new information. Few-shot prompting changes no weights at all, as GPT-3 showed, and fine-tuning moves them only slightly: Aghajanyan and colleagues tuned RoBERTa to 90% of its full fine-tuning performance on MRPC by training just 200 parameters projected into the full model. Overparameterised models also show double descent: test error falls again once a model is big enough to fit its training data exactly.

Don’t claim overfitting can’t happen: fine-tune on a few hundred examples for too many epochs and validation scores can still fall.

Brown et al., Language Models are Few-Shot Learners (GPT-3, 2020)Aghajanyan, Zettlemoyer and Gupta, Intrinsic Dimensionality Explains the Effectiveness of Language Model Fine-Tuning (2020)Belkin et al., Reconciling modern machine learning practice and the bias-variance trade-off (2018)

Practise it with Prism

At several levels. FlashAttention computes exact attention in tiles that stay in the GPU’s fast on-chip memory, cutting reads and writes to main memory; its paper reports 3× faster GPT-2 training at 1K tokens. Multi-query and grouped-query attention share key–value heads, shrinking the KV cache with quality close to full multi-head attention. Sliding-window attention, as in Mistral 7B with a window of 4,096 tokens, caps how far back each token looks, so a rolling cache stays a fixed size.

Sliding windows still reach far: Mistral notes that stacking layers gives a theoretical span of about 131K tokens.

Dao et al., FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness (2022)Ainslie et al., GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpoints (2023)Jiang et al., Mistral 7B (2023)

Practise it with Prism

Use automatic metrics for fast, repeatable checks: Fréchet Video Distance compares the statistics of generated and real videos, covering frame quality, motion and variety, and its authors found it correlates well with human judgement; VBench splits quality into 16 dimensions checked against human preferences. Since taste varies, run human studies too: pairwise comparisons with a clear rubric per dimension, several raters per pair, agreement checks and results by user group. Chatbot Arena ranks chat models from crowdsourced pairwise votes in the same spirit.

Report scores per dimension rather than one average, so a gain in visual quality can’t hide a loss in following the prompt.

Unterthiner et al., Towards Accurate Generative Models of Video: A New Metric and Challenges (FVD, 2018)Huang et al., VBench: Comprehensive Benchmark Suite for Video Generative Models (2023)Chiang et al., Chatbot Arena: An Open Platform for Evaluating LLMs by Human Preference (2024)

Practise it with Prism

After prefill, feed only the newest token: each layer appends its new key and value to its own cache and attends over all of it, so the scores have the shape new tokens × (past + new). Positions must continue from the cache length, or every new token is treated as position 0. The attention mask must cover past and new tokens, padding included; one new token needs no causal mask, but prefill still does. Cap the cache at the context length, and use it only for inference.

Test it: greedy generation with and without the cache should give the same tokens, apart from rare floating-point near-ties.

Hugging Face Transformers: How caching worksHugging Face Transformers: Cache strategies

Practise it with Prism

Treat context as a budget, since recall degrades as the window fills. Drop old tool results once they’ve been used, the lightest form of compaction; near the limit, summarise the conversation, keeping decisions and open issues, and continue from the summary. Have the agent write notes to memory outside the window and read them back when needed, retrieve documents instead of pasting them, and hand focused subtasks to sub-agents with fresh windows that return short summaries. Keep fixed instructions first, so prompt caching cuts the cost of the rest.

Anthropic calls that gradual loss of recall context rot, which is why bigger windows don’t remove the need for this.

Anthropic Engineering, Effective context engineering for AI agents (2025)Claude API docs: Prompt caching

Practise it with Prism

Investigate first: collect the failures, build an evaluation set of high-risk questions with expert-checked answers, including ones it should decline, and measure accuracy and calibration, since confident wording on a wrong answer is the real harm. Find the cause: missing knowledge, bad retrieval or the prompt. Then ground answers in vetted sources with citations, allow “I don’t know” explicitly, route high-risk topics to stricter handling or a person, and score evaluations so that abstaining beats guessing, because rewarding guesses is a root cause of hallucination.

Kadavath and colleagues found larger models well calibrated on multiple-choice and true/false questions in the right format, so asking the model to rate its own answer can be a useful signal.

Kalai et al., Why Language Models Hallucinate (2025)Kadavath et al., Language Models (Mostly) Know What They Know (2022)Claude API docs: Reduce hallucinations

Practise it with Prism

First estimate it. Sample several answers and check whether they agree: SelfCheckGPT treats divergent, contradictory samples as a sign of hallucination, and semantic entropy measures disagreement in meaning rather than wording. Token probabilities and the model’s own rating of its answer can help too, once calibrated on labelled data. Then act on it: above a threshold, answer; below it, ask a clarifying question, retrieve more, hedge, or hand over to a person or a fallback flow. At scale, sample a slice for human review and watch abstention and escalation rates.

Sampling several answers multiplies the cost, so save it for high-stakes or low-confidence requests.

Manakul, Liusie and Gales, SelfCheckGPT: Zero-Resource Black-Box Hallucination Detection for Generative Large Language Models (2023)Kuhn, Gal and Farquhar, Semantic Uncertainty: Linguistic Invariances for Uncertainty Estimation in Natural Language Generation (2023)Kadavath et al., Language Models (Mostly) Know What They Know (2022)

Practise it with Prism

Use its configuration: 12 layers, hidden size 768, a 3,072-wide feed-forward layer, 30,522 WordPiece tokens, 512 positions and 2 segment types. The embeddings take (30,522 + 512 + 2) × 768 plus a layer norm: about 23.8M. Each layer has four 768 × 768 attention projections with biases, about 2.4M, two 768 × 3,072 feed-forward matrices, about 4.7M, and two layer norms, about 7.1M in all; 12 layers make 85.1M. A 0.6M pooler brings the total to 109,482,240, the paper’s 110M.

The embedding table alone is more than a fifth of the model, which is why vocabulary size matters so much for small models.

Devlin et al., BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding (2018)Hugging Face: bert-base-uncased config.json

Practise it with Prism