Generative AI interview questions: LLMs, RAG and agents.

1. What is a large language model, and how does it generate text?Easy

A large language model is a neural network, usually a transformer, trained on huge amounts of text to predict the next token. It generates one token at a time, appending each to the input, so every answer is a chain of predictions. Instruction tuning and preference training make it follow requests; none of that turns it into a database of facts.

2. What are tokens, and why do they matter?Easy

Tokens are the pieces a model reads and writes: words, parts of words or characters. Common tokenisers such as byte-pair encoding start from characters and repeatedly merge the most frequent pair.7 Context limits, speed and bills are counted in tokens, and some languages split into more tokens than others, so measure your own text.

3. What is self-attention?Easy

Self-attention lets each token weigh every other token when building its representation. Each token has a query, a key and a value; scores are softmax(QK^T/√d), applied to the values. Multi-head attention runs several in parallel, and positional encodings add word order. The transformer, built on attention alone, comes from Vaswani and colleagues (2017).6

import numpy as np

def attention(Q, K, V):
    scores = Q @ K.T / np.sqrt(K.shape[-1])                  # every token against every token
    weights = np.exp(scores - scores.max(axis=-1, keepdims=True))
    weights /= weights.sum(axis=-1, keepdims=True)           # softmax over the keys
    return weights @ V

4. Encoder, decoder or encoder–decoder?Easy

Encoder-only models such as BERT read the whole input at once and suit classification and embeddings. Decoder-only models such as GPT generate left to right and power chat assistants. Encoder–decoder models such as T5 turn one sequence into another, as in translation or summarisation.

5. What do temperature, top-k and top-p do?Easy

They shape sampling from the next-token probabilities. Temperature rescales them: lower is more predictable, higher more varied. Top-k keeps the k likeliest tokens; top-p keeps the smallest set whose probabilities add up to p. In Hugging Face Transformers, unless a model’s config sets them, the defaults are 1.0, 50 and 1.0.8

outputs = model.generate(**inputs, do_sample=True, temperature=0.3, top_p=0.9, max_new_tokens=200)

6. What is a context window, and does a bigger one solve everything?Easy

The context window is the most tokens a model handles in one request, prompt and answer together. Bigger isn’t free: it costs more and runs slower, and models use information best at the start or end of a long input and worst in the middle. A few well-chosen chunks often beat pasting everything.2

7. What is an embedding, and how do you compare two?Easy

An embedding is a vector that places text by meaning, so similar texts sit close together; compare them with cosine similarity. For Indian-language text, pick a multilingual model: Qwen3-Embedding-0.6B, for example, supports over 100 languages.9

from sentence_transformers import SentenceTransformer

model = SentenceTransformer("Qwen/Qwen3-Embedding-0.6B")
query = model.encode_query("How do I update my KYC?")
docs = model.encode_document(["Steps to complete KYC online", "Today's cricket score"])
print(model.similarity(query, docs))        # the KYC sentence should score higher

8. What is hallucination, and how do you reduce it?Easy

Hallucination is fluent output that the input or the facts don’t support. Reduce it by grounding answers in retrieved documents with citations, telling the model to say when the documents don’t cover the question, lowering temperature for factual tasks, asking for output you can check, and adding a human for high-stakes answers.

9. Zero-shot, few-shot or chain-of-thought prompting?Easy

Zero-shot gives only the instruction. Few-shot adds a handful of worked examples, which helps with format and edge cases. Chain-of-thought asks the model to reason step by step first, which helps multi-step problems. Version and test prompts like code, against a fixed set of examples.

10. What is RAG, and why use it instead of retraining?Easy

Retrieval-augmented generation fetches the passages most relevant to a question and puts them in the prompt, so the model answers from your documents, with citations. The 2020 paper that named it paired a generator with a neural retriever over a dense vector index. Updating knowledge then means updating the index, not the model.1

11. Walk through a RAG pipeline, end to endMedium

12. How do you choose chunk size and overlap?Medium

Small chunks retrieve precisely but lose context; large ones keep context but dilute relevance and cost tokens. Split on structure first (headings, sections, paragraphs), add a modest overlap so a sentence cut at a boundary survives, and pick the size by testing retrieval on your own questions. LangChain’s RecursiveCharacterTextSplitter keeps paragraphs whole, then sentences, then words.10

from langchain_text_splitters import RecursiveCharacterTextSplitter

splitter = RecursiveCharacterTextSplitter(chunk_size=1000, chunk_overlap=150)   # in characters
chunks = splitter.split_text(circular_text)

13. Exact vector search or an approximate index such as HNSW?Medium

Exact (flat) search compares the query with every vector: perfect recall, slow at scale. HNSW, a graph-based approximate index, is far faster for a little lost recall, and FAISS notes it can’t remove vectors. For cosine similarity in FAISS, normalise the vectors and use inner product.11

import faiss
import numpy as np

vecs = np.asarray(model.encode_document(chunks), dtype="float32")
faiss.normalize_L2(vecs)                      # unit length: inner product = cosine
index = faiss.IndexFlatIP(vecs.shape[1])      # exact; move to HNSW at scale
index.add(vecs)

q = np.asarray([model.encode_query("Late fee for filing an ITR?")], dtype="float32")
faiss.normalize_L2(q)
scores, ids = index.search(q, 3)              # the top 3 chunks

14. Why hybrid search and re-ranking?Medium

Embeddings match meaning but can miss exact terms such as a section number, a circular ID or a product code; keyword search (BM25) catches those. Hybrid search runs both and merges the lists, often with reciprocal rank fusion. A cross-encoder re-ranker then reads the question and each candidate together and re-scores the top results.9

15. How do you evaluate a RAG system?Medium

Score retrieval and answers separately, on a golden set of real questions with known answers and sources. Retrieval: did the right passage reach the top k, and how high? Answers: are they supported by the retrieved text (faithfulness), and do they answer the question? Ragas calls these context precision, context recall, faithfulness and response relevancy.12

16. The answer isn’t in the documents. What should the assistant do?Medium

Say so, not guess. Instruct it to answer only from the retrieved context and to reply “I couldn’t find this in the documents”, set a minimum retrieval score below which it doesn’t answer, and put unanswerable questions, such as next year’s figures, in your test set so refusals get measured too.

17. Prompting, RAG or fine-tuning: how do you choose?Medium

Start with prompting: it’s cheapest to change. Add RAG when the model needs knowledge it lacks or that changes, such as policies or product data. Fine-tune for behaviour prompting can’t get reliably: a format, a tone, a narrow classification, or a small model doing one task cheaply. They combine: a fine-tuned model can still use RAG.

18. What is LoRA, and why is it popular?Medium

Low-Rank Adaptation freezes the pretrained weights and trains small low-rank matrices added to chosen layers. On GPT-3 175B, its paper reported 10,000 times fewer trainable parameters and 3 times less GPU memory than full fine-tuning with Adam, and no added inference latency. Hugging Face’s PEFT library implements it.314

from peft import LoraConfig, get_peft_model

config = LoraConfig(r=16, lora_alpha=32, lora_dropout=0.05,
                    target_modules="all-linear", task_type="CAUSAL_LM")
model = get_peft_model(base_model, config)
model.print_trainable_parameters()       # a small fraction of the total

19. What do quantisation and QLoRA trade?Medium

Quantisation stores weights in fewer bits, such as 8 or 4, cutting memory and cost for some accuracy you must measure. QLoRA trains LoRA adapters through a frozen 4-bit model, with a 4-bit NormalFloat type, double quantisation and paged optimisers; its paper fine-tuned a 65-billion-parameter model on a single 48 GB GPU.13

20. How do tool calling and structured output work?Medium

You describe a tool with a name, a description and a JSON Schema for its inputs; the model replies with a tool call, your code runs it, and you return the result. MCP tools use the same shape. Never trust the JSON blindly: validate it, then retry or hand over to a person when it fails.5

from pydantic import BaseModel, ValidationError

class Complaint(BaseModel):
    category: str
    urgency: int              # 1 to 5
    refund_requested: bool

try:
    complaint = Complaint.model_validate_json(llm_output)
except ValidationError as err:
    ...                       # retry with the error message, or route to a person

21. What is an AI agent, and how is it different from a RAG chain?Medium

A RAG chain runs fixed steps: retrieve, then answer. An agent is a loop: the model picks the next action (call a tool, search, ask the user), observes the result and repeats until done, the pattern ReAct described as interleaving reasoning and actions. Agents handle open tasks but need step limits, tool permissions and logs.15

def run_agent(task, llm, tools, max_steps=6):           # a sketch, not a framework
    history = [{"role": "user", "content": task}]
    for _ in range(max_steps):
        step = llm(history, tools)                       # the model picks a tool or answers
        if step.tool is None:
            return step.answer
        result = tools[step.tool](**step.arguments)      # your code runs the tool
        history.append({"role": "tool", "content": str(result)})
    return "Stopped at the step limit"

22. What is MCP, and why not paste the API docs into the prompt?Medium

The Model Context Protocol connects AI applications to external systems: a host, such as an IDE or chat app, runs one client per server, and servers expose tools, resources and prompts over stdio or Streamable HTTP. Pasted docs cost tokens on every call and run nothing; an MCP server is built once, works with any MCP host and keeps credentials on the server.5

from mcp.server import MCPServer

mcp = MCPServer("ifsc-lookup")

@mcp.tool()
def bank_branch(ifsc: str) -> str:
    """Return the bank and branch for an IFSC code."""
    return lookup_ifsc(ifsc)              # your own function

if __name__ == "__main__":
    mcp.run(transport="stdio")            # a stdio server must never print to stdout

23. Which agent framework would you choose?Medium

Choose by need, and show you could write the loop yourself. A plain chain suits fixed steps. LangGraph, a low-level framework for long-running, stateful agents, suits loops, checkpoints and human approval; multi-agent frameworks coordinate several specialised agents. Interviewers care more about state, retries and evaluation than about the library.16

24. RLHF or DPO?Medium

Both tune a model toward human preferences. RLHF first fits a reward model on human comparisons, then optimises the model against it with reinforcement learning. DPO skips the separate reward model and the sampling during fine-tuning, solving the same problem with a simple classification loss on preferred and rejected answers.17

25. How does the KV cache speed up generation, and what limits it?Hard

Each new token’s attention needs the keys and values of every earlier token; the KV cache stores them instead of recomputing them. It grows with sequence length and batch size, so long contexts become memory-bound. vLLM’s PagedAttention manages the cache like virtual-memory pages, which its paper reports raises throughput 2–4 times at the same latency.1819

26. How would you serve an open model fast and cheaply?Hard

Measure first: time to first token, tokens per second and cost per request. Then use a serving engine with continuous batching and paged KV caching, such as vLLM; quantise if quality holds; cap the output length; cache repeated prefixes; and route easy requests to a smaller model.19

27. How do you cut the cost of an LLM feature?Hard

Count tokens per request and requests per user, then shorten system prompts and retrieved context, cache answers to repeated questions, route simple requests to a smaller model, batch offline work, and set output limits and per-user budgets. OWASP lists runaway usage as a risk in its own right: unbounded consumption.4

28. What is prompt injection, and how do you defend against it?Hard

Prompt injection is input that turns the model against your instructions, typed by a user (direct) or hidden in a web page, email or retrieved document (indirect). It tops OWASP’s 2025 list. No prompt fully prevents it, so limit the damage: least-privilege tools, human approval for actions, instructions kept apart from data, and outputs checked before they run.4

29. Which OWASP LLM risks would you design for?Hard

Know the 2025 list: prompt injection, sensitive information disclosure, supply chain, data and model poisoning, improper output handling, excessive agency, system prompt leakage, vector and embedding weaknesses, misinformation and unbounded consumption. For a RAG assistant, stress vector and embedding weaknesses and misinformation; for agents, excessive agency.4

30. Keep one client’s documents away from another client’s usersHard

Enforce isolation in retrieval, never in the prompt: tag every chunk with a tenant ID and apply a metadata filter on every query, taken from the signed-in user’s identity on the server, or keep a separate index per tenant. Test with cross-tenant questions, and give caches and logs the same isolation.

31. How do you handle personal data in an LLM app?Hard

Collect less, then find and mask personal data (names, phone numbers, account numbers) before indexing, logging or sending text to an outside model. The open-source Presidio has an analyzer and an anonymizer, though its docs warn there’s no guarantee it finds everything. Add access controls and retention limits, and check India’s Digital Personal Data Protection Act with your legal team.20

32. LLM-as-a-judge: when can you trust it?Hard

Use it for scale, with checks. The MT-Bench study found strong judges agreed with human preferences over 80% of the time, about as often as humans agree with each other, but it also found position, verbosity and self-enhancement biases. So swap answer order, give a rubric and reference answers, and spot-check against human labels.21

33. Design a policy assistant for a bank’s branch staffHard

1. Scope: staff ask about circulars, products and procedures; every answer cites the clause and its date.
2. Data: parse circulars and policy PDFs, tables included; keep effective dates; retire superseded versions.
3. Retrieval: chunk by section; hybrid search so circular numbers match exactly; filter by date and department; re-rank.
4. Answers: quote and cite; refuse when not covered; no customer-specific advice.
5. Safety: role-based document access, personal-data masking, injection checks on uploads, audit logs.
6. Quality: a golden set reviewed by compliance, faithfulness and citation checks, feedback and monitoring.

34. Design a job that summarises 1,000 news articles every nightHard

A thousand articles won’t fit one context window, so map, then reduce: group near-duplicates, summarise each article or group, then summarise the summaries by topic. Run it as a batch job with retries and steps that are safe to repeat, process only new or changed articles, track cost per run, and spot-check before publishing.

35. Walk us through your GenAI projectHard

Every kind of company starts here. Cover:

1. The problem and users, and why GenAI rather than rules or classic ML.
2. The architecture: data, chunking, retrieval, model, prompts, tools.
3. Evaluation: your golden set, metrics and the numbers before and after changes.
4. Failures: hallucinations, bad retrieval, injection, and what you did.
5. Cost and latency per request, and what changes at ten times the users.