Deep learning interview questions, answered the way you’d say them.

It applies the chain rule backwards through the recorded computation graph and stores each parameter’s gradient in its .grad. For a prediction w × x + b with x = 2, w = 3, b = 0.5 and target 7, the prediction is 6.5 and the error -0.5, so the squared-error loss is 0.25; dL/dw = 2 × (-0.5) × 2 = -2 and dL/db = 2 × (-0.5) = -1. Gradients accumulate across calls, so zero them every step.

import torch
x, y = torch.tensor(2.0), torch.tensor(7.0)
w = torch.tensor(3.0, requires_grad=True)
b = torch.tensor(0.5, requires_grad=True)
loss = (w * x + b - y) ** 2          # prediction 6.5, error -0.5, loss 0.25
loss.backward()                       # chain rule from the loss back to w and b
print(w.grad, b.grad)                 # tensor(-2.) tensor(-1.)

Forgetting to call the optimiser’s zero_grad() lets gradients from earlier batches pile up, a classic bug.

PyTorch tutorial: Automatic differentiation with torch.autograd

Practise it with Prism

Without a non-linearity, stacked layers collapse into one linear map, since W2(W1x) = (W2W1)x, so depth adds nothing. Sigmoid and tanh saturate: for large inputs their gradients approach zero, which slows learning in deep stacks. ReLU, max(0, x), is cheap, doesn’t saturate for positive inputs and gives sparse activations, and Glorot and colleagues found rectifier networks matched or beat tanh ones. GELU, which weights inputs by their value instead of gating them by sign, is a smoother alternative.

ReLU outputs a hard zero for negative inputs, which passes no gradient; Glorot and colleagues flagged this risk but still found rectifiers trained well.

Glorot, Bordes and Bengio, Deep Sparse Rectifier Neural Networks (AISTATS 2011)Hendrycks and Gimpel, Gaussian Error Linear Units (GELUs) (2016)

Try it in the network trainerPractise it with Prism

During training, dropout zeroes each unit with probability p on every forward pass, so units can’t rely on particular partners; the original paper calls this preventing co-adaptation, and it is like training many thinned networks at once. At inference nothing is dropped. PyTorch scales the kept activations by 1 / (1 - p) during training, so the layer is simply the identity in evaluation mode; the original paper scaled the weights down at test time instead.

Call model.eval() before validation and inference, or dropout keeps removing units at random and your scores wobble.

Srivastava et al., Dropout: A Simple Way to Prevent Neural Networks from Overfitting (JMLR 2014)PyTorch: torch.nn.Dropout

Practise it with Prism

An embedding layer is a learned lookup table: row i is a dense vector for category or token i. Looking up a row gives the same result as multiplying a one-hot vector by the weight matrix, without building the huge sparse vector. The vectors train with the rest of the network, so similar items can end up close together, and a 10,000-item vocabulary at size 64 needs 640,000 numbers.

import torch, torch.nn as nn
emb = nn.Embedding(num_embeddings=10_000, embedding_dim=64)
ids = torch.tensor([[12, 857, 3]])                       # one sequence of three IDs
print(emb(ids).shape)                                    # torch.Size([1, 3, 64])
one_hot = nn.functional.one_hot(ids, 10_000).float()
print(torch.allclose(one_hot @ emb.weight, emb(ids)))    # True: the same lookup

Embeddings also suit high-cardinality tabular features such as PIN codes or product IDs, where one-hot columns would explode.

PyTorch: torch.nn.Embedding

Practise it with Prism

Because it applies log-softmax itself, in a numerically stable way, then takes the negative log-likelihood of the correct class. If you apply softmax first, the loss applies it again, squashing the scores and weakening the gradients: a silent bug that still trains, only badly. So the last layer should output raw scores, and you apply softmax only when you need probabilities, for display or thresholds.

import torch, torch.nn as nn
logits = torch.tensor([[2.0, 0.5, -1.0]])        # raw scores for 3 classes
target = torch.tensor([0])
loss = nn.CrossEntropyLoss()(logits, target)      # applies log-softmax itself
manual = -torch.log_softmax(logits, dim=1)[0, 0]
print(loss.item(), manual.item())                 # the same number, about 0.241

The same rule holds for binary tasks: BCEWithLogitsLoss takes logits and applies the sigmoid itself, more stably.

PyTorch: torch.nn.CrossEntropyLossPyTorch: torch.nn.BCEWithLogitsLoss

Practise it with Prism

Padding P on each side makes the input W + 2P wide. A kernel of size K can start anywhere from 0 to W + 2P − K, and stride S takes every S-th start, so there are ⌊(W + 2P − K) / S⌋ + 1 positions: the +1 counts the first one, as with fence posts. A 224-pixel input with a 3 × 3 kernel, padding 1 and stride 2 gives ⌊223 / 2⌋ + 1 = 112.

With dilation D, use the effective kernel size D(K − 1) + 1; pooling layers use the same formula, which PyTorch rounds down unless you set ceil_mode=True.

PyTorch: torch.nn.Conv2dPyTorch: torch.nn.MaxPool2dDumoulin and Visin, A guide to convolution arithmetic for deep learning (2016)

Practise it with Prism

Read the error first: it names the operation and both shapes. Then pass one small batch through and print each tensor’s shape after every layer. The usual culprits are a missing flatten before a linear layer, whose in_features must equal channels × height × width; dimensions in the wrong order, such as an RNN expecting batch_first; a single example with no batch dimension; and broadcasting that silently pairs a target of shape (N,) with predictions of shape (N, 1), which mse_loss warns about.

nn.LazyLinear infers in_features from the first input it sees, so you needn’t recalculate it whenever the layers before it change.

PyTorch: torch.nn.LazyLinearPyTorch source: torch.nn.functional.mse_loss (the size warning)

Practise it with Prism

Before the full run, check that the starting loss matches random guessing, −log(1/C) for C classes, and that the model can overfit a batch of a few examples; if it can’t, there’s a bug. Then watch training and validation loss, the gradient norm and the learning rate: a flat loss suggests a bug or a rate too low, spikes or NaN one too high. model.eval() switches dropout off and makes batch norm use its running statistics; it doesn’t stop gradient tracking, so also use torch.no_grad().

Watching predictions on one fixed batch as training runs, as Karpathy recommends, shows instability that a loss curve can hide.

Karpathy, A Recipe for Training Neural Networks (2019)PyTorch: Locally disabling gradient computationPyTorch: torch.nn.Module (eval)

Practise it with Prism

Training is mostly large matrix multiplications: millions of independent multiply-adds that can run at the same time. A CPU is built to run a sequence of operations as fast as possible and handles a few tens of threads at once; a GPU devotes more of its transistors to arithmetic than to caching and control, runs thousands of threads in parallel and has much higher memory bandwidth. Tiny models or batches may gain little, because launching work and copying data between CPU and GPU take time.

Keep the GPU fed: load data in parallel workers, so it isn’t waiting for the CPU between batches.

NVIDIA CUDA C++ Programming Guide (11.4), 1.1 The Benefits of Using GPUs

Practise it with Prism

Backpropagation multiplies one factor per layer or time step, so if those factors are mostly below 1 the gradient shrinks exponentially with depth, and if they are above 1 it blows up. Vanishing is eased by ReLU-type activations, careful initialisation, normalisation layers, residual connections and, in recurrent networks, LSTM gates. Exploding is handled by clipping the gradient norm, which Pascanu and colleagues proposed for recurrent networks, and by a lower learning rate.

Clip between loss.backward() and the optimiser’s step(), for example with torch.nn.utils.clip_grad_norm_(model.parameters(), max_norm=1.0), and log the norm it returns: a sudden spike before the loss turns NaN is the classic sign.

Pascanu, Mikolov and Bengio, On the difficulty of training Recurrent Neural Networks (2012)PyTorch: torch.nn.utils.clip_grad_norm_He et al., Deep Residual Learning for Image Recognition (2015)

Practise it with Prism

If every weight starts equal, every unit in a layer computes the same output and gets the same gradient, so the units never learn different features. Random weights break that symmetry, but their scale matters: too large and activations explode, too small and they vanish as depth grows. Xavier initialisation sets the weight variance to 2 / (fan_in + fan_out), designed around sigmoid and tanh; He initialisation uses 2 / fan_in for ReLU, which zeroes about half its inputs.

PyTorch’s nn.Linear already draws its weights uniformly between -1/√in_features and +1/√in_features, so you won’t start at zero by accident.

Glorot and Bengio, Understanding the difficulty of training deep feedforward neural networks (AISTATS 2010)He et al., Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classification (2015)PyTorch: torch.nn.initPyTorch: torch.nn.Linear

Practise it with Prism

Batch norm normalises each feature across the examples in a mini-batch and keeps running averages for inference, so it behaves differently in training and evaluation and depends on batch size. Layer norm normalises across the features of each single example, doing exactly the same computation at training and test time, which suits variable-length sequences. The original transformer applies layer normalisation after each residual sub-layer.

Forgetting model.eval() leaves batch norm using batch statistics at inference, so a prediction can change with whatever else is in the batch.

Ioffe and Szegedy, Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift (2015)Ba, Kiros and Hinton, Layer Normalization (2016)PyTorch: torch.nn.BatchNorm1dPyTorch: torch.nn.LayerNormVaswani et al., Attention Is All You Need (2017)

Practise it with Prism

Plain SGD steps against each mini-batch gradient with one learning rate. Momentum adds a decaying sum of past gradients, damping zig-zags and speeding progress in consistent directions. Adam keeps running averages of the gradient and of its square, corrects their start-up bias and so gives each parameter its own step size; its paper’s defaults are a learning rate of 0.001, β1 = 0.9, β2 = 0.999 and ε = 10⁻⁸. AdamW applies weight decay separately, which improved Adam’s generalisation.

Adam stores two extra values per parameter; with a 32-bit master copy of the weights, optimiser state takes 12 of the 16 bytes per parameter in ZeRO’s mixed-precision count.

Kingma and Ba, Adam: A Method for Stochastic Optimization (2014), Algorithm 1Loshchilov and Hutter, Decoupled Weight Decay Regularization (2017)PyTorch: torch.optim.SGDRajbhandari et al., ZeRO: Memory Optimizations Toward Training Trillion Parameter Models (2019), section 3.1

Practise it with Prism

A large learning rate makes fast early progress, and a smaller one later lets the model settle. Warm-up starts small and ramps the rate up over the first steps; the original transformer warmed up linearly for 4,000 steps, then decayed with the inverse square root of the step number. Cosine decay lowers the rate along half a cosine curve, the shape SGDR introduced, down to a floor that defaults to zero in PyTorch.

import torch
from torch.optim.lr_scheduler import LinearLR, CosineAnnealingLR, SequentialLR
model = torch.nn.Linear(10, 1)                                 # any model
opt = torch.optim.AdamW(model.parameters(), lr=3e-4)
warmup = LinearLR(opt, start_factor=0.01, total_iters=500)   # 1% to 100% over 500 steps
decay = CosineAnnealingLR(opt, T_max=9_500)                   # then down towards 0
sched = SequentialLR(opt, schedulers=[warmup, decay], milestones=[500])
# each step: loss.backward(); opt.step(); sched.step(); opt.zero_grad()

Step the scheduler as often as you designed it, per batch or per epoch; mixing the two up is a common bug.

Vaswani et al., Attention Is All You Need (2017)Loshchilov and Hutter, SGDR: Stochastic Gradient Descent with Warm Restarts (2016)PyTorch: torch.optim.lr_scheduler.SequentialLRPyTorch: torch.optim.lr_scheduler.LinearLRPyTorch: torch.optim.lr_scheduler.CosineAnnealingLR

Practise it with Prism

A convolution slides small learned filters over the image, connecting each output to a local patch and reusing the same weights at every position, so it detects a pattern wherever it appears and its size doesn’t grow with the image. Pooling downsamples by keeping the maximum or average of each window. A 3 × 3 convolution from 3 to 64 channels has 1,792 parameters; a dense layer from a 224 × 224 × 3 image to 64 units has 9,633,856.

import torch.nn as nn
conv = nn.Conv2d(3, 64, kernel_size=3, padding=1)
dense = nn.Linear(224 * 224 * 3, 64)
count = lambda m: sum(p.numel() for p in m.parameters())
print(count(conv), count(dense))     # 1792 9633856

Know the output-size formula for dilation 1: (input + 2 × padding - kernel) / stride + 1, rounded down.

PyTorch: torch.nn.Conv2dPyTorch: torch.nn.MaxPool2d

Practise it with Prism

Transfer learning starts from a network trained on a large dataset and adapts it to your task. Yosinski and colleagues showed early layers learn general features, such as Gabor-like filters and colour blobs, while later ones become task-specific, and transfer weakens as the tasks grow apart. So with little data similar to the original, freeze the backbone and train a new head; with more data or a different domain, fine-tune more layers at a lower learning rate.

import torch.nn as nn
from torchvision.models import resnet18, ResNet18_Weights
model = resnet18(weights=ResNet18_Weights.DEFAULT)     # ImageNet-trained features
for p in model.parameters():
    p.requires_grad = False                             # freeze the backbone
model.fc = nn.Linear(model.fc.in_features, 5)           # new trainable head, 5 classes

Feed images with the preprocessing the pretrained weights expect; torchvision bundles it as weights.transforms().

Yosinski et al., How transferable are features in deep neural networks? (2014)torchvision: resnet18torchvision: Models and pre-trained weights

Practise it with Prism

The network is overfitting, so first rule out leakage or a bug, then regularise. Keep the checkpoint with the best validation loss (early stopping), add data or augmentation, add weight decay or dropout, or use a smaller or pretrained model. Deep networks can memorise almost anything: Zhang and colleagues showed standard image networks easily fit randomly labelled training data, and explicit regularisation didn’t stop it, so trust validation curves, not model size.

You may be asked how to tell overfitting from a shift in the data: check whether the validation set still resembles the training set.

Zhang et al., Understanding deep learning requires rethinking generalization (2016)Srivastava et al., Dropout: A Simple Way to Prevent Neural Networks from Overfitting (JMLR 2014)

Practise it with Prism

In PyTorch, dropout zeroes each element with probability p during training and scales the survivors by 1 / (1 − p). Two in a row keep an element only if both keep it, with probability (1 − p₁)(1 − p₂), and the scales multiply, so the pair is exactly one dropout layer with p = 1 − (1 − p₁)(1 − p₂): two at 0.5 act as one at 0.75. That only adds heavier dropout. Between layers, each regularises a different representation; at evaluation, both are the identity.

The masks are drawn afresh on every forward pass, so the combined mask is still a new random draw each step.

PyTorch: torch.nn.Dropout

Practise it with Prism

The usual causes are a learning rate high enough to make gradients explode; an unstable operation, such as log of zero, division by zero, or softmax followed by a separate log instead of log-softmax; NaN or infinite values in the inputs or labels; and float16 overflow, since its largest value is 65,504. Check the data first and log the gradient norm every step. Then lower the learning rate, clip gradients, or use a gradient scaler or bfloat16; torch.autograd.detect_anomaly() points to the operation that produced the NaN.

bfloat16 has the same dynamic range as float32, so it rarely overflows where float16 does.

PyTorch: torch.autograd.detect_anomalyPyTorch: Automatic Mixed Precision package (torch.amp)NVIDIA: Train With Mixed PrecisionGoogle Cloud: The bfloat16 numerical format

Practise it with Prism

The gradient points in the direction of steepest increase, so a small step against it lowers the loss fastest: to first order, f(w − ηg) ≈ f(w) − η‖g‖². There’s no single best batch size. Larger batches give smoother gradients but fewer updates per epoch, and Keskar and colleagues linked very large batches to sharp minima that generalise worse. Goyal and colleagues trained ResNet-50 with batches of 8,192 images without losing accuracy by scaling the learning rate with the batch and warming up, so tune the two together.

Their linear scaling rule: when you multiply the batch size by k, multiply the learning rate by k, with a short warm-up at the start.

Goyal et al., Accurate, Large Minibatch SGD: Training ImageNet in 1 Hour (2017)Keskar et al., On Large-Batch Training for Deep Learning: Generalization Gap and Sharp Minima (2016)

Try it in the network trainerPractise it with Prism

It cuts the image into fixed-size patches, 16 × 16 pixels for example, so a 224 × 224 image becomes 196 patches, flattens each and projects it linearly into an embedding. It prepends a learnable class token, adds position embeddings and runs a standard transformer encoder, classifying from the class token’s output. Lacking a CNN’s built-in locality and translation equivariance, ViT trained on ImageNet alone scores a few points below comparable ResNets, but pretrained on 14 to 300 million images, it compares well with the best CNNs.

That’s why ViTs are usually fine-tuned from large pretrained checkpoints rather than trained from scratch on small datasets.

Dosovitskiy et al., An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale (2020)

Practise it with Prism

Start from a backbone pretrained on a large image dataset, replace the head with 500 outputs and fine-tune with softmax cross-entropy, using crops, flips and colour changes as augmentation. Species data is heavily imbalanced, with look-alike species; the iNaturalist authors saw poor results for classes with few training images. So rebalance the sampling or weight the loss, report top-1 and top-5 accuracy plus macro-averaged recall, so rare species count equally, and study the confusion matrix. Split by location or photographer, so near-duplicate photos don’t leak.

Ask how it will be used: a field app might show five candidate species for a person to confirm, which makes top-5 accuracy the number that matters.

Van Horn et al., The iNaturalist Species Classification and Detection Dataset (2017)

Practise it with Prism

Weight decay shrinks every weight a little towards zero at each step, which favours simpler functions; with plain SGD it’s equivalent to an L2 penalty, but with Adam it isn’t, which is why AdamW decouples the decay from the gradient step. Dropout instead zeroes random activations during training, so units can’t rely on particular partners, and it’s switched off at evaluation. One limits how large the weights grow, the other injects noise into the computation, and they’re often used together.

Loshchilov and Hutter showed that decoupling the decay from Adam’s update made the best decay factor independent of the learning rate and improved generalisation.

Loshchilov and Hutter, Decoupled Weight Decay Regularization (2017)PyTorch: torch.nn.Dropout

Practise it with Prism

A shallow network learns word vectors from a made-up task: skip-gram predicts the words around each word within a window, while CBOW predicts a word from its context. A full softmax over the vocabulary is too slow, so negative sampling turns each step into logistic regression: tell the true neighbour apart from k random noise words, with k of 5 to 20 for small datasets and 2 to 5 for large ones. Very frequent words are subsampled, and the learned input vectors become the embeddings.

Words used in similar contexts end up with similar vectors, which is why analogies such as king − man + woman ≈ queen work.

Mikolov et al., Efficient Estimation of Word Representations in Vector Space (2013)Mikolov et al., Distributed Representations of Words and Phrases and their Compositionality (2013)

Practise it with Prism

Almost everywhere. Token IDs pick rows of the embedding matrix, and queries, keys and values are the input X times learned matrices W_Q, W_K and W_V. QKᵀ takes the dot product of every query with every key, an n × n matrix for n tokens, which is scaled, softmaxed and multiplied by V, so each output is a weighted average of value vectors. The heads are concatenated and projected by W_O, and the feed-forward block is two matrix multiplications around a non-linearity.

That n × n score matrix is why attention’s cost grows with the square of the sequence length.

Vaswani et al., Attention Is All You Need (2017)

Practise it with Prism

Split it. Whisper turns each 30-second window into an 80-channel log-Mel spectrogram, so long audio is handled window by window. The original approach is sequential: transcribe one window, then move forward using the timestamps the model predicted, with fallbacks such as beam search and a higher temperature when output repeats. The chunked approach, used by Whisper-JAX, cuts the audio into 30-second chunks, transcribes them in parallel batches and stitches the results at the boundaries, which is far faster on accelerators. Cutting at silences avoids splitting words.

Whisper-JAX reports running over 70 times faster than OpenAI’s original PyTorch code.

Radford et al., Robust Speech Recognition via Large-Scale Weak Supervision (Whisper, 2022)Whisper JAX: README

Practise it with Prism

Tokens become embeddings, with position information added because attention alone ignores order. Each layer has two sub-layers: multi-head self-attention, where every token gathers information from the others, and a feed-forward network applied to each token separately; each is wrapped in a residual connection and layer normalisation, which keep deep stacks trainable. The original model stacked 6 encoder and 6 decoder layers; the decoder adds masked self-attention and attention over the encoder’s output, and a final linear layer and softmax give next-token probabilities.

Know the base model’s sizes: d_model of 512, 8 heads and a 2,048-wide feed-forward layer.

Vaswani et al., Attention Is All You Need (2017)

Practise it with Prism

As a quick baseline, engineer pair features: word and character n-gram overlap, TF-IDF cosine similarity, length difference, matching numbers and named entities, and embedding similarity, then train gradient-boosted trees on labelled pairs. For the best accuracy, fine-tune a cross-encoder such as BERT, which reads both sentences together; BERT was evaluated this way on the MRPC and QQP paraphrase tasks. To compare each sentence against millions, use a bi-encoder such as Sentence-BERT for speed, then re-rank the top matches with the cross-encoder.

Test the hard cases on purpose: negations, changed numbers and swapped roles, such as “A paid B” against “B paid A”, which share almost every word.

Devlin et al., BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding (2018)Reimers and Gurevych, Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks (2019)

Practise it with Prism

Frameworks pick a value: PyTorch uses the gradient where it exists, a minimum-norm subgradient at kinks of convex functions such as ReLU at 0, and continuity elsewhere. Where the gradient is zero, as for ReLU’s negative inputs or a saturated sigmoid, nothing flows back, so the weights feeding that unit get no update from that example, and a unit that’s always off never recovers. Step functions, rounding and argmax have zero gradient almost everywhere, so train through them with a smooth surrogate or the straight-through estimator.

Bengio and colleagues’ straight-through estimator simply passes the gradient through the hard threshold as if it weren’t there.

PyTorch: Gradients for non-differentiable functionsBengio, Léonard and Courville, Estimating or Propagating Gradients Through Stochastic Neurons for Conditional Computation (2013)

Practise it with Prism

During training, batch norm normalises each feature with the current mini-batch’s mean and variance, then applies a learned scale γ and shift β, while keeping running estimates of the mean and variance; PyTorch updates them with a momentum of 0.1. At inference, after model.eval(), it uses those running estimates instead, so a prediction depends only on its own input, not on the rest of the batch. With fixed statistics, the whole layer becomes a linear transform that can be folded into the previous layer’s weights.

Very small training batches give noisy statistics; that’s where layer normalisation, which doesn’t depend on the batch, can work better.

Ioffe and Szegedy, Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift (2015)PyTorch: torch.nn.BatchNorm2d

Practise it with Prism

Plain recurrent networks pass information through every time step, so gradients over long sequences vanish or explode. An LSTM adds a cell state updated additively under input, forget and output gates, letting information and gradients survive many steps. But recurrence is sequential: a recurrent layer needs O(n) sequential operations and an O(n) path between distant tokens, against O(1) for self-attention, so transformers train in parallel and connect distant words directly, at a cost quadratic in sequence length.

Vaswani and colleagues also note that self-attention is cheaper per layer than recurrence when the sequence is shorter than the representation size.

Pascanu, Mikolov and Bengio, On the difficulty of training Recurrent Neural Networks (2012)PyTorch: torch.nn.LSTMVaswani et al., Attention Is All You Need (2017)

Practise it with Prism

A residual block outputs x + F(x), so its layers learn only a correction to their input, and the identity path carries signals and gradients straight through. He and colleagues found plain networks got worse as they grew deeper, with higher training error, not just overfitting, while residual networks kept gaining accuracy: they trained 152-layer models, eight times deeper than VGG, and an ensemble reached 3.57% error on ImageNet. Transformers put one around every sub-layer.

If F(x) and x differ in shape, the shortcut uses a projection, such as a 1 × 1 convolution, to match them.

He et al., Deep Residual Learning for Image Recognition (2015)Vaswani et al., Attention Is All You Need (2017)

Practise it with Prism

It runs a forward pass to predict, scores the error with a loss, then uses backpropagation, the chain rule, to work out each weight’s gradient. Gradient descent nudges every weight against its gradient, usually one mini-batch at a time, by a step the learning rate sets: too large and training diverges, too small and it crawls. Non-linear activations like ReLU let it fit curves, piling up sigmoid layers can cause vanishing gradients, and dropout helps against overfitting.

The full answer, in the guideTry it in the network trainerPractise it with Prism

CLIP trains an image encoder and a text encoder together on 400 million image–text pairs. For a batch of N pairs it scores all N × N cosine similarities, scaled by a learned temperature, with a symmetric cross-entropy: each image must pick its caption from the N, and each caption its image, so 32 pairs make a 32-way classification both ways. BLIP-2’s Q-Former is a lightweight transformer whose 32 learned queries extract a fixed set of visual features from a frozen image encoder for a frozen language model.

For recommendation, the shared space lets you embed catalogue images once, then match a shopper’s text search or a liked product’s photo to them by nearest-neighbour search.

Radford et al., Learning Transferable Visual Models From Natural Language Supervision (CLIP, 2021)Li et al., BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Models (2023)

Practise it with Prism

For imbalance, oversample or up-weight rare classes and report recall per class, not just the average. Hard negatives are non-matching pairs the model currently scores as similar, such as near-identical products; BLIP samples negatives with higher contrastive similarity more often, which teaches finer distinctions than random ones. Against forgetting, use a low learning rate, freeze most of the backbone or train adapters, mix in general data and re-test zero-shot benchmarks. WiSE-FT, which averages the fine-tuned and original weights, gained 4 to 6 points over prior work under distribution shift.

WiSE-FT costs nothing extra at inference: the averaged weights form a single model.

Li et al., BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generation (2022)Wortsman et al., Robust fine-tuning of zero-shot models (WiSE-FT, 2021)

Practise it with Prism

So each can be changed on its own. If what is said is encoded apart from who says it and how, you can convert one voice into another while keeping the words, make a text-to-speech system read any text in a chosen voice or style, and make recognition less sensitive to the speaker. AutoVC did this with an autoencoder whose bottleneck is just small enough to squeeze speaker information out of the content code, and Tacotron’s global style tokens learn style embeddings with no style labels at all.

AutoVC was the first to perform zero-shot voice conversion, to and from voices it never heard in training.

Qian et al., AutoVC: Zero-Shot Voice Style Transfer with Only Autoencoder Loss (2019)Wang et al., Style Tokens: Unsupervised Style Modeling, Control and Transfer in End-to-End Speech Synthesis (2018)

Practise it with Prism