Machine learning interview questions, answered the way you’d say them.

Content-based filtering recommends items whose features resemble what the user already liked, such as the same brand or category, and needs no data about other users. Collaborative filtering learns from everyone’s behaviour, using similarities between users and items at once, so it can surface unexpected items that similar users liked. Matrix factorisation, the classic collaborative method, learns user and item embeddings whose dot products approximate the user-item feedback matrix.

With clicks or purchases instead of ratings, missing entries aren’t confirmed negatives; weighted matrix factorisation counts them as zeros with their own, tunable weight.

Google for Developers, Recommendation systems: Content-based filteringGoogle for Developers, Recommendation systems: Collaborative filteringGoogle for Developers, Recommendation systems: Matrix factorization

Question adapted from data-science-interviews by Alexey Grigorev and contributors, CC BY 4.0; reworded, with our own answer.

Practise it with Prism

Machine learning means a model picks up patterns from data rather than running rules someone wrote by hand. In supervised learning it trains on labelled examples, such as customers marked churned or not. Unsupervised learning has no labels and looks for structure, like customer segments, while reinforcement learning works out which actions earn rewards, the way a game-playing agent does.

The full answer, in the guidePractise it with Prism

Classification outputs a category, for example whether a customer will churn or whether a UPI payment is fraud. Regression outputs a number, such as next month’s sales or the price of a flat. Some problems fit both, since you could ask whether sales will drop or how far they’ll drop, so I’d first check how the answer will be used.

The full answer, in the guidePractise it with Prism

So that the final score is an honest estimate. The model learns from the training data, I tune and compare models on a validation set or by cross-validation, and the test set is used once, to check the final choice; any decision made on it would spoil that. By default, train_test_split keeps back 25%, and stratify preserves the class proportions.

scikit-learn: train_test_split

The full answer, in the guidePractise it with Prism

Overfitting is when the model memorises noise, so its training score is far higher than its validation score. Underfitting is when it never captures the pattern, so both scores stay poor. For overfitting I’d add data, simplify the model, regularise or stop training early; for underfitting I’d improve the features or move to a more flexible model.

The full answer, in the guideTry it in the overfitting playgroundPractise it with Prism

Bias is error from mistaken assumptions, like fitting a straight line to curved data. Variance is error from leaning too hard on one training sample, the way a deep tree comes out different every time the sample changes. Making a model more flexible pushes bias down and variance up, so the target is the lowest error on unseen data, which I find through validation.

The full answer, in the guideTry it in the overfitting playgroundPractise it with Prism

Four things: the target relates linearly to the features, the errors are independent, their variance is constant, which is homoscedasticity, and there’s no perfect multicollinearity. If you want p-values and confidence intervals, the errors should also be roughly normal. Residual plots are the check: a funnel shape signals changing variance, and a curved pattern says the link isn’t linear.

The full answer, in the guidePractise it with Prism

Because underneath it’s a regression: a linear model of the features predicts the positive class’s log-odds. The sigmoid maps that to a probability, and only the threshold applied afterwards makes it a classifier. Training minimises log loss, and scikit-learn adds L2 regularisation unless told otherwise.

scikit-learn: LogisticRegression

The full answer, in the guidePractise it with Prism

It’s a table of actual classes against predicted ones, with four cells: true positives, false positives or false alarms, false negatives or misses, and true negatives. Precision, recall and F1 are all computed from those counts. Worth knowing: scikit-learn puts the actual classes in the rows and the predictions in the columns.

For a binary model, confusion_matrix(...).ravel() returns tn, fp, fn, tp, in that order.

scikit-learn: confusion_matrix

The full answer, in the guideTry it in the confusion matrixPractise it with Prism

Precision, TP / (TP + FP), asks how many of the cases I flagged were real; recall, TP / (TP + FN), asks how many of the real cases I caught. F1 is the harmonic mean of the two. I wouldn’t lean on accuracy when a class is rare: with 2% of payments fraudulent, a model that never flags anything scores 98% and is worthless.

scikit-learn: Metrics and scoring

The full answer, in the guideTry it in the confusion matrixPractise it with Prism

Anything that works with distances or gradients is sensitive to units, so k-means, KNN, PCA, SVM and regularised linear models need scaling; otherwise a monthly charge in rupees drowns out tenure in months. Tree models, from a single tree to random forests and boosting, only compare a feature with a threshold, so scaling doesn’t affect them. Whatever the model, the scaler is fitted on the training data alone.

The full answer, in the guidePractise it with Prism

One-hot gives each category its own 0/1 column, which suits categories with no order, like a plan or a city tier. Label or ordinal encoding turns categories into integers; that’s fine when they’re ordered or the model is a tree, but a linear model will read an order that isn’t there. With one-hot I set handle_unknown="ignore", so a category first seen in production becomes all zeros rather than an error.

scikit-learn: OneHotEncoder

The full answer, in the guidePractise it with Prism

I start by asking why the values are missing, because a blank can carry meaning: an empty income field may mean the person chose not to disclose it. Then I choose between dropping, imputing, with the median for numbers and the most frequent value for categories, or adding a flag that marks the gap. Some models cope directly: scikit-learn 1.9’s HistGradientBoosting and RandomForest, for example, accept NaN natively.

scikit-learn: HistGradientBoostingClassifierscikit-learn: RandomForestClassifier

The full answer, in the guidePractise it with Prism

The ROC curve plots the true positive rate, or recall, against the false positive rate at every possible threshold, so it shows the trade-off without committing to one cut-off. AUC, the area under it, is the probability that the model scores a randomly chosen positive above a randomly chosen negative: 0.5 is random guessing and 1.0 is perfect. It measures ranking, not calibration, and with rare positives it can look strong while precision is poor, so check the precision–recall curve too.

AUC depends only on the order of the scores, so doubling every score leaves it unchanged; it says nothing about whether 0.8 means 80%.

Google Machine Learning Crash Course: ROC and AUCSaito and Rehmsmeier, The Precision-Recall Plot Is More Informative than the ROC Plot When Evaluating Binary Classifiers on Imbalanced Datasets (PLOS ONE, 2015)

Try it in the confusion matrixPractise it with Prism

Feature selection keeps the useful inputs and drops the rest, to improve predictions, make models faster and cheaper, and make them easier to understand. It must be fitted on training data only: selecting on the full dataset lets the test rows influence the choice, which is leakage. In scikit-learn’s own demonstration, 25 features picked from 10,000 random ones scored 0.76 accuracy on pure noise, against the expected 0.5 when selection ran inside a pipeline.

from sklearn.ensemble import HistGradientBoostingClassifier
from sklearn.feature_selection import SelectKBest
from sklearn.model_selection import cross_val_score
from sklearn.pipeline import make_pipeline

pipe = make_pipeline(SelectKBest(k=25), HistGradientBoostingClassifier(random_state=1))
print(cross_val_score(pipe, X, y).mean())   # selection is refitted inside every fold

The same rule applies to scaling, imputing and PCA: fit them on the training folds only.

scikit-learn: Common pitfalls and recommended practices (data leakage)Guyon and Elisseeff (2003), An introduction to variable and feature selection, JMLR

Question adapted from data-science-interviews by Alexey Grigorev and contributors, CC BY 4.0; reworded, with our own answer.

Practise it with Prism

Filters score features before modelling, independently of the model: a variance threshold, or chi-squared, F-tests or mutual information with SelectKBest. Wrappers search subsets by training the model itself, as recursive feature elimination and forward or backward selection do, which costs many fits. Embedded methods select while training, as L1 regularisation does by setting coefficients to zero; PCA isn’t selection, because it builds new features from all the old ones.

from sklearn.feature_selection import RFECV, SelectFromModel
from sklearn.linear_model import LogisticRegression
from sklearn.svm import LinearSVC

embedded = SelectFromModel(LinearSVC(C=0.01, penalty="l1", dual=False))  # non-zero weights
wrapper = RFECV(LogisticRegression(max_iter=1000), cv=5)  # number of features chosen by CV

Say when you’d use each: filters for a quick first cut on thousands of columns, embedded methods with linear models, wrappers when fits are cheap.

Guyon and Elisseeff (2003), An introduction to variable and feature selection, JMLRscikit-learn: 1.13 Feature selection

Question adapted from data-science-interviews by Alexey Grigorev and contributors, CC BY 4.0; reworded, with our own answer.

Practise it with Prism

Choose DBSCAN when clusters have irregular shapes, you don’t know how many there are, or you want outliers flagged: it grows clusters from dense core points and labels isolated points as noise (-1). k-means needs k up front and assumes convex, similar-sized clusters, but it’s simple and scales to very large data. DBSCAN depends heavily on eps, the neighbourhood radius, and struggles when clusters differ in density, which HDBSCAN handles.

import numpy as np
from sklearn.cluster import DBSCAN

coords = np.radians(df[["lat", "lon"]].to_numpy())   # haversine needs radians, latitude first
db = DBSCAN(eps=0.5 / 6371, min_samples=20, metric="haversine").fit(coords)  # 0.5 km radius
print((db.labels_ == -1).mean())                     # share of delivery points marked as noise

Scale features first unless they share a unit, as coordinates do, because eps is a single distance across all of them.

scikit-learn: 2.3 Clustering (k-means and DBSCAN)

Question adapted from data-science-interviews by Alexey Grigorev and contributors, CC BY 4.0; reworded, with our own answer.

Practise it with Prism

As you add features, the space grows exponentially, so a fixed amount of data covers it ever more thinly and distances lose contrast: under broad conditions the nearest point ends up almost as far away as the farthest. One study found this can happen with as few as 10 to 15 dimensions. It hurts distance-based methods most, such as KNN, k-means and similarity search, so select or reduce features, regularise, or add data.

A quick illustration: with 10 values per feature, covering every combination takes 10 cells for one feature and 10 billion for ten.

Beyer et al. (1999), When is “nearest neighbor” meaningful?, ICDTscikit-learn: 1.6 Nearest neighbors

Question adapted from data-science-interviews by Alexey Grigorev and contributors, CC BY 4.0; reworded, with our own answer.

Practise it with Prism

Precision@k is the share of the top k results that are relevant, ignoring their order within k. MAP averages the precision at each rank where a relevant item appears, so relevant items near the top count more; it suits yes-or-no relevance. NDCG handles graded relevance, discounts each gain by the log of its position and divides by the ideal ranking’s score, giving 0 to 1; MRR looks only at the first relevant result.

Search "kurta": relevant items at ranks 1, 3 and 4 of the top 5 (3 relevant in all)
P@5 = 3/5 = 0.60
AP  = (1/1 + 2/3 + 3/4) / 3 = 0.81     MAP = the mean AP over all queries
RR  = 1/1 = 1.00                        MRR = the mean RR over all queries

Set k to what users actually see: if the app shows 5 results on the first screen, measure at 5.

TensorFlow Ranking: PrecisionMetric (precision@k)TensorFlow Ranking: MeanAveragePrecisionMetricTensorFlow Ranking: MRRMetricscikit-learn: ndcg_score

Question adapted from data-science-interviews by Alexey Grigorev and contributors, CC BY 4.0; reworded, with our own answer.

Practise it with Prism

Start simple and make every new model beat them: the most popular items overall and by segment, such as city or category; the user’s own recent and repeat items; and item-to-item similarity, such as products often bought together. Evaluate them offline on a split by time, with precision@k or NDCG. Simple baselines are hard to beat: a 2019 study could reproduce only 7 of 18 recent neural recommenders, and simple nearest-neighbour or graph methods often beat 6 of those 7.

Check how much of your traffic is repeat buying: where it’s high, “buy again” from the user’s own history is a demanding baseline.

Ferrari Dacrema, Cremonesi and Jannach (2019), Are we really making much progress?, RecSys

Question adapted from data-science-interviews by Alexey Grigorev and contributors, CC BY 4.0; reworded, with our own answer.

Practise it with Prism

Collaborative filtering can’t recommend what it has no interactions for: a new user has no history, and a new item has no learned embedding. For new items, use content features or an embedding averaged from similar items, such as others in the same category, and once a few interactions arrive, solve for the item’s embedding without retraining everything. For new users, start with popular items for their city or segment, ask a few onboarding questions, and personalise as clicks arrive.

Hybrid recommenders blend content and collaborative signals, so new items can be shown before they have any interactions.

Google for Developers, Recommendation systems: Collaborative filtering advantages and disadvantagesGoogle for Developers, Recommendation systems: Content-based filtering

Question adapted from data-science-interviews by Alexey Grigorev and contributors, CC BY 4.0; reworded, with our own answer.

Practise it with Prism

In k-fold cross-validation I train the model k times, keeping a different fold aside for scoring each time, and average the results, so the estimate doesn’t hang on one fortunate split. Stratified folds preserve the class balance in every fold. For classifiers, cross_val_score stratifies automatically when cv is given as a number, and I’d report the mean and the standard deviation of the scores.

scikit-learn: cross_val_score

The full answer, in the guidePractise it with Prism

Data leakage is when training uses information the model won’t have at prediction time, so validation scores look better than reality. Typical examples are scaling or imputing the whole dataset before the split, a feature recorded after the outcome, such as “refund issued” in a returns model, and random splits on data ordered in time. The scikit-learn rule is simple: never call fit on test data.

scikit-learn: Common pitfalls and recommended practices

The full answer, in the guidePractise it with Prism

Because it rules out leakage and mismatches between training and serving. A Pipeline fits the imputer, scaler and encoder on training data alone, refits them within each cross-validation fold, and applies exactly the same steps at prediction time. Inside it, a ColumnTransformer routes numeric and categorical columns to their own steps.

scikit-learn: Common pitfalls and recommended practices

The full answer, in the guidePractise it with Prism

Both add a penalty on large coefficients to rein in overfitting. L2, as in Ridge, shrinks every coefficient gradually, while L1, as in Lasso, can drive some to exactly zero, so it doubles as feature selection; Elastic Net blends the two. In scikit-learn’s LogisticRegression, C is the inverse of the regularisation strength, and since 1.8 the type is chosen with l1_ratio: 0 for L2, 1 for L1.

Older tutorials still pass penalty to LogisticRegression; scikit-learn 1.8 deprecated that argument in favour of l1_ratio.

scikit-learn: LogisticRegression

The full answer, in the guideTry it in the overfitting playgroundPractise it with Prism

At every node it searches over features and thresholds and greedily takes the split that leaves the child nodes as pure as possible, measured by Gini impurity, which is the scikit-learn default, or by entropy. Left without a depth limit, it keeps splitting until every leaf is pure, which means memorising the training data. So I’d cap it with max_depth or min_samples_leaf.

scikit-learn: DecisionTreeClassifier

The full answer, in the guidePractise it with Prism

Bagging fits many models independently, each on its own bootstrap sample, and averages them, which mostly reduces variance; a random forest is the standard example. Boosting fits models one after another, with each new one fixing what the earlier ones got wrong, which mostly reduces bias, as in gradient boosting and XGBoost.

Practise saying the difference in two sentences.

The full answer, in the guidePractise it with Prism

By averaging many trees that are deliberately different, so their individual errors largely cancel out. Each tree trains on a bootstrap sample of the rows, and each split considers only a random subset of the features, sized by default at the square root of their count in scikit-learn’s classifier. The default is 100 trees; extra trees mainly cost time, not accuracy.

scikit-learn: RandomForestClassifier

The full answer, in the guidePractise it with Prism

Trees are added one at a time, each fitting the remaining errors, the gradient of the loss, and each scaled down by the learning rate; the smaller the rate, the more trees you need. The key settings are that rate, how many trees, how deep they go, and early stopping. In scikit-learn, HistGradientBoosting is far quicker once you reach 10,000 rows, handles NaN itself and turns on early stopping by default above 10,000 samples, while XGBoost also learns which way missing values should go.

scikit-learn: HistGradientBoostingClassifierXGBoost: Frequently Asked Questions

The full answer, in the guidePractise it with Prism

KNN finds the k training points nearest to a new example and lets them decide: a vote for a class, an average for a number. By default, scikit-learn uses k = 5 and Euclidean distance. A small k overfits and a large k underfits, so I pick k by cross-validation after scaling, and I’d warn that it struggles when there are many features, which is the curse of dimensionality, and predicts slowly on big data.

scikit-learn: KNeighborsClassifier

The full answer, in the guidePractise it with Prism

An SVM draws the boundary that leaves the widest possible margin between the classes, and the points sitting on that margin are the support vectors. A kernel, RBF unless you choose otherwise, lets the boundary curve; C balances a wider margin against mistakes on training data; gamma controls how far a single point’s influence extends. Training time rises at least quadratically with the number of rows, so for big data I’d use LinearSVC.

Older tutorials set SVC’s probability option, which scikit-learn 1.9 deprecated.

scikit-learn: SVC

The full answer, in the guidePractise it with Prism

Because it uses Bayes’ theorem under a strong simplifying assumption: given the class, each feature is treated as independent of every other feature, which real data rarely satisfies. Even so, it does well on text, with spam filtering and document classification as the classic uses, and it’s extremely fast to run.

scikit-learn: Naive Bayes

The full answer, in the guidePractise it with Prism

For rare positives I’d lean on the precision–recall curve. ROC AUC measures ranking over every threshold, but when positives are scarce the false-positive rate hardly moves, so it can stay impressive even when precision is weak. Average precision, which summarises the precision–recall curve, is about the positive class, and a random model scores the positive share: with fraud at 2%, 0.30 is a strong result.

scikit-learn: Metrics and scoring

The full answer, in the guideTry it in the confusion matrixPractise it with Prism

First I pick a metric that reflects the rare class. Then I try three options, one after another: class_weight="balanced", which sets each class’s weight to n_samples / (n_classes × class count); a tuned decision threshold; and resampling, such as SMOTE, which makes synthetic points along the line from a minority sample to one of its neighbours. Resampling touches only the training folds and sits inside an imbalanced-learn pipeline, with SMOTE-NC if some columns are categorical.

scikit-learn: LogisticRegressionimbalanced-learn: Over-samplingimbalanced-learn: Common pitfalls and recommended practices

The full answer, in the guidePractise it with Prism

MAE averages the absolute errors and stays in the target’s own units. RMSE squares the errors before averaging, so large misses count for more. R² is the share of variance the model explains: 1.0 is perfect, a model that always predicts the mean gets 0, and anything worse is negative. I’d avoid MAPE when actual values can be close to zero, because it breaks down there.

scikit-learn: r2_score

The full answer, in the guidePractise it with Prism

Grid search fits every combination of the values you list, while random search tries a set number of sampled combinations. When only a few hyperparameters really matter, random search often lands on good settings with far fewer fits. Either way, the search cross-validates on the training data, and the test set is kept for one final check at the end.

The full answer, in the guidePractise it with Prism

Each label is a Bernoulli outcome with probability p = σ(wᵀx), so its likelihood is p^y (1 − p)^(1 − y). Taking logs and negating turns maximising the likelihood of all the rows into minimising −Σ [y log p + (1 − y) log(1 − p)], the binary cross-entropy or log loss. Adding a prior gives MAP estimation: a zero-mean Gaussian prior on the weights adds a term proportional to ‖w‖², which is exactly an L2 penalty, and a tighter prior means a stronger penalty.

A common follow-up is the gradient: for one example it’s (p − y)x, the same form as linear regression’s.

scikit-learn: Logistic regressionscikit-learn: Bayesian regression (L2 as a Gaussian prior)

Practise it with Prism

Score both directions, image-to-text and text-to-image, on held-out pairs. Recall@K is the share of queries whose correct match appears in the top K, usually reported at 1, 5 and 10, and median rank is where the correct match typically lands, so lower is better. Each query has one match among thousands of candidates, so if you treat pairs as yes-or-no decisions, positives are rare and PR-AUC tells you more than ROC-AUC. If scores drive a threshold, such as auto-tagging, check calibration too.

Check for near-duplicate images between the training and test sets, or recall will look better than it is.

Karpathy and Fei-Fei, Deep Visual-Semantic Alignments for Generating Image Descriptions (2015)Saito and Rehmsmeier, The Precision-Recall Plot Is More Informative than the ROC Plot When Evaluating Binary Classifiers on Imbalanced Datasets (PLOS ONE, 2015)scikit-learn: Probability calibration

Practise it with Prism

Without labels, use internal measures. Inertia, the within-cluster sum of squares, keeps falling as k grows, so look for an elbow rather than the minimum. The silhouette coefficient runs from −1 to +1, higher meaning dense, well-separated clusters; Calinski–Harabasz is better when higher, and Davies–Bouldin when lower, with zero the best. If true labels exist, compare against them with the adjusted Rand index or adjusted mutual information. Finally, check the clusters are stable across random seeds and useful to the business.

Inertia isn’t normalised, so compare it only between runs on the same data.

scikit-learn: Clustering performance evaluation

Practise it with Prism

A convex loss is bowl-shaped: the straight line between any two points on its graph lies on or above it, so every local minimum is a global minimum, and gradient descent with a suitable step size reaches the best solution. Linear regression’s squared error and logistic regression’s log loss are convex in the weights. A neural network’s loss is non-convex, so where you end up depends on initialisation, learning rate and optimiser; Dauphin and colleagues argued that in high dimensions, saddle points, not poor local minima, are the main obstacle.

Convexity is about the loss as a function of the weights: a linear model on non-linear features of the input still has a convex loss.

Dive into Deep Learning: ConvexityDauphin et al., Identifying and attacking the saddle point problem in high-dimensional non-convex optimization (2014)

Practise it with Prism

Picture minimising the loss within a budget on the weights. The L1 budget, |w₁| + |w₂| ≤ t, is a diamond with corners on the axes, so the loss contours usually first touch it at a corner, where some weights are exactly zero; the L2 budget is a circle, with no corners to catch on. In gradient terms, L1 pulls every weight towards zero with the same force, λ, however small the weight, so small weights reach zero and stay there, while L2’s pull, 2λw, fades as the weight shrinks.

Tibshirani’s original lasso paper made the same point: the constraint tends to produce some coefficients that are exactly zero, which makes the model easier to interpret.

Tibshirani, Regression Shrinkage and Selection via the Lasso (JRSS B, 1996)

Practise it with Prism

MLE picks the parameters that make the observed data most probable, maximising P(data | θ). MAP maximises the posterior, proportional to P(data | θ) × P(θ), so it adds a prior: with a flat prior the two agree, and as data grows the likelihood dominates. For 3 heads in 3 tosses, MLE says p = 1, while MAP with a Beta(2, 2) prior gives (3 + 1) / (3 + 2) = 0.8. In regression, a zero-mean Gaussian prior on the weights turns MAP into ridge’s L2 penalty.

MAP still returns a single point, the peak of the posterior; fully Bayesian methods keep the whole distribution.

Dive into Deep Learning: Maximum likelihoodscikit-learn: Bayesian regression (L2 as a Gaussian prior)

Practise it with Prism

Cross-entropy, H(p, q) = −Σ p(x) log q(x), measures how well the model’s distribution q matches the true distribution p. It splits into H(p) + KL(p ‖ q): the entropy of the true labels, which the model can’t change, plus the KL divergence. So minimising cross-entropy is the same as minimising KL, and both are the same as maximising the likelihood. With one-hot labels, H(p) = 0, so the loss is simply −log of the probability the model gives the correct class.

PyTorch’s CrossEntropyLoss also accepts class probabilities as targets, such as smoothed labels or a teacher’s outputs; then H(p) is no longer zero.

Dive into Deep Learning: Information theoryPyTorch: torch.nn.CrossEntropyLoss

Practise it with Prism

High bias, within reason. A point estimator is any function of the data used to estimate a quantity, and its expected squared error splits into bias squared plus variance, plus noise nothing can remove. With few samples, variance dominates: a flexible model fits the noise and changes a lot from one sample to the next, so a simpler model, stronger regularisation or prior knowledge, such as a pretrained model, usually predicts better. As data grows, variance shrinks and more flexible models start to pay off.

Expect the follow-up “how would you know?”: compare training and validation scores, and plot a learning curve to see whether more data would help.

Dive into Deep Learning: Statistics (estimators, bias and variance)

Practise it with Prism

Exploiting shows the option that looks best now; exploring tries others to learn whether they’re better. Epsilon-greedy explores at random with probability ε, so it keeps spending traffic on options already known to be poor and treats uncertain and well-understood options alike. Thompson sampling keeps a posterior over each option’s reward, such as a Beta distribution for a click rate, draws one sample per option and plays the highest, so it explores in proportion to the chance an option is best. Chapelle and Li found it highly competitive in practice.

For click rates, the Beta update is just counting: add each click to α and each non-click to β.

Russo et al., A Tutorial on Thompson Sampling (2017)Chapelle and Li, An Empirical Evaluation of Thompson Sampling (NeurIPS 2011)

Practise it with Prism

A calibrated model’s probabilities mean what they say: of the cases it scores about 0.8, about 80% turn out positive. You check with a calibration curve, the observed share of positives against the mean predicted probability in each bin. Platt scaling fits a sigmoid to the model’s scores on data it wasn’t trained on; scikit-learn’s CalibratedClassifierCV does this with method="sigmoid". Isotonic regression can fix any monotonic distortion but overfits small sets; scikit-learn says it does as well or better with more than about 1,000 samples.

A lower Brier score doesn’t prove better calibration, because it also rewards good ranking; look at the calibration curve.

scikit-learn: Probability calibration

Practise it with Prism

Latent semantic indexing, or LSA, applies a truncated SVD to the term–document matrix of counts or TF-IDF weights and keeps the top k components, so words used in similar documents, such as car and automobile, end up close together. LDA is probabilistic: each document is a mixture of topics, and each topic a distribution over words. Clustering, such as k-means on TF-IDF vectors or embeddings, puts each document in exactly one group, whereas a topic model can call a document 70% cricket and 30% business.

In scikit-learn, TruncatedSVD applied to a TF-IDF matrix is exactly LSA.

scikit-learn: TruncatedSVD (latent semantic analysis)Blei, Ng and Jordan, Latent Dirichlet Allocation (JMLR, 2003)

Practise it with Prism

Validation must copy how the model will be used. For a classifier, use stratified k-fold with a metric that fits the costs, such as PR-AUC for rare positives; for a ranker, split by user or query, so no group appears on both sides, and score whole lists with NDCG@k; for a forecaster, train on the past and test on the period that follows, against a naive baseline. Regularisation follows the model: penalties for linear models, depth and leaf limits for trees, dropout, weight decay and early stopping for neural networks.

For neural networks, keep the validation set used for early stopping separate from the final test set.

scikit-learn: Cross-validation (StratifiedKFold, GroupKFold, TimeSeriesSplit)

Practise it with Prism

Score both on the same fresh data with the metric that matters, and check that the gap is bigger than noise: a bootstrap confidence interval on the difference, or a paired test on the same examples, such as McNemar’s test for classifiers. For overfitting, compare each model’s score on its training data with its score on the fresh data: a large drop means it memorised its training set, while poor scores on both mean underfitting. With little fresh data, repeated or nested cross-validation gives steadier estimates.

Make sure the fresh data looks like production, such as the latest time period, or you may pick the model that wins on the wrong distribution.

Raschka, Model Evaluation, Model Selection, and Algorithm Selection in Machine Learning (2018)

Practise it with Prism

First check that the probabilities are calibrated, because cut-points on uncalibrated scores mean little. Then choose four cut-points: equal widths of 0.2 are simplest, quantiles give each grade the same share of cases, and business-driven cut-points can guarantee, say, that grade 4 has at least 90% precision. If you have true graded labels, model the order directly instead: an ordinal regression, such as statsmodels’ OrderedModel, learns one latent score with k − 1 thresholds, so the grades stay ordered.

Check the grades on held-out data: the observed share of positives should rise steadily from grade 0 to grade 4.

scikit-learn: Probability calibrationstatsmodels: Ordinal regression (OrderedModel)

Practise it with Prism

Like gradient boosting, it adds trees one at a time, each correcting the current predictions. XGBoost minimises a regularised objective, the loss plus γ for every leaf and ½λ times the squared leaf weights, and approximates the loss to second order with each point’s gradient and Hessian. That gives each leaf a closed-form weight, −G / (H + λ), and a gain formula for splits, which prunes splits gaining less than γ. It also learns a default direction for missing values and finds splits with quantile sketches.

Know the parameters this maps to: gamma, reg_lambda for λ, max_depth, learning_rate and subsample.

XGBoost: Introduction to Boosted TreesChen and Guestrin, XGBoost: A Scalable Tree Boosting System (2016)

Practise it with Prism

Start with cheap filters: drop constant and near-constant features with VarianceThreshold, and duplicates. For multicollinearity, check correlations and variance inflation factors, where a VIF above 5 is one common warning sign, and keep one feature from each tightly correlated group. Then score the rest against the target, with an F-test or mutual information, and use embedded selection such as L1 or tree importances; with 4,000 features, avoid wrappers that refit the model for every subset. If interpretability doesn’t matter, PCA can compress what remains.

Do every selection step inside a pipeline fitted on the training folds only, or the selection leaks information from the test data.

scikit-learn: Feature selectionstatsmodels: variance_inflation_factor

Practise it with Prism

Numeric, continuous features suit Gaussian Naive Bayes, which models each feature within each class as a normal distribution. Categorical features suit Categorical Naive Bayes, which learns a frequency table per feature and class, with categories encoded as integers. For word counts use Multinomial, for binary features Bernoulli, and for imbalanced text Complement. Keep smoothing on, with alpha = 1 being Laplace smoothing, so a category never seen with a class doesn’t zero out the whole probability.

scikit-learn warns that Naive Bayes is a decent classifier but a poor probability estimator, so don’t take its predict_proba values at face value.

scikit-learn: Naive Bayes

Practise it with Prism

It tries every combination in the grid, scores each with cross-validation, by default 5-fold and stratified for classifiers, then refits the best setting on all the training data. Watch the cost: four values for each of three parameters with 5 folds means 320 fits. Put preprocessing inside a Pipeline, so scaling is learned within each fold, and set scoring to the metric you care about, since the default is the estimator’s own score. The best CV score is optimistic, so report a held-out test score or use nested cross-validation.

Set n_jobs=-1 to run the fits in parallel; by default they run one at a time.

scikit-learn: GridSearchCVscikit-learn: Nested versus non-nested cross-validation

Practise it with Prism

Hyperparameters are set before training, not learned, and decide how flexible the model is and how it learns, moving it between underfitting and overfitting. Raising a tree’s max_depth or lowering min_samples_leaf makes it more complex; stronger regularisation, such as a lower C in logistic regression, makes it simpler; a smaller k in KNN follows the training data more closely. A learning rate that’s too high makes training diverge, and one too low makes it crawl. Tune them with cross-validation on training data, never on the test set.

More trees in a random forest mostly cost time rather than accuracy, so the number of trees is rarely the first setting to tune.

scikit-learn: Tuning the hyper-parameters of an estimatorscikit-learn: LogisticRegression (C)

Practise it with Prism

Treat it as learning to rank: each training row is a query and an item, with features such as text match, price and past clicks, a relevance label, and a query ID that groups the rows. Pointwise models score items one at a time, pairwise models learn which of two items should rank higher, and listwise models optimise the whole list. LambdaMART, the boosted-tree version of LambdaRank, targets list metrics such as NDCG; XGBoost’s rank:ndcg and LightGBM’s lambdarank objectives implement it.

from xgboost import XGBRanker

ranker = XGBRanker(objective="rank:ndcg", n_estimators=300, learning_rate=0.1)
ranker.fit(X_train, y_train, qid=qid_train)   # rows sorted by query ID
scores = ranker.predict(X_test)                # sort each query's items by score

Click logs make cheap labels but carry position bias, because top results get more clicks; XGBoost 2.0 added an experimental unbiased LambdaMART for this.

Burges (2010), From RankNet to LambdaRank to LambdaMART: an overview, Microsoft ResearchMicrosoft Research: Learning to rank: from pairwise approach to listwise approach (ICML 2007)XGBoost: Learning to rankLightGBM: Parameters (lambdarank)

Question adapted from data-science-interviews by Alexey Grigorev and contributors, CC BY 4.0; reworded, with our own answer.

Practise it with Prism

Trees make piecewise-constant predictions, so they can’t extrapolate: if sales keep rising, a forest trained on past levels never predicts above the highest values it has seen. Remove the trend first, for example by predicting the change from the previous period, and let the trees learn the rest from lag and calendar features. Alternatively, fit the trend with a linear model and use the trees on what’s left.

df["diff_1"] = df["sales"].diff()                    # change from the previous day
df["target"] = df["sales"].shift(-1) - df["sales"]   # predict tomorrow's change, not its level
# forecast = today's sales + the predicted change

Validate on the most recent period: a trend the model can’t follow shows up there, not in a random split.

scikit-learn: 1.10 Decision treesHyndman and Athanasopoulos, Forecasting: Principles and Practice (3rd ed.): 9.1 Stationarity and differencing

Question adapted from data-science-interviews by Alexey Grigorev and contributors, CC BY 4.0; reworded, with our own answer.

Practise it with Prism

Fix a moderate learning rate and let early stopping on a validation set choose the number of trees. Then tune tree complexity (max_depth or num_leaves, min_child_weight or min_data_in_leaf), then randomness (row and column subsampling), then L1 and L2 penalties. Finally, lower the learning rate and add trees for a last small gain; in LightGBM keep num_leaves below 2^max_depth, because it grows trees leaf-wise.

from xgboost import XGBClassifier

model = XGBClassifier(n_estimators=2000, learning_rate=0.05, max_depth=6,
                      subsample=0.8, colsample_bytree=0.8, early_stopping_rounds=50)
model.fit(X_train, y_train, eval_set=[(X_valid, y_valid)], verbose=False)
print(model.best_iteration)   # trees kept; predict() uses them automatically

Keep the early-stopping set separate from the final test set, or the reported score is optimistic.

XGBoost: Notes on parameter tuningXGBoost: Using the scikit-learn estimator interface (early stopping)LightGBM: Parameters tuning

Question adapted from data-science-interviews by Alexey Grigorev and contributors, CC BY 4.0; reworded, with our own answer.

Practise it with Prism

Bayesian optimisation fits a cheap model of how settings affect the score and uses it to pick the most promising next trial; Optuna’s default sampler, TPE, works this way. Successive halving starts many candidates on a small budget, such as a sample of rows, and gives more to the best each round; scikit-learn’s HalvingRandomSearchCV implements it, still as experimental. Random search remains the baseline these methods must beat.

import optuna
from sklearn.model_selection import cross_val_score
from xgboost import XGBClassifier

def objective(trial):
    model = XGBClassifier(max_depth=trial.suggest_int("max_depth", 3, 10),
                          learning_rate=trial.suggest_float("lr", 0.01, 0.3, log=True))
    return cross_val_score(model, X_train, y_train, cv=5, scoring="roc_auc").mean()
study = optuna.create_study(direction="maximize")   # TPE sampler by default
study.optimize(objective, n_trials=50)

Tune on cross-validation folds of the training data, and touch the test set once, at the end.

Snoek, Larochelle and Adams (2012), Practical Bayesian optimization of machine learning algorithms, NIPSOptuna: TPESamplerOptuna: create_study (the default sampler)scikit-learn: 3.2 Tuning the hyper-parameters of an estimator (successive halving)Bergstra and Bengio (2012), Random search for hyper-parameter optimization, JMLR

Question adapted from data-science-interviews by Alexey Grigorev and contributors, CC BY 4.0; reworded, with our own answer.

Practise it with Prism

I’d trust permutation importance more. A tree’s built-in, impurity-based importance is calculated on training data, and the scikit-learn docs warn that it’s strongly biased towards high-cardinality features like IDs or exact amounts. Permutation importance instead shuffles a single feature’s values in held-out data and measures how far the score drops; the catch is that correlated features share the credit, so both can appear weaker.

scikit-learn: Permutation feature importance

The full answer, in the guidePractise it with Prism

K-means places k centroids, assigns every point to its nearest centroid, moves each centroid to the mean of its points, and repeats, which minimises inertia, the squared distance within clusters. I choose k by looking for the elbow in the inertia curve, checking silhouette scores, and asking what makes business sense. Scale the features first, and remember it expects round clusters of similar size; scikit-learn initialises with k-means++ and has defaulted to n_init="auto" since 1.4.

scikit-learn: KMeans

The full answer, in the guidePractise it with Prism

PCA re-expresses the data on new axes, the principal components, ranked by the variance each one explains, so you can keep only the leading few. It’s useful for visualising data, compressing it, or removing correlated features. The scikit-learn version subtracts the mean but doesn’t rescale, so I standardise beforehand, and setting n_components to a fraction, such as 0.95, keeps that share of the variance.

scikit-learn: PCA

The full answer, in the guidePractise it with Prism

Because a random split lets the model learn from later rows and be tested on earlier ones, which is leakage. I validate forward in time: TimeSeriesSplit fits on the earliest k folds and scores the fold that follows, so each training window contains the previous one, and gap leaves a buffer before each test fold. Every feature also has to come from data that existed before the prediction date.

scikit-learn: TimeSeriesSplit

The full answer, in the guidePractise it with Prism

I’d start from the costs rather than the default 0.5: a missed fraud loses money and trust, while a false alarm turns away a genuine customer. With the business I’d agree a target, say catching 90% of fraud, and on validation data pick the threshold with the highest precision at that recall level. Since scikit-learn 1.5, TunedThresholdClassifierCV can also choose the threshold that maximises a metric of my choice, using cross-validation.

scikit-learn: Tuning the decision threshold for class predictionscikit-learn: TunedThresholdClassifierCV

The full answer, in the guideTry it in the confusion matrixPractise it with Prism

It helps with high-cardinality categories, like pin codes or merchant IDs, where one-hot would create thousands of columns. Target encoding swaps each category for a smoothed average of the target. The risk is leakage, since a row would be encoded using its own label, so scikit-learn’s TargetEncoder, added in 1.3, uses cross-fitting within fit_transform; that’s why fit_transform doesn’t give the same result as fit(X).transform(X).

scikit-learn: TargetEncoder

The full answer, in the guidePractise it with Prism

I’d check five causes in order: leakage, a feature that isn’t available when predicting; training–serving skew, where production builds a feature differently from training; data drift, when inputs shift, say after a new plan launches; concept drift, when behaviour changes, say after a festive season; and changed labels. To catch them, I’d watch the inputs and the live metrics, and retrain on a fixed schedule or when alerts fire.

The full answer, in the guidePractise it with Prism

The logs show rewards only for the actions the old policy chose, so reweight them: inverse propensity scoring multiplies each logged reward by π_new(a | x) / π_old(a | x), unbiased if the old policy’s probabilities were logged and never zero where the new one acts. That’s noisy, so clip or self-normalise them, or use a doubly robust estimator, which combines a reward model with the weights and stays accurate if either is good. To learn a policy, optimise this weighted objective, as counterfactual risk minimisation does.

If the old system was deterministic and never logged probabilities, run a small randomised slice first to create usable logs.

Dudík, Langford and Li, Doubly Robust Policy Evaluation and Learning (2011)Swaminathan and Joachims, Counterfactual Risk Minimization: Learning from Logged Bandit Feedback (2015)

Practise it with Prism

Mostly bias. Round-to-nearest is deterministic, so each weight gets a fixed, systematic error and every prediction shifts the same way each time; nothing about it depends on which training sample you drew. Fewer bits also mean fewer functions the model can represent, which tends to raise bias and can lower variance, a little like a regulariser. Stochastic rounding behaves differently: it rounds up with probability proportional to closeness, so the expected error is zero, removing the bias at the cost of added variance.

Gupta and colleagues trained networks in 16-bit fixed point with stochastic rounding and saw little or no loss in accuracy.

Gupta et al., Deep Learning with Limited Numerical Precision (2015)

Practise it with Prism

A series is stationary when its statistical properties don’t depend on when you observe it; trends, seasonality and changing variance break that. Detect it with plots and unit-root tests: the augmented Dickey–Fuller test’s null is a unit root, while KPSS’s null is stationarity. Remove it by differencing, seasonally if needed, and log-transform to steady the variance. Then read the differenced series’ ACF and PACF: a PACF spike at lag p and none beyond suggests AR(p), an ACF spike at lag q suggests MA(q); compare candidates by AICc.

When both p and q are non-zero the plots stop being clear, so fit a few candidates and let AICc and residual checks decide.

Hyndman and Athanasopoulos, Forecasting: Principles and Practice (3rd ed.): Stationarity and differencingHyndman and Athanasopoulos, Forecasting: Principles and Practice (3rd ed.): Non-seasonal ARIMA modelsstatsmodels: adfuller

Practise it with Prism

Assume f is convex and β-smooth, meaning its gradient is β-Lipschitz, and use step size 1/β. Smoothness gives the descent inequality f(x − ∇f(x)/β) ≤ f(x) − ‖∇f(x)‖² / (2β), so every step lowers the loss unless the gradient is zero. Combining that with convexity and summing over steps gives f(x_t) − f(x*) ≤ 2β‖x_1 − x*‖² / (t − 1), an O(1/t) rate. If f is also α-strongly convex, the distance to the optimum shrinks geometrically, roughly like exp(−t/κ), with κ = β/α.

Without convexity, the same argument only shows the gradient norm goes to zero, so you reach a stationary point, not necessarily the best one.

Bubeck, Convex Optimization: Algorithms and Complexity (2014), sections 3.2 and 3.4

Practise it with Prism