Foundational ML Models and Sklearn

1. Start with the output you need

Regression predicts a numeric quantity, such as duration. Classification predicts a label or class probabilities, such as whether an event occurs. Clustering groups observations without supervised target labels. A clustering label is not automatically a known class label.

Linear regression models a numeric target as a weighted sum plus a bias. Binary logistic regression instead maps a weighted sum through a sigmoid to estimate the probability of class 1. Despite its name, LogisticRegression is a classifier. A threshold turns its probability estimate into a class decision.

Linear regression: prediction = w · x + b
Logistic regression: P(y = 1 | x) = 1 / (1 + exp(-(w · x + b)))

Documentation: Linear and logistic models

2. Know what each model can express

A model is a set of possible prediction rules plus a procedure for fitting one of them. Simpler rules may miss nonlinear structure; flexible rules may fit noise. Start with a small number of models whose assumptions you can explain. These are starting points, not a universal ranking.

Documentation: Choosing an estimator

3. Use the estimator interface consistently

fit(X_train, y_train) learns from training examples. predict(X_new) returns labels or numeric predictions. Classifiers that implement predict_proba return one column per class, in classes_ order. That order must be checked before interpreting a probability column.

A DummyClassifier ignores the inputs and follows a simple rule, such as always choosing the most frequent training class. Beating that baseline is a minimum useful comparison. It can already achieve high accuracy when one class dominates. Each candidate's preprocessing belongs inside its pipeline.

Documentation: DummyClassifier

4. Compare models on the same validation folds

First reserve a test set. Then divide the remaining training data into folds. For each candidate and fold, fit on the other folds and score on the held-out fold. Reusing the same folds makes the comparisons easier to interpret. Pick the model using mean validation performance, refit it on all training data, and evaluate it on the reserved test set.

The fold standard deviation describes variation across those folds; it is not automatically a confidence interval. Training scores can reveal a large training/validation gap, but a good training score alone says little about generalization. Model selection uses validation scores, never the final test score.

Documentation: Cross-validation and model evaluation

5. Read errors as well as scores

For a chosen positive class, a true positive is a correctly detected positive; a false positive is a negative incorrectly flagged positive. A false negative is a missed positive. Precision asks how many positive predictions are correct. Recall asks how many actual positives were found.

Suppose TP=8, FP=2, FN=4, and TN=6. Precision is 8/10=0.80, recall is 8/12≈0.67, and accuracy is 14/20=0.70. Choose metrics according to the mistakes that matter. Balanced accuracy averages recall across classes. ROC AUC evaluates ranking from scores across thresholds; it is not the accuracy at a particular threshold.

Documentation: Metrics and scoring

Complete example: Compare classifiers on two curved classes

The two-moons dataset has two numeric features and two curved classes. Compare a dummy baseline, logistic regression, a tree, and a random forest using five-fold cross-validation. Only the selected model sees the final test evaluation.

Install the packages in your Python environment, then download and run the script. The example creates its data locally.

python -m pip install numpy scikit-learn
python ml-week-1.py

import numpy as np
from sklearn.base import clone
from sklearn.datasets import make_moons
from sklearn.dummy import DummyClassifier
from sklearn.ensemble import RandomForestClassifier
from sklearn.linear_model import LogisticRegression
from sklearn.metrics import (balanced_accuracy_score, confusion_matrix,
                             f1_score, roc_auc_score)
from sklearn.model_selection import (StratifiedKFold, cross_validate,
                                     train_test_split)
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import StandardScaler
from sklearn.tree import DecisionTreeClassifier

X, y = make_moons(n_samples=1000, noise=0.25, random_state=7)
X_train, X_test, y_train, y_test = train_test_split(
    X, y, test_size=0.2, random_state=42, stratify=y
)
cv = StratifiedKFold(n_splits=5, shuffle=True, random_state=42)
folds = list(cv.split(X_train, y_train))
models = {
    "dummy": DummyClassifier(strategy="most_frequent"),
    "logistic": make_pipeline(StandardScaler(),
                              LogisticRegression(max_iter=1000)),
    "tree": DecisionTreeClassifier(max_depth=4, random_state=42),
    "forest": RandomForestClassifier(n_estimators=100,
                                    min_samples_leaf=3,
                                    random_state=42, n_jobs=1),
}
means = {}
for name, estimator in models.items():
    scores = cross_validate(estimator, X_train, y_train, cv=folds,
                            scoring="balanced_accuracy",
                            return_train_score=True)
    means[name] = scores["test_score"].mean()
    # Here test_score means each CV validation fold, not X_test.
    print(name, "train", round(scores["train_score"].mean(), 3),
          "validation", round(means[name], 3),
          "fold std", round(scores["test_score"].std(), 3))

winner_name = max(means, key=means.get)
winner = clone(models[winner_name]).fit(X_train, y_train)
prediction = winner.predict(X_test)
positive_column = np.flatnonzero(winner.classes_ == 1)[0]
probability = winner.predict_proba(X_test)[:, positive_column]
assert prediction.shape == y_test.shape
assert np.isfinite(probability).all()
print("Selected:", winner_name)
print("Test balanced accuracy:", balanced_accuracy_score(y_test, prediction))
print("Test F1:", f1_score(y_test, prediction))
print("Test ROC AUC:", roc_auc_score(y_test, probability))
print("Confusion matrix; rows=true, columns=predicted, classes=[0, 1]:")
print(confusion_matrix(y_test, prediction, labels=[0, 1]))

What to look for

• The dummy classifier's balanced accuracy is 0.5 when both classes are present.
• The selected candidate has the highest mean validation score among these choices, not a guaranteed advantage on every future dataset.
• The confusion matrix counts sum to the 200 test observations. The held-out evaluation happens after model selection.

Practice and completion checks

1. Use only cross-validation on X_train to compare tree depths 1, 4, and unrestricted. Record training and validation scores.

   You are done when: You can explain the gap for each depth without choosing settings from X_test.

2. Calculate accuracy, precision, recall, and F1 by hand for TP=8, FP=2, FN=4, TN=6.

   You are done when: You obtain 0.70, 0.80, approximately 0.667, and approximately 0.727, respectively.

3. Describe a regression version of the workflow with DummyRegressor and a numeric target.

   You are done when: Your baseline uses training data, your comparison uses validation MAE or RMSE, and your test set remains reserved.

Check your understanding

Answer each question first, then expand it to compare your reasoning.