Machine Learning for Researchers: A Practical Introduction
Understand the ML workflow — training, validation, testing — and how it differs from traditional statistical hypothesis testing. No coding required.
You'll learn
The core ML workflow — training, validation, and testing — and how it differs from traditional hypothesis testing.
Use this when
You are building or evaluating a prediction model in a clinical or research context.
How ML Differs from Traditional Statistics
Traditional statistical analysis aims to test hypotheses about population parameters and quantify uncertainty. Machine learning aims to build a model that predicts outcomes for new, unseen data. These are different goals — and they require different evaluation frameworks.
| Traditional Statistics | Machine Learning | |
|---|---|---|
| Primary goal | Test a hypothesis; estimate effect with uncertainty | Predict outcome for new cases accurately |
| Output | p-value, CI, coefficient | Predicted probability or class label |
| Model selection | Based on theory, confounding, interpretation | Based on predictive performance on held-out data |
| Evaluation | Goodness-of-fit, AIC, residual plots | AUC-ROC, accuracy, calibration on test set |
| Overfitting concern | Mostly addressed by model parsimony | Central problem — requires train/test split |
The Train / Validation / Test Split
The single most important concept in ML is that you must evaluate your model on data it has never seen during training. If you evaluate on the same data used to fit the model, you will overestimate its real-world performance — this is called overfitting.
- ●Training set (60–70%): used to fit model parameters
- ●Validation set (10–20%): used to tune hyperparameters and select the model
- ●Test set (10–20%): held out entirely until final evaluation — never touch during development
- ●With small datasets (< 500 cases): use k-fold cross-validation instead of a single split
⚠️ The most common ML mistake in clinical research
Fitting a model on the full dataset and then reporting AUC from the same data. This inflates performance dramatically. Any clinical ML paper must report performance on a held-out test set — or ideally, an external validation cohort.
Evaluation Metrics
| Metric | What it measures | When to use |
|---|---|---|
| AUC-ROC | Discrimination — ability to rank cases higher than controls | Binary classification, class-balanced or imbalanced |
| AUC-PR (Precision-Recall) | Performance on the positive class specifically | Highly imbalanced datasets (rare events) |
| Calibration (Brier score / calibration plot) | Whether predicted probabilities match observed frequencies | Clinical risk scores where probability matters |
| Accuracy | Proportion of correct predictions | Balanced classes only — misleading for rare events |
| F1 score | Harmonic mean of precision and recall | Imbalanced classification with focus on positive class |
What Clinical ML Papers Must Report
- 1.Dataset: source, time period, sample size, event rate, inclusion/exclusion criteria
- 2.Feature selection: which variables were candidate predictors and how they were selected
- 3.Model development: algorithm, hyperparameter tuning method, cross-validation approach
- 4.Performance: AUC-ROC with 95% CI on held-out test set (not training set)
- 5.Calibration: calibration plot and Hosmer-Lemeshow or Brier score
- 6.SHAP or feature importance: which variables drove the model
- 7.External validation: ideally validation on a separate cohort from a different center or time period
Practice with your own dataset
Train a classification model on your dataset.
Required variables
- • Target variable (binary or continuous)
- • Predictor variables
- 1.Upload a dataset with a binary outcome variable
- 2.Go to Analyze → ML Prediction
- 3.Select the outcome (target) variable and predictor variables
- 4.Choose a model type (Logistic Regression, Random Forest, or Gradient Boosting)
- 5.Review AUC-ROC, feature importance, and SHAP plots on the held-out test set
Trusted sources behind this lesson
Further reading
Read next
SHAP Values: Explaining ML Model Predictions
Understand how SHAP (SHapley Additive exPlanations) values measure each feature's contribution to individual predictions — and how to present them to clinical audiences.