If you hear "this model is 99% accurate" and conclude it is a good model, MLA-C01 will catch you in a trap. Task 2.3 of the exam guide covers evaluation metrics such as the confusion matrix, heat maps, F1, accuracy, precision, recall, RMSE, ROC, and AUC, along with creating performance baselines and identifying overfitting, while Task 1.3 asks about pre-training bias metrics such as class imbalance (CI) and difference in proportions of labels (DPL), strategies for imbalance such as synthetic data and resampling, and dataset splitting and shuffling. In our question bank, imbalanced situations such as "data in which only 0.5% of transactions are fraud" come up again and again.
This post starts with the confusion matrix and then covers choosing classification metrics, adjusting the threshold, regression metrics, baseline models, handling class imbalance, and the bias metrics of SageMaker Clarify. Diagnosing overfitting and regularization was covered first in ML Fundamentals for Cloud Engineers.
---
Start with the Confusion Matrix: What the Four Cells Mean
Think of a fire alarm. If it rings when there is a fire, it caught it; if it rings when there is no fire, that is a false alarm; and if it stays silent during a fire, that is the most dangerous miss. Every metric of a classification model comes from these four cases.
| Aspect | Actually positive (fire) | Actually negative (no fire) | |------|------|------| | Predicted positive (alarm rings) | TP true positive | FP false positive (false alarm) | | Predicted negative (alarm silent) | FN false negative (miss) | TN true negative |
Which is more expensive, FP or FN, is the starting point for choosing a metric. In some situations sending a legitimate email to spam (FP) is costly, and in others missing actual fraud (FN) is costly.
!How precision, recall, accuracy, and F1 are calculated from the four cells of the confusion matrix: TP, FP, FN, and TN
---
Classification Metrics: Choose for the Situation
| Metric | Calculation | Meaning | When it matters | |------|------|------|------| | Accuracy | (TP + TN) ÷ total | Share of all predictions that are correct | Meaningful only when classes are balanced | | Precision | TP ÷ (TP + FP) | Of those predicted positive, the share that are truly positive | When FP is costly (spam filters, choosing marketing targets) | | Recall (sensitivity) | TP ÷ (TP + FN) | Of the true positives, the share found | When FN is costly (cancer diagnosis, fraud detection) | | Specificity | TN ÷ (TN + FP) | Share of true negatives correctly predicted negative | When leaving normal cases alone matters | | F1 score | Harmonic mean of precision and recall | The balance of the two | When both matter on imbalanced data | | ROC AUC | Area under the curve of the true positive rate vs. the false positive rate across all thresholds | 0.5 is random, closer to 1 is better | Comparing models regardless of threshold | | PR AUC | Area under the precision-recall curve | Shows differences better than ROC when positives are very rare | Extreme imbalance |
Numbers make the accuracy trap obvious. If 10 of 1,000 transactions are fraud and the model answers "legitimate" for all of them, accuracy is 99% but recall is 0%. The model did not catch a single fraud.
Exam tip: "Not a single case of a rare disease may be missed" is recall, "blocking legitimate transactions as fraud causes major customer complaints" is precision, "both matter on imbalanced data" is F1, and "compare several models regardless of threshold" is AUC.
---
Adjusting the Threshold: Same Model, Different Results
Most classification models output a probability and predict positive when it exceeds a threshold (usually 0.5). Changing the threshold changes the results without retraining.
| Threshold | Recall | Precision | When to use it | |------|------|------|------| | Lowered (for example, 0.5 → 0.3) | Goes up | Goes down | Fraud or disease detection where misses are unacceptable | | Raised (for example, 0.5 → 0.8) | Goes down | Goes up | Automatic blocking where false alarms must be reduced |
The ROC curve is drawn by moving the threshold from 0 to 1, so each point on the curve is one threshold.
Evaluating Multiclass Models
With three or more classes, you compute precision and recall per class and then average them. A macro average gives every class the same weight, so small classes count just as much, while a weighted average weights each class by its number of samples. A heat map that colors the confusion matrix shows at a glance which classes are confused with each other.
Exam tip: "Catch more fraud without retraining the model" is solved by lowering the threshold.
---
Regression Metrics
Regression models, which predict numbers, are evaluated by the size of their errors.
| Metric | Meaning | Characteristics | |------|------|------| | RMSE | Square root of the mean of squared errors | Penalizes large errors more; use it when large errors must especially be avoided | | MAE | Mean of absolute errors | Less sensitive to outliers, easy to interpret ("off by how many dollars on average") | | MAPE | Mean error as a percentage of the actual value | Compares predictions in different units; unstable when actual values are near 0 | | R² | Share of variance explained by the model | Closer to 1 is better |
Exam tip: "An occasional large miss is especially damaging" is RMSE, and "there are many outliers and we want a stable view of the average error" is MAE.
---
Baseline Models and Shadow Comparison
To judge whether a complex model is good, you need something to compare it with. A baseline is the performance of the simplest possible approach.
Classification: a model that always predicts the majority class, or a simple model such as logistic regression or Linear Learner Regression: a model that always predicts the mean, or linear regression Tabular data: XGBoost with default settings as a strong baseline
The model performance baseline here is different from the baseline Model Monitor builds from training data (statistics and constraints). The first tells you "is this model useful?", and the second tells you "has the production data drifted from what we trained on?"
Whether a new model is also better on real traffic is checked with shadow testing. The same requests sent to the production variant also go to the new model (the shadow variant), but users receive only the production model's responses while you compare the metrics of the two.
Exam tip: "Evaluate a new model on real traffic without affecting users" is shadow testing. "Actually send part of the traffic to the new model and compare results" is A/B testing (production variant weights).
---
Handling Class Imbalance
| Technique | How it works | What to watch for | |------|------|------| | Stratified split | Keeps the same class ratio in the training, validation, and test sets | The default for splitting imbalanced data | | Oversampling | Duplicates minority class samples | Risk of overfitting from repeated samples | | SMOTE | Creates new synthetic samples between minority class samples | Apply only to the training data, never to validation or test data | | Undersampling | Reduces majority class samples | Loss of information | | Class weights | Penalizes mistakes on the minority class more heavily | Data stays as is; in XGBoost, scale_pos_weight = number of negatives ÷ number of positives | | Augmentation and synthetic data | Rotating images, paraphrasing sentences, generating samples with generative models | Effective for image and text data | | Changing the metric | Recall, F1, or PR AUC instead of accuracy | Accuracy is misleading under imbalance |
SageMaker Data Wrangler has a balance transform that evens out data with random oversampling, random undersampling, or SMOTE. Order matters too. Split the data first and apply resampling only to the training set. If you apply SMOTE before splitting, synthetic samples leak into the validation set and performance looks better than it really is.
Exam tip: Candidate answers for "fraud is 0.5% of transactions, so the model barely catches any" are SMOTE, class weights (scale_pos_weight), and evaluating with recall and F1. "Accuracy is 99%, so deploy it" is wrong.
---
Measuring Bias in Data and Models: Clarify Metrics
Class imbalance is about the ratio of labels (answers), while bias is about a particular group (a facet such as gender or age group) being treated unfavorably. SageMaker Clarify computes bias metrics before and after training.
| When | Typical metric | Meaning | |------|------|------| | Before training (data) | CI, class imbalance | Difference in the number of samples between groups (for example, most data is from middle-aged people) | | Before training (data) | DPL, difference in proportions of labels | Difference in the share of positive labels between groups (for example, loan approval rates differ by group) | | After training (predictions) | DPPL, difference in positive proportions in predicted labels | Difference in how often the model predicts positive for each group | | After training (predictions) | DI, disparate impact | Ratio of positive prediction rates between groups |
Other pre-training metrics include KL, JS, LP, TVD, KS, and CDD, and SHAP values are used to explain the model. Sources of data bias include samples that do not represent the population (selection bias) and values measured differently for different groups (measurement bias).
Clarify has not accepted new customers since July 30, 2026. AWS suggests computing the same bias metrics directly with pandas and scikit-learn and using the SHAP library directly as the alternative. On MLA-C01, however, it is still the answer for bias detection and explainability.
Exam tip: "Measure the difference in the share of positive labels between groups before training" is the Clarify pre-training bias metric DPL, "check whether the model's predictions differ