Understanding Confusion Matrices
A confusion matrix is a table that visualizes classification performance by showing exactly where a model gets confused. It's one of the most useful tools for understanding what's going right and wrong with a classifier.
What is a Confusion Matrix?
A confusion matrix shows the count of predictions for each combination of actual vs. predicted class:
PREDICTED
Cat Dog Bird
Cat 45 5 2 ← 45 cats correctly identified
ACTUAL Dog 3 38 4 ← 3 dogs misclassified as cats
Bird 2 7 41 ← 7 birds misclassified as dogs
Each row represents actual classes; each column represents predictions.
Reading a Binary Confusion Matrix
For binary classification (positive/negative):
PREDICTED
Positive Negative
ACTUAL Positive TP FN
Negative FP TN
Where:
- TP (True Positive): Correctly predicted positive
- TN (True Negative): Correctly predicted negative
- FP (False Positive): Incorrectly predicted positive (Type I error)
- FN (False Negative): Incorrectly predicted negative (Type II error)
Example: Medical Diagnosis
PREDICTED
Disease Healthy
ACTUAL Disease 85 15 ← 15 missed cases (FN)
Healthy 10 890 ← 10 false alarms (FP)
From this matrix:
- TP = 85: Correctly diagnosed diseases
- TN = 890: Correctly identified healthy patients
- FP = 10: Healthy patients incorrectly told they're sick
- FN = 15: Sick patients missed (dangerous!)
Extracting Metrics
All classification metrics come from the confusion matrix:
Accuracy = (TP + TN) / Total = (85 + 890) / 1000 = 97.5%
Precision = TP / (TP + FP) = 85 / 95 = 89.5%
Recall = TP / (TP + FN) = 85 / 100 = 85%
F1 = 2 × (0.895 × 0.85) / (0.895 + 0.85) = 87.2%
Multi-Class Confusion Matrices
For K classes, the matrix is K × K:
Example: Digit Recognition (0-9)
PREDICTED: 0 1 2 3 4 5 6 7 8 9
ACTUAL 0: 980 0 2 1 0 4 5 1 3 4
1: 0 1120 3 2 0 1 2 0 5 2
2: 6 5 970 8 3 2 3 4 6 5
3: 2 3 7 960 0 10 0 4 8 6
4: 1 2 4 0 950 0 6 3 4 12
5: 4 2 1 12 3 850 9 1 5 5
6: 3 3 0 1 4 5 942 0 0 0
7: 0 4 10 4 0 1 0 990 2 17
8: 4 3 5 7 3 6 3 2 940 1
9: 3 3 1 7 15 5 1 6 5 963
What We Can Learn
- Diagonal: Correct predictions (want these high)
- Off-diagonal: Errors (want these low)
- Row i, column j: How often class i is confused with class j
From the example:
- 4 and 9 are often confused (look similar)
- 7 and 9 are confused (similar top curve)
- 1 is rarely confused (distinctive shape)
Normalized Confusion Matrix
Normalize by row to see per-class accuracy:
Raw:
Cat Dog
Cat 45 5 → 50 total cats
Dog 3 47 → 50 total dogs
Normalized:
Cat Dog
Cat 0.90 0.10 → 90% of cats correctly identified
Dog 0.06 0.94 → 94% of dogs correctly identified
This shows:
- Cats are correctly classified 90% of the time
- Dogs are correctly classified 94% of the time
- Cats are more often confused with dogs than vice versa
Visualizing Confusion Matrices
import matplotlib.pyplot as plt
import seaborn as sns
# Plot confusion matrix as heatmap
plt.figure(figsize=(8, 6))
sns.heatmap(confusion_matrix, annot=True, fmt='d', cmap='Blues')
plt.xlabel('Predicted')
plt.ylabel('Actual')
plt.title('Confusion Matrix')
Predicted
Cat Dog Bird
Cat [███] [░░░] [ ]
Actual Dog [░] [███] [░]
Bird [ ] [░░] [███]
Dark = Many predictions
Light = Few predictions
Per-Class Metrics
From a multi-class confusion matrix, compute metrics for each class:
For class "Cat":
- TP: Predicted cat, actually cat (diagonal)
- FP: Predicted cat, actually not cat (column sum - diagonal)
- FN: Actually cat, predicted not cat (row sum - diagonal)
def per_class_metrics(cm, class_idx):
tp = cm[class_idx, class_idx]
fp = cm[:, class_idx].sum() - tp
fn = cm[class_idx, :].sum() - tp
precision = tp / (tp + fp) if (tp + fp) > 0 else 0
recall = tp / (tp + fn) if (tp + fn) > 0 else 0
return precision, recall
Micro vs. Macro Averaging
For multi-class, we can aggregate metrics two ways:
Macro-Average
Calculate metric for each class, then average:
macro_precision = sum(precision[c] for c in classes) / num_classes
Treats all classes equally regardless of size.
Micro-Average
Pool all predictions, then calculate:
total_tp = sum(tp[c] for c in classes)
total_fp = sum(fp[c] for c in classes)
micro_precision = total_tp / (total_tp + total_fp)
Larger classes dominate.
When to Use Which
- Macro: When you care equally about all classes
- Micro: When you care about overall performance
Common Patterns in Confusion Matrices
Diagonal Dominance (Good)
A B C
A [95] 3 2
B 2 [91] 7
C 3 5 [92]
Most predictions are correct (on diagonal).
Off-Diagonal Cluster (Confusing Classes)
A B C
A [90] [8] 2
B [9] [85] 6
C 3 4 [93]
A and B are often confused with each other.
Row/Column Bias (Prediction Imbalance)
A B C
A 80 15 5
B 10 75 15
C 40 20 40
Class C is under-predicted (row sum > column sum).
Single Row Failure (Class Collapse)
A B C
A 95 3 2
B 85 10 5
C 80 15 5
Model barely predicts B or C—just defaults to A.
Using Confusion Matrices for Debugging
1. Identify Problematic Classes
Look for rows/columns with low diagonal values.
2. Find Confusion Patterns
Which classes are confused? Often visually or semantically similar.
3. Check for Bias
Are some classes over/under predicted?
4. Guide Data Collection
If A→B confusion is high, collect more A/B training examples.
5. Consider Class Merging
If two classes are always confused, maybe they should be one class.
Threshold Analysis
For binary classification, the confusion matrix changes with threshold:
Threshold = 0.5:
Pos Neg
Actual Pos 80 20
Actual Neg 10 90
Threshold = 0.3 (more aggressive):
Pos Neg
Actual Pos 95 5 ← More TP, fewer FN
Actual Neg 30 70 ← More FP
Threshold = 0.7 (more conservative):
Pos Neg
Actual Pos 60 40 ← Fewer TP, more FN
Actual Neg 5 95 ← Fewer FP
Plotting metrics across thresholds gives ROC and PR curves.
Cost-Sensitive Analysis
Different errors have different costs:
Medical screening:
- FN (miss disease): Cost = $50,000 (late treatment)
- FP (false alarm): Cost = $500 (extra tests)
Total cost = FN × $50,000 + FP × $500
= 15 × $50,000 + 10 × $500
= $750,000 + $5,000
= $755,000
Optimize for minimum cost, not just accuracy.
Confusion Matrix in Production
Monitor confusion matrices over time:
January: Cat Dog June: Cat Dog
Cat 90 10 Cat 85 15 ← Performance dropping
Dog 5 95 Dog 25 75 ← Dog detection degraded
Possible causes:
- Data drift (new dog breeds?)
- Season change (winter coats?)
- Bug in preprocessing
Summary
- Confusion matrices show exactly where models make mistakes
- Diagonal = correct, off-diagonal = errors
- Derive precision, recall, F1 from the matrix
- Normalize by row to see per-class performance
- Use for debugging: find confusing classes, bias, and patterns
- Different thresholds give different matrices
- Monitor in production to catch degradation
This completes the Probability & Statistics for AI course! You now have the probabilistic foundations to understand how AI models learn, predict, and handle uncertainty.

اسأل Byte عن هذا الدرس
محادثة خاصة حول هذا الدرس. قرأ Byte الدرس كاملًا، ويجيب خلال ثوانٍ، ويُظهر القسم الذي تأتي منه كل إجابة. يمكنك السؤال بأي لغة.
الأسئلة والأجوبة
أسئلة عامة حول هذا الدرس. يجيب AI Tutor فورًا بلغتك، ويراجع فريق FreeAcademy الإجابات، ويمكن لكل المتعلمين قراءتها.

