Computing / Lesson 05
Learning systems and evaluation
Treat a trained model as one component in an evidence-producing system.
orient
Why this idea had to exist
Machine learning replaces hand-written rules with behavior fitted from examples. That flexibility makes evaluation central: a model can fit its training distribution while failing where deployment conditions differ.
intuition
Build a picture you can reason with
Training is rehearsal, validation is direction, and testing is opening night. Reusing the audience's reactions to rewrite the play turns the test into more rehearsal. Separate data roles preserve the meaning of the final evidence.
formalize
Give the intuition a precise edge
A learning algorithm selects parameters using training data. Hyperparameters and decisions use validation evidence. A held-out test estimates performance for a specified sampling process. Subgroup, shift, calibration, and error-cost analyses test assumptions hidden by one average metric.
work through
Follow the decisions, not just the symbols
For a rare failure class, 99 percent accuracy can come from always predicting the common class. Build a confusion matrix, inspect recall and precision, and choose a threshold based on the real cost of misses and false alarms.
experiment
Change one thing and watch the model answer
Move a classification threshold and watch false positives and false negatives trade places. Then change class prevalence and observe why precision changes even when sensitivity and specificity stay fixed.
retrieve
Close the page and reconstruct it
Answer before opening the explanation. Retrieval is evidence only when the answer is produced without a hint.
Why should the test set not guide model selection?
Using it to choose the model adapts decisions to that sample, so its score no longer estimates performance on untouched data.
transfer
Move the idea into a new setting
Write an evaluation plan for a model used across common, code-heavy, and dialogue-heavy environments, including one worst-environment measure.
reflect
Leave with a diagnostic habit
A score answers only the question encoded by its data, metric, and decision threshold. Write that question beside every result.
Source record
Follow the idea back.
- Reference
- Machine Learning Crash Course
- Publisher
- Google for Developers
- License
- CC BY 4.0
- Accessed
- 2026-08-12
- URL
- https://developers.google.com/machine-learning/crash-course