Training and Evaluation

machine-learning-ai
Split data, fit a first model, evaluate predictions, and recognize overfitting.
  • Level: Beginner to intermediate
  • Estimated time: 35–50 minutes
  • You will learn: Split data, fit a first model, evaluate predictions, and recognize overfitting.
  • Practice in: Jupyter or Google Colab

Open this lesson in Google Colab

Questions

  • What problem does Training and Evaluation help us solve in a small Python program?
  • What should we predict before running the example?
  • What value, output, or error should we inspect after changing one line?

Objectives

  • Run a complete example for training and evaluation in Colab.
  • Explain the example line by line using plain language.
  • Change one part of the code and predict the result before running it.
  • Recognize one common mistake and use the error message as evidence.

Hands-on episode: Training and Evaluation

Training fits a model to examples. Evaluation tests predictions on held-out data using metrics that match the problem and the cost of mistakes.

We will learn this by running code, not by memorizing a definition first. Open the Colab notebook from the button above, find this section, and run each cell in order. Keep a small note beside the notebook with three columns: prediction, actual result, and what changed.

Example 1.1

Read the workflow as fit, predict, evaluate.

from sklearn.model_selection import train_test_split
from sklearn.metrics import accuracy_score
from sklearn.tree import DecisionTreeClassifier

X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.25, random_state=42)
model = DecisionTreeClassifier(random_state=42)
model.fit(X_train, y_train)
predictions = model.predict(X_test)
print(accuracy_score(y_test, predictions))

Run the cell once without editing it. If the result is different from your prediction, leave the prediction visible and write one sentence about the difference. That sentence is more useful than a perfect first guess.

Explain Example 1.1

  • The split separates practice data from exam data.
  • fit lets the model learn patterns from the training examples.
  • predict asks the trained model for answers on test features.
  • The metric compares predictions with known test labels.

Now explain the example out loud or in a Markdown cell. Use short sentences: “this line creates…”, “this name stores…”, “this output appears because…”. If you cannot explain a line yet, run only the lines above it and inspect the values that exist at that moment.

Challenge 1.1

NoteChallenge

Evaluate on X_train and compare the score with X_test. A very high training score with a lower test score can signal overfitting.

Show a safe way to approach the challenge
  1. Copy Example 1.1 into a new Colab cell.
  2. Change exactly one value, name, condition, or line.
  3. Write the expected output before running the cell.
  4. Run the cell and compare the actual result with your prediction.
  5. If the result surprises you, undo the change and try a smaller one.

Suggested first move: Evaluate on X_train and compare the score with X_test.

Debugging checkpoint 1.1

WarningDebugging checkpoint

Do not tune decisions by repeatedly peeking at the test set. The test set should stay as honest as possible. Also choose metrics based on consequences; accuracy can hide serious failure when classes are imbalanced.

Do not debug by rewriting the whole example. Read the error type or surprising output, inspect the closest value with print(...) or type(...), then change one thing. This is the same routine you will use in larger projects.

Apply it

Train a simple classifier or regressor in Colab, print one metric, inspect a few predictions, and write whether the metric is enough to trust the model.

Finish by adding a Markdown cell that answers: What did this example teach me that I can reuse in a project?

Key points

  • Learn the concept by running a complete, small example first.
  • Predict before execution so your thinking becomes visible.
  • Change one thing at a time so cause and effect stay clear.
  • Treat errors as clues about the exact line or value Python could not handle.

Why this matters

Split data, fit a first model, evaluate predictions, and recognize overfitting.

This lesson combines related subtopics that belong together in one learning conversation. You will still pause for a quiz after each section, but you do not need to jump between separate pages while building one clear explanation.

NoteGuiding questions

By the end of this lesson, you should be able to answer:

  • How do the sections in Training and Evaluation fit together?
  • Which small example demonstrates each section?
  • Which debugging clue should I check first for each section?
NoteLearning objectives

You will practice how to:

  • explain the shared concept for this lesson;
  • use each section as one step in a larger workflow;
  • complete 4 short section quizzes before moving on;
  • connect examples, mistakes, and debugging routines.

Lesson map

  • 1. Train/Test Split — Separate data for learning from data for honest evaluation.
  • 2. First Model with scikit-learn — Fit a simple model and make predictions using scikit-learn’s estimator pattern.
  • 3. Evaluating Models — Use metrics to compare predictions with known answers.
  • 4. Overfitting and Underfitting — Recognize when a model memorizes too much or learns too little.

How model training flows

A basic supervised machine-learning workflow separates examples used for learning from examples used for evaluation.

flowchart LR
  data["labeled dataset"] --> split["train/test split"]
  split --> train["training data"]
  split --> test["test data"]
  train --> fit["fit model"]
  fit --> model["trained model"]
  test --> predict["predict labels"]
  model --> predict
  predict --> metrics["evaluate metrics"]

If evaluation uses examples the model already trained on, the score can be too optimistic.

1. Train/Test Split

Separate data for learning from data for honest evaluation.

TipAnalogy

A test set is a closed-book exam: the student should not see the exact questions during practice.

What this means

A train/test split holds back examples so performance is measured on data the model did not train on.

Example 1

Predict what will happen before you run the code.

from sklearn.model_selection import train_test_split

X = [[1], [2], [3], [4]]
y = [2, 4, 6, 8]
X_train, X_test, y_train, y_test = train_test_split(X, y, random_state=42)

Step-by-step explanation

  1. from sklearn.model_selection import train_test_split — pause here and say what this line reads, creates, changes, or displays.
  2. X = [[1], [2], [3], [4]] — pause here and say what this line reads, creates, changes, or displays.
  3. y = [2, 4, 6, 8] — pause here and say what this line reads, creates, changes, or displays.
  4. X_train, X_test, y_train, y_test = train_test_split(X, y, random_state=42) — pause here and say what this line reads, creates, changes, or displays.

After running the example, compare the actual output with your prediction. If they differ, do not erase your prediction. The difference is the part that can teach you the most.

Challenge

NotePractice

Change one input value, predict the new output, run the code, and explain the difference in one sentence.

Show one possible solution path
  1. Copy Example 1 into Colab, Jupyter, or a .py file.
  2. Mark the line you plan to change.
  3. Write a one-sentence prediction.
  4. Run the changed code.
  5. If the result surprises you, restore the original and change a smaller part.

The goal is not to find the only correct answer. The goal is to create a small experiment where you can explain cause and effect.

Common mistakes

WarningCommon mistake

Preprocessing the full dataset before splitting can leak information from test data.

When you get stuck, use this debugging routine:

  1. Read the last line of the error message or inspect the unexpected output.
  2. Find the smallest line of code that could be responsible.
  3. Print or inspect the value and type at that point.
  4. Change one thing.
  5. Run again and record what changed.

Check your understanding

This quiz checks the ideas in this section before you move on.

2. First Model with scikit-learn

Fit a simple model and make predictions using scikit-learn’s estimator pattern.

TipAnalogy

The estimator pattern is like coaching: fit is practice, predict is game day.

What this means

scikit-learn models usually use fit to learn and predict to produce outputs.

Example 2

Predict what will happen before you run the code.

from sklearn.linear_model import LinearRegression

model = LinearRegression()
model.fit([[1], [2], [3]], [2, 4, 6])
print(model.predict([[4]]))

Step-by-step explanation

  1. from sklearn.linear_model import LinearRegression — pause here and say what this line reads, creates, changes, or displays.
  2. model = LinearRegression() — pause here and say what this line reads, creates, changes, or displays.
  3. model.fit([[1], [2], [3]], [2, 4, 6]) — pause here and say what this line reads, creates, changes, or displays.
  4. print(model.predict([[4]])) — pause here and say what this line reads, creates, changes, or displays.

After running the example, compare the actual output with your prediction. If they differ, do not erase your prediction. The difference is the part that can teach you the most.

Challenge

NotePractice

Change one input value, predict the new output, run the code, and explain the difference in one sentence.

Show one possible solution path
  1. Copy Example 1 into Colab, Jupyter, or a .py file.
  2. Mark the line you plan to change.
  3. Write a one-sentence prediction.
  4. Run the changed code.
  5. If the result surprises you, restore the original and change a smaller part.

The goal is not to find the only correct answer. The goal is to create a small experiment where you can explain cause and effect.

Common mistakes

WarningCommon mistake

The shape of X matters. scikit-learn usually expects a 2D feature matrix, even for one feature.

When you get stuck, use this debugging routine:

  1. Read the last line of the error message or inspect the unexpected output.
  2. Find the smallest line of code that could be responsible.
  3. Print or inspect the value and type at that point.
  4. Change one thing.
  5. Run again and record what changed.

Check your understanding

This quiz checks the ideas in this section before you move on.

3. Evaluating Models

Use metrics to compare predictions with known answers.

TipAnalogy

Metrics are scorecards: they summarize performance but do not tell the whole story alone.

What this means

Evaluation measures how well predictions match labels on held-out data.

Example 3

Predict what will happen before you run the code.

from sklearn.metrics import mean_absolute_error

y_true = [10, 20, 30]
y_pred = [12, 18, 33]
print(mean_absolute_error(y_true, y_pred))

Step-by-step explanation

  1. from sklearn.metrics import mean_absolute_error — pause here and say what this line reads, creates, changes, or displays.
  2. y_true = [10, 20, 30] — pause here and say what this line reads, creates, changes, or displays.
  3. y_pred = [12, 18, 33] — pause here and say what this line reads, creates, changes, or displays.
  4. print(mean_absolute_error(y_true, y_pred)) — pause here and say what this line reads, creates, changes, or displays.

After running the example, compare the actual output with your prediction. If they differ, do not erase your prediction. The difference is the part that can teach you the most.

Challenge

NotePractice

Change one input value, predict the new output, run the code, and explain the difference in one sentence.

Show one possible solution path
  1. Copy Example 1 into Colab, Jupyter, or a .py file.
  2. Mark the line you plan to change.
  3. Write a one-sentence prediction.
  4. Run the changed code.
  5. If the result surprises you, restore the original and change a smaller part.

The goal is not to find the only correct answer. The goal is to create a small experiment where you can explain cause and effect.

Common mistakes

WarningCommon mistake

One metric can hide important failures. Choose metrics that match the real-world cost of mistakes.

When you get stuck, use this debugging routine:

  1. Read the last line of the error message or inspect the unexpected output.
  2. Find the smallest line of code that could be responsible.
  3. Print or inspect the value and type at that point.
  4. Change one thing.
  5. Run again and record what changed.

Check your understanding

This quiz checks the ideas in this section before you move on.

4. Overfitting and Underfitting

Recognize when a model memorizes too much or learns too little.

TipAnalogy

Overfitting is memorizing the practice answers; underfitting is not studying enough.

What this means

Overfitting fits training data too closely; underfitting misses important patterns.

Example 4

Predict what will happen before you run the code.

training_score = 0.99
test_score = 0.60
print("possible overfitting", training_score - test_score)

Step-by-step explanation

  1. training_score = 0.99 — pause here and say what this line reads, creates, changes, or displays.
  2. test_score = 0.60 — pause here and say what this line reads, creates, changes, or displays.
  3. print("possible overfitting", training_score - test_score) — pause here and say what this line reads, creates, changes, or displays.

After running the example, compare the actual output with your prediction. If they differ, do not erase your prediction. The difference is the part that can teach you the most.

Challenge

NotePractice

Change one input value, predict the new output, run the code, and explain the difference in one sentence.

Show one possible solution path
  1. Copy Example 1 into Colab, Jupyter, or a .py file.
  2. Mark the line you plan to change.
  3. Write a one-sentence prediction.
  4. Run the changed code.
  5. If the result surprises you, restore the original and change a smaller part.

The goal is not to find the only correct answer. The goal is to create a small experiment where you can explain cause and effect.

Common mistakes

WarningCommon mistake

A high training score alone is not proof of a good model. Compare against validation or test performance.

When you get stuck, use this debugging routine:

  1. Read the last line of the error message or inspect the unexpected output.
  2. Find the smallest line of code that could be responsible.
  3. Print or inspect the value and type at that point.
  4. Change one thing.
  5. Run again and record what changed.

Check your understanding

This quiz checks the ideas in this section before you move on.

Notebook and Colab practice

Open a blank notebook at https://colab.new. Use one section at a time: copy the Example 1, predict the result, run it, answer the section quiz, and then move to the next section. This is better than copying the entire page at once.

Instructor note

Teaching notes
  • Treat each section as a short teaching episode.
  • Pause for the section quiz before introducing the next section.
  • Ask learners to compare sections: what stayed the same, and what changed?
  • If time is short, teach the first two sections live and assign the rest as practice.

Key points

TipKey points
  • Train/Test Split: A train/test split holds back examples so performance is measured on data the model did not train on.
  • First Model with scikit-learn: scikit-learn models usually use fit to learn and predict to produce outputs.
  • Evaluating Models: Evaluation measures how well predictions match labels on held-out data.
  • Overfitting and Underfitting: Overfitting fits training data too closely; underfitting misses important patterns.
  • Use the section quizzes as gates: review before moving on if a quiz feels uncertain.

References

  • scikit-learn documentation: https://scikit-learn.org/stable/
  • PyTorch documentation: https://pytorch.org/docs/stable/index.html
  • TensorFlow documentation: https://www.tensorflow.org/api_docs
  • Hugging Face Transformers documentation: https://huggingface.co/docs/transformers/
  • Python Tutorial: https://docs.python.org/3/tutorial/
  • Quarto OJS documentation: https://quarto.org/docs/interactive/ojs/
  • ipywidgets documentation: https://ipywidgets.readthedocs.io/en/stable/
Back to top