Inspecting, Selecting, and Cleaning

Understand DataFrames, choose relevant data, and handle missing values.

• Level: Beginner to intermediate
• Estimated time: 35–50 minutes
• You will learn: Understand DataFrames, choose relevant data, and handle missing values.
• Practice in: Jupyter or Google Colab

Open this lesson in Google Colab

Questions

• What problem does Inspecting, Selecting, and Cleaning help us solve in a small Python program?
• What should we predict before running the example?
• What value, output, or error should we inspect after changing one line?

Objectives

• Run a complete example for inspecting, selecting, and cleaning data in Colab.
• Explain the example line by line using plain language.
• Change one part of the code and predict the result before running it.
• Recognize one common mistake and use the error message as evidence.

Hands-on episode: Inspecting, Selecting, and Cleaning

Inspection reveals structure and quality. Selection focuses on relevant rows and columns. Cleaning fixes or documents issues such as missing values, inconsistent types, and impossible values.

We will learn this by running code, not by memorizing a definition first. Open the Colab notebook from the button above, find this section, and run each cell in order. Keep a small note beside the notebook with three columns: prediction, actual result, and what changed.

Example 1.1

Predict which rows remain after selecting scores that are not missing.

import pandas as pd

data = pd.DataFrame({"name": ["Ada", "Grace", "Lin"], "score": [9, None, 7]})
clean = data.dropna(subset=["score"])
print(clean)

Run the cell once without editing it. If the result is different from your prediction, leave the prediction visible and write one sentence about the difference. That sentence is more useful than a perfect first guess.

Explain Example 1.1

• The original data has one missing score.
• dropna(subset=["score"]) removes rows where the score column is missing.
• The cleaned DataFrame keeps Ada and Lin but removes Grace for this specific analysis.

Now explain the example out loud or in a Markdown cell. Use short sentences: “this line creates…”, “this name stores…”, “this output appears because…”. If you cannot explain a line yet, run only the lines above it and inspect the values that exist at that moment.

Challenge 1.1

NoteChallenge

Instead of dropping the row, fill the missing score with a value and explain why that may or may not be honest. Cleaning choices should match the question.

Show a safe way to approach the challenge

1. Copy Example 1.1 into a new Colab cell.
2. Change exactly one value, name, condition, or line.
3. Write the expected output before running the cell.
4. Run the cell and compare the actual result with your prediction.
5. If the result surprises you, undo the change and try a smaller one.

Suggested first move: Instead of dropping the row, fill the missing score with a value and explain why that may or may not be honest.

Debugging checkpoint 1.1

WarningDebugging checkpoint

Do not drop missing data automatically. Ask what missing means, how many values are missing, and whether removal changes the population you are studying. Also be careful with chained indexing; use clear .loc selections when assigning.

Do not debug by rewriting the whole example. Read the error type or surprising output, inspect the closest value with print(...) or type(...), then change one thing. This is the same routine you will use in larger projects.

Apply it

Inspect a small DataFrame with .shape, .head(), .dtypes, and .isna().sum(). Select two columns and clean one missing-value issue while writing down your reason.

Finish by adding a Markdown cell that answers: What did this example teach me that I can reuse in a project?

Key points

• Learn the concept by running a complete, small example first.
• Predict before execution so your thinking becomes visible.
• Change one thing at a time so cause and effect stay clear.
• Treat errors as clues about the exact line or value Python could not handle.

Why this matters

Understand DataFrames, choose relevant data, and handle missing values.

This lesson combines related subtopics that belong together in one learning conversation. You will still pause for a quiz after each section, but you do not need to jump between separate pages while building one clear explanation.

NoteGuiding questions

By the end of this lesson, you should be able to answer:

• How do the sections in Inspecting, Selecting, and Cleaning fit together?
• Which small example demonstrates each section?
• Which debugging clue should I check first for each section?

NoteLearning objectives

You will practice how to:

• explain the shared concept for this lesson;
• use each section as one step in a larger workflow;
• complete 3 short section quizzes before moving on;
• connect examples, mistakes, and debugging routines.

Lesson map

• 1. Inspecting DataFrames — Use shape, columns, dtypes, head, info, and describe to understand data.
• 2. Selecting and Filtering Data — Choose columns, rows, and conditions from a DataFrame.
• 3. Cleaning Missing Values — Find and handle missing data deliberately.

1. Inspecting DataFrames

Use shape, columns, dtypes, head, info, and describe to understand data.

TipAnalogy

Inspection is checking a shipment before unpacking everything: count boxes, read labels, notice damage.

What this means

Inspection reveals structure before you transform or model data.

Example 1

Predict what will happen before you run the code.

import pandas as pd

df = pd.DataFrame({"score": [1, 2, 3], "group": ["a", "a", "b"]})
print(df.shape)
print(df.dtypes)
print(df.describe())

Step-by-step explanation

1. import pandas as pd — pause here and say what this line reads, creates, changes, or displays.
2. df = pd.DataFrame({"score": [1, 2, 3], "group": ["a", "a", "b"]}) — pause here and say what this line reads, creates, changes, or displays.
3. print(df.shape) — pause here and say what this line reads, creates, changes, or displays.
4. print(df.dtypes) — pause here and say what this line reads, creates, changes, or displays.
5. print(df.describe()) — pause here and say what this line reads, creates, changes, or displays.

After running the example, compare the actual output with your prediction. If they differ, do not erase your prediction. The difference is the part that can teach you the most.

Challenge

NotePractice

Change one input value, predict the new output, run the code, and explain the difference in one sentence.

Show one possible solution path

1. Copy Example 1 into Colab, Jupyter, or a .py file.
2. Mark the line you plan to change.
3. Write a one-sentence prediction.
4. Run the changed code.
5. If the result surprises you, restore the original and change a smaller part.

The goal is not to find the only correct answer. The goal is to create a small experiment where you can explain cause and effect.

Common mistakes

WarningCommon mistake

Assuming column types from appearance is risky. Always inspect dtypes.

When you get stuck, use this debugging routine:

1. Read the last line of the error message or inspect the unexpected output.
2. Find the smallest line of code that could be responsible.
3. Print or inspect the value and type at that point.
4. Change one thing.
5. Run again and record what changed.

Check your understanding

This quiz checks the ideas in this section before you move on.

{
  const scripts = Array.from(
    document.querySelectorAll("script.fcpython-ojs-quiz-config")
  );
  const script = scripts.find(
    (node) => node.dataset.fcpythonRendered !== "true"
  );

  if (!script) {
    return html`<div class="fcpython-quiz fcpython-quiz-warning">
      Quiz configuration was not found.
    </div>`;
  }

  script.dataset.fcpythonRendered = "true";
  const quiz = JSON.parse(script.textContent);
  const container = html`<div class="fcpython-quiz"></div>`;
  const title = document.createElement("h3");
  title.textContent = quiz.title;
  container.appendChild(title);

  const instructions = document.createElement("p");
  instructions.textContent = quiz.instructions;
  container.appendChild(instructions);

  const feedbackNodes = [];

  quiz.questions.forEach((question, questionIndex) => {
    const fieldset = document.createElement("fieldset");
    fieldset.className = "fcpython-quiz-question";

    const legend = document.createElement("legend");
    legend.textContent = `${questionIndex + 1}. ${question.prompt}`;
    fieldset.appendChild(legend);

    question.options.forEach((option, optionIndex) => {
      const label = document.createElement("label");
      label.className = "fcpython-quiz-option";

      const input = document.createElement("input");
      input.type = "radio";
      input.name = `${quiz.id}-${question.id}`;
      input.value = String(optionIndex);

      const text = document.createElement("span");
      text.textContent = option;

      label.appendChild(input);
      label.appendChild(text);
      fieldset.appendChild(label);
    });

    const feedback = document.createElement("p");
    feedback.className = "fcpython-quiz-feedback";
    feedback.setAttribute("aria-live", "polite");
    feedbackNodes.push(feedback);
    fieldset.appendChild(feedback);
    container.appendChild(fieldset);
  });

  const actions = document.createElement("div");
  actions.className = "fcpython-quiz-actions";

  const check = document.createElement("button");
  check.type = "button";
  check.textContent = "Check answers";

  const reset = document.createElement("button");
  reset.type = "button";
  reset.textContent = "Reset";

  const score = document.createElement("p");
  score.className = "fcpython-quiz-score";
  score.setAttribute("aria-live", "polite");

  check.addEventListener("click", () => {
    let correctCount = 0;

    quiz.questions.forEach((question, questionIndex) => {
      const selected = container.querySelector(
        `input[name="${quiz.id}-${question.id}"]:checked`
      );
      const feedback = feedbackNodes[questionIndex];

      if (!selected) {
        feedback.textContent = "Choose an answer before checking.";
        feedback.className = "fcpython-quiz-feedback";
        return;
      }

      const selectedIndex = Number(selected.value);
      if (selectedIndex === question.answer_index) {
        correctCount += 1;
        feedback.textContent = `✅ Correct. ${question.explanation}`;
        feedback.className = "fcpython-quiz-feedback is-correct";
      } else {
        const answer = question.options[question.answer_index];
        feedback.textContent = `❌ Not yet. Correct answer: ${answer}. ${question.explanation}`;
        feedback.className = "fcpython-quiz-feedback is-incorrect";
      }
    });

    score.textContent = `Score: ${correctCount}/${quiz.questions.length}`;
  });

  reset.addEventListener("click", () => {
    container.querySelectorAll("input[type='radio']").forEach((input) => {
      input.checked = false;
    });
    feedbackNodes.forEach((feedback) => {
      feedback.textContent = "";
      feedback.className = "fcpython-quiz-feedback";
    });
    score.textContent = "";
  });

  actions.appendChild(check);
  actions.appendChild(reset);
  container.appendChild(actions);
  container.appendChild(score);
  return container;
}

2. Selecting and Filtering Data

Choose columns, rows, and conditions from a DataFrame.

TipAnalogy

Filtering is like using a sieve: keep rows that match the condition and let others pass away.

What this means

Selection narrows a table to relevant columns or rows.

Example 2

Predict what will happen before you run the code.

import pandas as pd

df = pd.DataFrame({"name": ["Ada", "Grace"], "score": [95, 88]})
high = df[df["score"] >= 90]
print(high[["name", "score"]])

Step-by-step explanation

1. import pandas as pd — pause here and say what this line reads, creates, changes, or displays.
2. df = pd.DataFrame({"name": ["Ada", "Grace"], "score": [95, 88]}) — pause here and say what this line reads, creates, changes, or displays.
3. high = df[df["score"] >= 90] — pause here and say what this line reads, creates, changes, or displays.
4. print(high[["name", "score"]]) — pause here and say what this line reads, creates, changes, or displays.

After running the example, compare the actual output with your prediction. If they differ, do not erase your prediction. The difference is the part that can teach you the most.

Challenge

NotePractice

Change one input value, predict the new output, run the code, and explain the difference in one sentence.

Show one possible solution path

1. Copy Example 1 into Colab, Jupyter, or a .py file.
2. Mark the line you plan to change.
3. Write a one-sentence prediction.
4. Run the changed code.
5. If the result surprises you, restore the original and change a smaller part.

The goal is not to find the only correct answer. The goal is to create a small experiment where you can explain cause and effect.

Common mistakes

WarningCommon mistake

Use parentheses around combined conditions in pandas: (a) & (b), not a and b.

When you get stuck, use this debugging routine:

1. Read the last line of the error message or inspect the unexpected output.
2. Find the smallest line of code that could be responsible.
3. Print or inspect the value and type at that point.
4. Change one thing.
5. Run again and record what changed.

Check your understanding

This quiz checks the ideas in this section before you move on.

{
  const scripts = Array.from(
    document.querySelectorAll("script.fcpython-ojs-quiz-config")
  );
  const script = scripts.find(
    (node) => node.dataset.fcpythonRendered !== "true"
  );

  if (!script) {
    return html`<div class="fcpython-quiz fcpython-quiz-warning">
      Quiz configuration was not found.
    </div>`;
  }

  script.dataset.fcpythonRendered = "true";
  const quiz = JSON.parse(script.textContent);
  const container = html`<div class="fcpython-quiz"></div>`;
  const title = document.createElement("h3");
  title.textContent = quiz.title;
  container.appendChild(title);

  const instructions = document.createElement("p");
  instructions.textContent = quiz.instructions;
  container.appendChild(instructions);

  const feedbackNodes = [];

  quiz.questions.forEach((question, questionIndex) => {
    const fieldset = document.createElement("fieldset");
    fieldset.className = "fcpython-quiz-question";

    const legend = document.createElement("legend");
    legend.textContent = `${questionIndex + 1}. ${question.prompt}`;
    fieldset.appendChild(legend);

    question.options.forEach((option, optionIndex) => {
      const label = document.createElement("label");
      label.className = "fcpython-quiz-option";

      const input = document.createElement("input");
      input.type = "radio";
      input.name = `${quiz.id}-${question.id}`;
      input.value = String(optionIndex);

      const text = document.createElement("span");
      text.textContent = option;

      label.appendChild(input);
      label.appendChild(text);
      fieldset.appendChild(label);
    });

    const feedback = document.createElement("p");
    feedback.className = "fcpython-quiz-feedback";
    feedback.setAttribute("aria-live", "polite");
    feedbackNodes.push(feedback);
    fieldset.appendChild(feedback);
    container.appendChild(fieldset);
  });

  const actions = document.createElement("div");
  actions.className = "fcpython-quiz-actions";

  const check = document.createElement("button");
  check.type = "button";
  check.textContent = "Check answers";

  const reset = document.createElement("button");
  reset.type = "button";
  reset.textContent = "Reset";

  const score = document.createElement("p");
  score.className = "fcpython-quiz-score";
  score.setAttribute("aria-live", "polite");

  check.addEventListener("click", () => {
    let correctCount = 0;

    quiz.questions.forEach((question, questionIndex) => {
      const selected = container.querySelector(
        `input[name="${quiz.id}-${question.id}"]:checked`
      );
      const feedback = feedbackNodes[questionIndex];

      if (!selected) {
        feedback.textContent = "Choose an answer before checking.";
        feedback.className = "fcpython-quiz-feedback";
        return;
      }

      const selectedIndex = Number(selected.value);
      if (selectedIndex === question.answer_index) {
        correctCount += 1;
        feedback.textContent = `✅ Correct. ${question.explanation}`;
        feedback.className = "fcpython-quiz-feedback is-correct";
      } else {
        const answer = question.options[question.answer_index];
        feedback.textContent = `❌ Not yet. Correct answer: ${answer}. ${question.explanation}`;
        feedback.className = "fcpython-quiz-feedback is-incorrect";
      }
    });

    score.textContent = `Score: ${correctCount}/${quiz.questions.length}`;
  });

  reset.addEventListener("click", () => {
    container.querySelectorAll("input[type='radio']").forEach((input) => {
      input.checked = false;
    });
    feedbackNodes.forEach((feedback) => {
      feedback.textContent = "";
      feedback.className = "fcpython-quiz-feedback";
    });
    score.textContent = "";
  });

  actions.appendChild(check);
  actions.appendChild(reset);
  container.appendChild(actions);
  container.appendChild(score);
  return container;
}

3. Cleaning Missing Values

Find and handle missing data deliberately.

TipAnalogy

Missing values are blank cells on a form: you must decide whether to ask again, fill carefully, or exclude.

What this means

Missing data are absent values that can affect calculations and conclusions.

Example 3

Predict what will happen before you run the code.

import pandas as pd

df = pd.DataFrame({"score": [10, None, 30]})
print(df.isna().sum())
print(df.dropna())

Step-by-step explanation

1. import pandas as pd — pause here and say what this line reads, creates, changes, or displays.
2. df = pd.DataFrame({"score": [10, None, 30]}) — pause here and say what this line reads, creates, changes, or displays.
3. print(df.isna().sum()) — pause here and say what this line reads, creates, changes, or displays.
4. print(df.dropna()) — pause here and say what this line reads, creates, changes, or displays.

After running the example, compare the actual output with your prediction. If they differ, do not erase your prediction. The difference is the part that can teach you the most.

Challenge

NotePractice

Change one input value, predict the new output, run the code, and explain the difference in one sentence.

Show one possible solution path

1. Copy Example 1 into Colab, Jupyter, or a .py file.
2. Mark the line you plan to change.
3. Write a one-sentence prediction.
4. Run the changed code.
5. If the result surprises you, restore the original and change a smaller part.

The goal is not to find the only correct answer. The goal is to create a small experiment where you can explain cause and effect.

Common mistakes

WarningCommon mistake

Dropping missing rows can bias results. Explain your cleaning decision.

When you get stuck, use this debugging routine:

1. Read the last line of the error message or inspect the unexpected output.
2. Find the smallest line of code that could be responsible.
3. Print or inspect the value and type at that point.
4. Change one thing.
5. Run again and record what changed.

Check your understanding

This quiz checks the ideas in this section before you move on.

{
  const scripts = Array.from(
    document.querySelectorAll("script.fcpython-ojs-quiz-config")
  );
  const script = scripts.find(
    (node) => node.dataset.fcpythonRendered !== "true"
  );

  if (!script) {
    return html`<div class="fcpython-quiz fcpython-quiz-warning">
      Quiz configuration was not found.
    </div>`;
  }

  script.dataset.fcpythonRendered = "true";
  const quiz = JSON.parse(script.textContent);
  const container = html`<div class="fcpython-quiz"></div>`;
  const title = document.createElement("h3");
  title.textContent = quiz.title;
  container.appendChild(title);

  const instructions = document.createElement("p");
  instructions.textContent = quiz.instructions;
  container.appendChild(instructions);

  const feedbackNodes = [];

  quiz.questions.forEach((question, questionIndex) => {
    const fieldset = document.createElement("fieldset");
    fieldset.className = "fcpython-quiz-question";

    const legend = document.createElement("legend");
    legend.textContent = `${questionIndex + 1}. ${question.prompt}`;
    fieldset.appendChild(legend);

    question.options.forEach((option, optionIndex) => {
      const label = document.createElement("label");
      label.className = "fcpython-quiz-option";

      const input = document.createElement("input");
      input.type = "radio";
      input.name = `${quiz.id}-${question.id}`;
      input.value = String(optionIndex);

      const text = document.createElement("span");
      text.textContent = option;

      label.appendChild(input);
      label.appendChild(text);
      fieldset.appendChild(label);
    });

    const feedback = document.createElement("p");
    feedback.className = "fcpython-quiz-feedback";
    feedback.setAttribute("aria-live", "polite");
    feedbackNodes.push(feedback);
    fieldset.appendChild(feedback);
    container.appendChild(fieldset);
  });

  const actions = document.createElement("div");
  actions.className = "fcpython-quiz-actions";

  const check = document.createElement("button");
  check.type = "button";
  check.textContent = "Check answers";

  const reset = document.createElement("button");
  reset.type = "button";
  reset.textContent = "Reset";

  const score = document.createElement("p");
  score.className = "fcpython-quiz-score";
  score.setAttribute("aria-live", "polite");

  check.addEventListener("click", () => {
    let correctCount = 0;

    quiz.questions.forEach((question, questionIndex) => {
      const selected = container.querySelector(
        `input[name="${quiz.id}-${question.id}"]:checked`
      );
      const feedback = feedbackNodes[questionIndex];

      if (!selected) {
        feedback.textContent = "Choose an answer before checking.";
        feedback.className = "fcpython-quiz-feedback";
        return;
      }

      const selectedIndex = Number(selected.value);
      if (selectedIndex === question.answer_index) {
        correctCount += 1;
        feedback.textContent = `✅ Correct. ${question.explanation}`;
        feedback.className = "fcpython-quiz-feedback is-correct";
      } else {
        const answer = question.options[question.answer_index];
        feedback.textContent = `❌ Not yet. Correct answer: ${answer}. ${question.explanation}`;
        feedback.className = "fcpython-quiz-feedback is-incorrect";
      }
    });

    score.textContent = `Score: ${correctCount}/${quiz.questions.length}`;
  });

  reset.addEventListener("click", () => {
    container.querySelectorAll("input[type='radio']").forEach((input) => {
      input.checked = false;
    });
    feedbackNodes.forEach((feedback) => {
      feedback.textContent = "";
      feedback.className = "fcpython-quiz-feedback";
    });
    score.textContent = "";
  });

  actions.appendChild(check);
  actions.appendChild(reset);
  container.appendChild(actions);
  container.appendChild(score);
  return container;
}

Notebook and Colab practice

Open a blank notebook at https://colab.new. Use one section at a time: copy the Example 1, predict the result, run it, answer the section quiz, and then move to the next section. This is better than copying the entire page at once.

Instructor note

Teaching notes

• Treat each section as a short teaching episode.
• Pause for the section quiz before introducing the next section.
• Ask learners to compare sections: what stayed the same, and what changed?
• If time is short, teach the first two sections live and assign the rest as practice.

Key points

TipKey points

• Inspecting DataFrames: Inspection reveals structure before you transform or model data.
• Selecting and Filtering Data: Selection narrows a table to relevant columns or rows.
• Cleaning Missing Values: Missing data are absent values that can affect calculations and conclusions.
• Use the section quizzes as gates: review before moving on if a quiz feels uncertain.

References

• pandas documentation: https://pandas.pydata.org/docs/
• seaborn documentation: https://seaborn.pydata.org/
• Matplotlib documentation: https://matplotlib.org/stable/
• Python Tutorial: https://docs.python.org/3/tutorial/
• Quarto OJS documentation: https://quarto.org/docs/interactive/ojs/
• ipywidgets documentation: https://ipywidgets.readthedocs.io/en/stable/