Transformers Overview

Understand transformer models and why they matter for modern AI.

• Level: Beginner to intermediate
• Estimated time: 25–40 minutes
• You will learn: Understand transformer models and why they matter for modern AI.
• Practice in: Jupyter or Google Colab

Open this lesson in Google Colab

Questions

• What problem does Transformers Overview help us solve in a small Python program?
• What should we predict before running the example?
• What value, output, or error should we inspect after changing one line?

Objectives

• Run a complete example for transformers in Colab.
• Explain the example line by line using plain language.
• Change one part of the code and predict the result before running it.
• Recognize one common mistake and use the error message as evidence.

Hands-on episode: Transformers Overview

A transformer processes token sequences using embeddings, attention, and layers that update contextual representations. The output can be useful, but it is not automatically verified truth.

We will learn this by running code, not by memorizing a definition first. Open the Colab notebook from the button above, find this section, and run each cell in order. Keep a small note beside the notebook with three columns: prediction, actual result, and what changed.

Example 1.1

Think through tokenization before thinking about the model’s answer.

text = "Python helps learners practice."
tokens = text.split()
print(tokens)
print(len(tokens))

Run the cell once without editing it. If the result is different from your prediction, leave the prediction visible and write one sentence about the difference. That sentence is more useful than a perfect first guess.

Explain Example 1.1

• This simple split is not how production tokenizers work, but it introduces the idea of breaking text into pieces.
• The model does not receive a paragraph as human meaning directly; it receives token IDs derived from text.
• The number of tokens matters because models have context limits.

Now explain the example out loud or in a Markdown cell. Use short sentences: “this line creates…”, “this name stores…”, “this output appears because…”. If you cannot explain a line yet, run only the lines above it and inspect the values that exist at that moment.

Challenge 1.1

NoteChallenge

Change the punctuation or spacing and inspect the pieces. Real tokenizers are more complex, but the beginner lesson is that representation choices affect model input.

Show a safe way to approach the challenge

1. Copy Example 1.1 into a new Colab cell.
2. Change exactly one value, name, condition, or line.
3. Write the expected output before running the cell.
4. Run the cell and compare the actual result with your prediction.
5. If the result surprises you, undo the change and try a smaller one.

Suggested first move: Change the punctuation or spacing and inspect the pieces.

Debugging checkpoint 1.1

WarningDebugging checkpoint

Do not assume generated text is verified just because it sounds fluent. Check sources, protect private data, and remember that prompts can change outputs in surprising ways.

Do not debug by rewriting the whole example. Read the error type or surprising output, inspect the closest value with print(...) or type(...), then change one thing. This is the same routine you will use in larger projects.

Apply it

Compare two prompts for the same task. Before judging the answers, write what context each prompt gave the model and what ambiguity remained.

Finish by adding a Markdown cell that answers: What did this example teach me that I can reuse in a project?

Key points

• Learn the concept by running a complete, small example first.
• Predict before execution so your thinking becomes visible.
• Change one thing at a time so cause and effect stay clear.
• Treat errors as clues about the exact line or value Python could not handle.

Why this matters

Understand transformer models and why they are important for modern language and multimodal AI.

Beginners often try to learn Python as a list of commands. This lesson teaches one idea at a time: what problem it solves, how to recognize it in code, how to practice it, and how to debug a common mistake. Treat the code examples as small experiments, not as text to memorize.

NoteGuiding questions

By the end of this lesson, you should be able to answer:

• What problem does Transformers Overview help me solve?
• What words or symbols should I recognize in a small example?
• What should I check first when the example does not behave as expected?

NoteLearning objectives

You will practice how to:

• explain the main idea in everyday language;
• read a short example line by line;
• predict the result before running the code;
• make one safe variation of the example;
• debug one common beginner mistake.

TipAnalogy

Attention is like reading a sentence with highlighters: the model learns which words should influence each other.

Vocabulary

• transformer — a key term for this lesson; after the example, write a one-sentence definition in your own words.
• token — a key term for this lesson; after the example, write a one-sentence definition in your own words.
• attention — a key term for this lesson; after the example, write a one-sentence definition in your own words.
• embedding — a key term for this lesson; after the example, write a one-sentence definition in your own words.
• model — a key term for this lesson; after the example, write a one-sentence definition in your own words.

What this means

Transformers are neural network architectures that learn relationships between tokens using attention.

A useful explanation has three parts:

1. Name the thing. Say what concept you are using.
2. Name the input. Identify the values, files, objects, or settings involved.
3. Name the result. Explain what changes, what is returned, or what is printed.

Example 1

Predict what will happen before you run the code.

tokens = ["Python", "is", "useful"]
attention_question = "Which tokens are most relevant to the next prediction?"
print(tokens, attention_question)

Step-by-step explanation

1. tokens = ["Python", "is", "useful"] — pause here and say what this line reads, creates, changes, or displays.
2. attention_question = "Which tokens are most relevant to the next predic... — pause here and say what this line reads, creates, changes, or displays.
3. print(tokens, attention_question) — pause here and say what this line reads, creates, changes, or displays.

After running the example, compare the actual output with your prediction. If they differ, do not erase your prediction. The difference is the part that can teach you the most.

Worked example: read, predict, modify

Use this routine with every example in the course:

Step	What to do	Why it helps
Read	Point to each name, value, and operator.	Slows the code down enough to understand it.
Predict	Write what you think will happen.	Creates a testable prediction.
Run	Execute the smallest complete example.	Lets Python give evidence.
Explain	Say what happened in plain language.	Converts recognition into understanding.
Modify	Change one small thing and run again.	Shows which part caused which result.

Challenge

NotePractice

Change one input value, predict the new output, run the code, and explain the difference in one sentence.

Show one possible solution path

1. Copy Example 1 into Colab, Jupyter, or a .py file.
2. Mark the line you plan to change.
3. Write a one-sentence prediction.
4. Run the changed code.
5. If the result surprises you, restore the original and change a smaller part.

The goal is not to find the only correct answer. The goal is to create a small experiment where you can explain cause and effect.

Common mistakes

WarningCommon mistake

Using a transformer model does not remove the need to understand data, evaluation, bias, and privacy.

When you get stuck, use this debugging routine:

1. Read the last line of the error message or inspect the unexpected output.
2. Find the smallest line of code that could be responsible.
3. Print or inspect the value and type at that point.
4. Change one thing.
5. Run again and record what changed.

Checkpoint quiz

Answer these questions before moving on. The quiz runs with Quarto OJS in the browser, so it does not need a Python kernel during website rendering.

{
  const scripts = Array.from(
    document.querySelectorAll("script.fcpython-ojs-quiz-config")
  );
  const script = scripts.find(
    (node) => node.dataset.fcpythonRendered !== "true"
  );

  if (!script) {
    return html`<div class="fcpython-quiz fcpython-quiz-warning">
      Quiz configuration was not found.
    </div>`;
  }

  script.dataset.fcpythonRendered = "true";
  const quiz = JSON.parse(script.textContent);
  const container = html`<div class="fcpython-quiz"></div>`;
  const title = document.createElement("h3");
  title.textContent = quiz.title;
  container.appendChild(title);

  const instructions = document.createElement("p");
  instructions.textContent = quiz.instructions;
  container.appendChild(instructions);

  const feedbackNodes = [];

  quiz.questions.forEach((question, questionIndex) => {
    const fieldset = document.createElement("fieldset");
    fieldset.className = "fcpython-quiz-question";

    const legend = document.createElement("legend");
    legend.textContent = `${questionIndex + 1}. ${question.prompt}`;
    fieldset.appendChild(legend);

    question.options.forEach((option, optionIndex) => {
      const label = document.createElement("label");
      label.className = "fcpython-quiz-option";

      const input = document.createElement("input");
      input.type = "radio";
      input.name = `${quiz.id}-${question.id}`;
      input.value = String(optionIndex);

      const text = document.createElement("span");
      text.textContent = option;

      label.appendChild(input);
      label.appendChild(text);
      fieldset.appendChild(label);
    });

    const feedback = document.createElement("p");
    feedback.className = "fcpython-quiz-feedback";
    feedback.setAttribute("aria-live", "polite");
    feedbackNodes.push(feedback);
    fieldset.appendChild(feedback);
    container.appendChild(fieldset);
  });

  const actions = document.createElement("div");
  actions.className = "fcpython-quiz-actions";

  const check = document.createElement("button");
  check.type = "button";
  check.textContent = "Check answers";

  const reset = document.createElement("button");
  reset.type = "button";
  reset.textContent = "Reset";

  const score = document.createElement("p");
  score.className = "fcpython-quiz-score";
  score.setAttribute("aria-live", "polite");

  check.addEventListener("click", () => {
    let correctCount = 0;

    quiz.questions.forEach((question, questionIndex) => {
      const selected = container.querySelector(
        `input[name="${quiz.id}-${question.id}"]:checked`
      );
      const feedback = feedbackNodes[questionIndex];

      if (!selected) {
        feedback.textContent = "Choose an answer before checking.";
        feedback.className = "fcpython-quiz-feedback";
        return;
      }

      const selectedIndex = Number(selected.value);
      if (selectedIndex === question.answer_index) {
        correctCount += 1;
        feedback.textContent = `✅ Correct. ${question.explanation}`;
        feedback.className = "fcpython-quiz-feedback is-correct";
      } else {
        const answer = question.options[question.answer_index];
        feedback.textContent = `❌ Not yet. Correct answer: ${answer}. ${question.explanation}`;
        feedback.className = "fcpython-quiz-feedback is-incorrect";
      }
    });

    score.textContent = `Score: ${correctCount}/${quiz.questions.length}`;
  });

  reset.addEventListener("click", () => {
    container.querySelectorAll("input[type='radio']").forEach((input) => {
      input.checked = false;
    });
    feedbackNodes.forEach((feedback) => {
      feedback.textContent = "";
      feedback.className = "fcpython-quiz-feedback";
    });
    score.textContent = "";
  });

  actions.appendChild(check);
  actions.appendChild(reset);
  container.appendChild(actions);
  container.appendChild(score);
  return container;
}

Notebook and Colab practice

Open a blank notebook at https://colab.new, copy Example 1, and run three small variations. You can also use a local Jupyter notebook or VS Code. Keep one cell for the original example, one cell for your prediction, and one cell for your modified version.

Instructor note

Teaching notes

• Ask learners to predict before execution; do not skip this step.
• Invite one learner to explain the analogy and another to explain the code.
• When an error appears, model calm traceback reading instead of immediately fixing it.
• If time is short, keep Example 1 and quiz; move the challenge to homework.

Key points

TipKey points

• Transformers are neural network architectures that learn relationships between tokens using attention.
• Small examples are more useful than large copied programs when a concept is new.
• Prediction, execution, explanation, and one small modification form the core practice loop.
• Debugging starts by reading clues and changing one thing at a time.

References

• Hugging Face Transformers documentation: https://huggingface.co/docs/transformers/
• scikit-learn documentation: https://scikit-learn.org/stable/
• PyTorch documentation: https://pytorch.org/docs/stable/index.html
• TensorFlow documentation: https://www.tensorflow.org/api_docs
• Python Tutorial: https://docs.python.org/3/tutorial/
• Quarto OJS documentation: https://quarto.org/docs/interactive/ojs/
• ipywidgets documentation: https://ipywidgets.readthedocs.io/en/stable/