Regression Basics

Scatterplots, correlation, least-squares lines, residuals, prediction, and causation cautions.

Regression models relationships between variables. In simple linear regression, we use one quantitative explanatory variable x to predict one quantitative response variable y.

Learning objectives

After this lesson you should be able to:

  • read scatterplots for direction, form, strength, and outliers;
  • interpret correlation;
  • interpret slope and intercept in a simple linear regression line;
  • compute and interpret residuals;
  • use a regression line for prediction;
  • explain why association alone does not prove causation.

Scatterplots first

Before fitting a line, look at the data. A scatterplot can show:

  • direction: positive, negative, or no clear association;
  • form: linear or curved;
  • strength: tight or scattered pattern;
  • outliers: unusual points;
  • clusters: groups that may need separate attention.

A line is appropriate only when the relationship is roughly linear.

Correlation

Pearson correlation r measures the strength and direction of a linear relationship.

Properties:

  • r is between -1 and 1;
  • positive r means y tends to increase as x increases;
  • negative r means y tends to decrease as x increases;
  • r near 0 means little linear association;
  • correlation is unitless;
  • correlation is sensitive to outliers.

Regression line

A simple linear regression line has the form:

predicted y = intercept + slope * x

The slope is the predicted change in y for a one-unit increase in x. The intercept is the predicted y value when x=0, if that value is meaningful in context.

Residuals

A residual is the prediction error for one point:

residual = observed y - predicted y

Positive residual: observed value is above the line. Negative residual: observed value is below the line.

Residual plots help check whether a linear model is appropriate. A curved pattern in residuals suggests the line is missing structure.

Prediction and extrapolation

Interpolation means predicting within the range of observed x values. Extrapolation means predicting outside that range. Extrapolation can be risky because the relationship may not continue.

Causation caution

Regression describes association. It does not automatically prove that changing x causes a change in y. Causal conclusions require careful design, such as random assignment or strong observational methods.

Worked example 1: interpret a slope

A regression line predicts exam score from study hours:

predicted score = 62 + 4.5 * hours

Question: Interpret the slope.

Answer: For each additional hour studied, the model predicts an average increase of 4.5 exam points.

This is a prediction statement. It does not automatically prove that forcing one more hour of study causes exactly 4.5 more points for every student.

Worked example 2: make a prediction

Use the same line:

predicted score = 62 + 4.5 * hours

Predict score for 6 hours.

Step 1: Substitute.

predicted score = 62 + 4.5(6)

Step 2: Compute.

62 + 27 = 89

The predicted score is 89.

Worked example 3: compute a residual

A student studied 6 hours and scored 84. The model predicted 89.

residual = observed - predicted = 84 - 89 = -5

The residual is -5, meaning the student scored 5 points below the model’s prediction.

Worked example 4: fit a perfect line

Data:

x: 1, 2, 3
y: 3, 5, 7

Each time x increases by 1, y increases by 2, so slope is 2. Using point (1,3):

3 = intercept + 2(1)
intercept = 1

The line is:

predicted y = 1 + 2x

For x=4, predicted y=9.

Common pitfalls

  • Interpreting the intercept outside a meaningful range.
  • Using correlation for a curved relationship.
  • Ignoring outliers that strongly affect the line.
  • Treating a high correlation as proof of causation.
  • Extrapolating far beyond observed data.
  • Forgetting that residuals are observed minus predicted.

Practice exercises

  1. Interpret the slope in predicted y = 10 + 3x.
  2. Predict y when x=5 for predicted y = 10 + 3x.
  3. If observed y=22 and predicted y=25, find the residual.
  4. What does correlation near -1 mean?
  5. Why is extrapolation risky?
  6. Why does regression alone not prove causation?
  1. For each one-unit increase in x, predicted y increases by 3 units.
  2. 10 + 3(5) = 25.
  3. 22 - 25 = -3.
  4. A strong negative linear association.
  5. The relationship may change outside the observed range.
  6. Confounding, selection effects, or reverse causation may explain the association.

Subtopic guided practice and checkpoints

Lab 1: interpret direction

Guess first. If taller plants tend to have larger leaves, is the association positive or negative?

Guided exercise.

As height increases, leaf size tends to increase, so the association is positive.

Checkpoint. If price increases and demand decreases, what is the direction?

Lab 2: use the line

Guess first. In y = 5 + 2x, what is the predicted change in y when x increases by 1?

Guided exercise.

The slope is 2, so predicted y increases by 2.

Checkpoint. Predict y for x=4 in y=5+2x.

Lab 3: residual sign

Guess first. If a point is above the regression line, is the residual positive or negative?

Guided exercise.

Above the line means observed y is greater than predicted y, so the residual is positive.

Checkpoint. If observed is 12 and predicted is 15, find the residual.

Lab 4: causation language

Guess first. Can an observational regression prove causation by itself?

Guided exercise.

No. It can show association, but causal claims require design and reasoning about confounders.

Checkpoint. Name one reason association may not be causation.

  1. Negative.
  2. 5+2(4)=13.
  3. 12-15=-3.
  4. Confounding, reverse causation, selection effects, or coincidence.

Regression guessing game

Using this lesson with edumath and SymPy

Use edumath to fit a simple regression line and compute residuals.

from edumath.statistics import predict, residuals, simple_linear_regression

line = simple_linear_regression([1, 2, 3], [3, 5, 7])
line
RegressionLine(slope=2.0, intercept=1.0, correlation=1.0)
predict(line, 4)
9.0
residuals(line, [1, 2, 3], [3, 5, 7])
(0.0, 0.0, 0.0)

Use SymPy to solve for an intercept from one point and a known slope.

import sympy as sp

b = sp.symbols("b")
sp.solve(sp.Eq(3, b + 2 * 1), b)
[1]