Hypothesis Tests

Null and alternative hypotheses, test statistics, p-values, significance, and errors.

A hypothesis test asks whether observed data are surprising under a baseline claim. Tests do not prove truth. They organize evidence, probability, and a pre-chosen decision rule.

Learning objectives

After this lesson you should be able to:

  • write null and alternative hypotheses;
  • compute a simple z test statistic;
  • interpret p-values correctly;
  • make a reject/fail-to-reject decision using a significance level;
  • distinguish statistical significance from practical importance;
  • identify Type I and Type II errors.

The logic of a test

A hypothesis test begins by temporarily assuming the null hypothesis is true. Then it asks:

If the null hypothesis were true, how unusual would our data be?

If the data would be very unusual under the null, we have evidence against the null.

Null and alternative hypotheses

The null hypothesis H0 is the baseline claim. It often says no effect, no difference, or a known value.

The alternative hypothesis Ha is what we look for evidence toward. It can be two-sided or one-sided.

Examples:

H0: mu = 100
Ha: mu != 100
H0: p = 0.50
Ha: p > 0.50

Test statistic

A test statistic measures how far the estimate is from the null value in standard error units.

z = (estimate - null value) / standard error

A large positive or negative statistic means the estimate is far from the null value relative to expected sampling variability.

P-values

A p-value is the probability, assuming the null hypothesis is true, of obtaining a result at least as extreme as the observed one.

A p-value is not:

  • the probability that the null hypothesis is true;
  • the probability that the alternative hypothesis is true;
  • the size of the effect;
  • a guarantee that the result matters practically.

Significance level

The significance level alpha is chosen before the test, often 0.05.

Decision rule:

if p-value <= alpha: reject H0
if p-value > alpha: fail to reject H0

“Fail to reject” does not mean “prove the null.” It means the data did not provide strong enough evidence against the null using that test and alpha.

Type I and Type II errors

Decision Reality Error?
reject H0 H0 true Type I error
fail to reject H0 H0 false Type II error

A Type I error is a false positive. A Type II error is a false negative.

Worked example 1: one-sample z test

A process is supposed to have mean 100. A random sample gives mean 105, and the standard error is 2.5. Test whether the mean is greater than 100.

Step 1: State hypotheses.

H0: mu = 100
Ha: mu > 100

Step 2: Compute the z statistic.

z = (105 - 100) / 2.5
z = 5 / 2.5
z = 2

Step 3: Interpret.

The sample mean is 2 standard errors above the null value.

Step 4: Use a p-value.

For a greater-than test, z=2 gives p-value about 0.023.

Step 5: Make a decision at alpha 0.05.

Since 0.023 <= 0.05, reject H0.

Conclusion: The data provide evidence that the process mean is greater than 100, assuming the test conditions are appropriate.

Worked example 2: interpret a p-value

A test gives p-value 0.18.

Incorrect interpretation: There is an 18% chance the null is true.

Correct interpretation: If the null hypothesis were true, results this extreme or more extreme would happen about 18% of the time. That is not very surprising, so at alpha 0.05 we fail to reject the null.

Worked example 3: significance versus importance

A huge study finds a treatment improves average score by 0.2 points with p-value 0.001.

The result is statistically significant, but the effect may be too small to matter in practice. Always ask:

How large is the effect in real units?

Common pitfalls

  • Saying the p-value is the probability the null is true.
  • Choosing alpha after seeing the p-value.
  • Treating “fail to reject” as proof that the null is true.
  • Ignoring practical importance.
  • Running many tests and reporting only the significant ones.
  • Forgetting to check study design and assumptions.

Practice exercises

  1. Write hypotheses for testing whether a population mean differs from 50.
  2. Compute z when estimate is 42, null value is 40, and standard error is 1.
  3. At alpha 0.05, what decision is made for p-value 0.03?
  4. At alpha 0.05, what decision is made for p-value 0.20?
  5. What is a Type I error?
  6. Why can a tiny p-value still describe an unimportant effect?
  1. H0: mu = 50; Ha: mu != 50.
  2. z=(42-40)/1=2.
  3. Reject H0.
  4. Fail to reject H0.
  5. Rejecting a true null hypothesis; a false positive.
  6. A very large sample can detect a very small effect that has little practical importance.

Subtopic guided practice and checkpoints

Lab 1: write hypotheses

Guess first. If the question says “has increased,” is the alternative usually > or !=?

Guided exercise.

“Increased” is directional, so the alternative is usually >.

Checkpoint. Write the alternative for “the mean is less than 20.”

Lab 2: compute a statistic

Guess first. If an estimate is exactly equal to the null value, what is the z statistic?

Guided exercise.

z = (estimate - null) / SE = 0 / SE = 0

Checkpoint. Compute z for estimate 12, null 10, and SE=4.

Lab 3: interpret a p-value

Guess first. Is a smaller p-value stronger or weaker evidence against H0?

Guided exercise.

Smaller p-values mean the observed result is more surprising under H0, so they are stronger evidence against H0.

Checkpoint. At alpha 0.01, is p-value 0.03 significant?

Lab 4: classify errors

Guess first. Is a false positive Type I or Type II?

Guided exercise.

A false positive means rejecting a true null hypothesis. That is Type I error.

Checkpoint. What kind of error is failing to reject a false null?

  1. Ha: mu < 20.
  2. z=(12-10)/4=0.5.
  3. No. 0.03 > 0.01, so fail to reject at alpha 0.01.
  4. Type II error.

Hypothesis testing guessing game

Using this lesson with edumath and SymPy

Use edumath for z statistics and normal-approximation p-values.

from edumath.statistics import one_sample_z_test, z_test_statistic

z_test_statistic(105, 100, 2.5)
2.0
one_sample_z_test(105, 100, 2.5, alternative="greater")
HypothesisTestResult(statistic=2.0, p_value=0.02275013194817921, alternative='greater')

Use SymPy to rearrange the z-statistic formula.

import sympy as sp

z, estimate, null, se = sp.symbols("z estimate null se")
sp.solve(sp.Eq(z, (estimate - null) / se), estimate)
[null + se*z]