Lesson 14: Significance Testing

October 14, 2025

Review:

• Small Sample Estimation
• Exam 2 on Thu, Oct 23

Presentation:

• Significance Testing
  • aka Hypothesis Testing
  • Purpose: to evaluate data for evidence of significant agreement or disagreement
• 4-step Testing Process
  • Step 1. Setup the null hypothesis (Ho) and alternate hypothesis (Ha)
  • Step 2. Calculate the appropriate test statistic
  • Step 3. Find the P-value (probability of obtaining result by chance)
  • Step 4. Compare P-value to α = 0.05 ; if P-value < 0.05, “Reject Ho” else “Fail to Reject Ho”

• Significance testing is like a courtroom trial.
  • Null Hypothesis (Ho): The defendant is innocent until proven guilty.
    → We start by assuming there’s no effect, no difference, no relationship.
  • Alternative Hypothesis (Ha): The defendant is guilty.
    → We’re testing whether there’s enough evidence to reject the assumption of innocence.
  • Evidence (Data): The prosecution presents data—witness testimony, forensics, etc.
    → In statistics, we gather a sample and compute a test statistic (like a z-score).
  • Judge or Jury (Decision Rule): They set a high standard for conviction (e.g., beyond a reasonable doubt).
    → In statistics, this is our significance level (α), often 0.05, meaning we accept at most a 5% chance of wrongly rejecting H₀.
  • Verdict (Statistical Decision):
    • If the evidence is strong enough (p < α), we reject H₀ — “guilty” verdict.
    • If not (p ≥ α), we fail to reject H₀ — “not guilty” (but not necessarily innocent).
  • Type I Error: Convicting an innocent person.
    → Rejecting a true null hypothesis (false positive).
  • Type II Error: Letting a guilty person go free.
    → Failing to reject a false null hypothesis (false negative).

• Example A: Is the soda company short-changing customers?
  • A soda company claims that its cans contain 12.0 oz on average.
  • You take a sample of n=30 cans and find:
  • sample mean = 11.8 oz
  • sample standard deviation = 0.3 oz
• Step 1. Set up hypotheses
  • Ho: μ = 12.0 (the cans are filled correctly)
  • Ha: μ < 12.0 (the cans are underfilled)
• Step 2. Compute the Test Statistic
  • Z-test = (x̅ – μ)/(σ/√n)
  • (11.8 – 12.0)/(0.3/√30) = -0.2/0.05477 = -3.65
• Step 3. Find the p-Value
  • p=P(Z<−3.65)≈0.00013
• Step 4: Interpret Results
  • 0.00013 ≅ 0.013% probability of getting this result by chance
  • 0.00013 < 0.05
  • Reject Ho
  • The soda company seems to be ripping us off!

• Significance Testing video (unit 25)
• Example B: Do Math SAT scores improve significantly with coaching?
  • National Math SAT scores are normally distributed with mean = 505 and stdev = 62.
  • Sample of 1,000 students who received coaching.
  • Sample mean score was 509.
  • Are these results significantly better than the national average?
• Step 1: Setup the hypothesis test
  • μ = 505, σ = 62, x̅ = 509, n = 1,000
    • Ho: μ = 505
      Ha: μ > 505
• Step 2: Calculate the test statistic
  • Z-test = (x̅ – μ)/(σ/√n)
  • (509 – 505)/(62/√1000) = 4/1.96 = 2.04
• Step 3: Find P-value
  • P = 1 – P(Z<2.04) = 1 – 0.9793 = 0.0207
• Step 4: Interpret results
  • 0.0207 ≅ 2.07% probability of getting this result by chance
  • 0.0207 < 0.05
  • Reject Ho
  • Coaching seems to improve scores significantly

Activity:

• Problem 1.1. More than 200,000 people worldwide take the GMAT examination each year as they apply for MBA programs. Their scores vary Normally with mean about μ = 525 and standard deviation about σ = 100. One hundred students, n = 100, go through a rigorous training program designed to raise their GMAT scores. The students who go through the program have an average score of x̅ = 541.4. Is there evidence to suggest the training program significantly improves GMAT scores?
• Problem 1.2. A newly installed rooftop solar system has been producing energy for n = 100 days. Average energy production is 41.8 kWh per day with a standard deviation of 13.9 kWh. The solar panel manufacturer claims the panels typically produce 40 kWh per day. Is the newly installed system producing significantly more energy than estimated by the manufacturer?

* Most example and activity problems presented above are derived from Moore, D.S., McCabe, G.P., and Craig, B.A., 2009. Introduction to the Practice of Statistics, 6th Edition. New York: W.H. Freeman and Company.