Intermediate9 min readSource-backed

Introduction to Hypothesis Testing

Understand null and alternative hypotheses, Type I and Type II errors, the logic of p-values, and the difference between statistical and practical significance.

You'll learn

What p-values actually mean and how to avoid common misinterpretations.

Use this when

You run a test and need to interpret whether the result is significant.

The Logic of Hypothesis Testing

Hypothesis testing uses sample data to make decisions about a population. The procedure always follows the same structure: assume the null hypothesis is true, then determine how likely your observed data would be under that assumption.

1.State H₀ (null hypothesis): the "no effect" or "no difference" claim
2.State H₁ (alternative hypothesis): what you believe or hope to show
3.Choose a significance level α (usually 0.05)
4.Compute a test statistic from your data
5.Find the p-value: probability of getting a result this extreme if H₀ were true
6.Decision: if p < α, reject H₀; if p ≥ α, fail to reject H₀

"Failing to reject H₀" is NOT the same as proving H₀ is true. Absence of evidence is not evidence of absence.

Null and Alternative Hypotheses

The null hypothesis (H₀) represents the status quo or "no effect." The alternative hypothesis (H₁ or Hₐ) represents the effect you are testing for.

●H₀: "There is no difference in mean blood pressure between treatment groups" → μ₁ = μ₂
●H₁ (two-tailed): "Mean blood pressure differs between groups" → μ₁ ≠ μ₂
●H₁ (one-tailed): "Treatment group has lower blood pressure" → μ₁ < μ₂
●Use two-tailed tests unless you have a strong a priori reason for a directional hypothesis (and pre-specify it before data collection)

Type I and Type II Errors

Two types of errors can occur in hypothesis testing:

●Type I Error (False Positive, α): Rejecting H₀ when it is actually true. You conclude there is an effect when there is none. Controlled by your significance level α = 0.05 means a 5% chance of a Type I error.
●Type II Error (False Negative, β): Failing to reject H₀ when H₁ is actually true. You miss a real effect. Typically set at β = 0.20 (80% power).
●Power (1 - β): The probability of correctly detecting a real effect. High power = low Type II error rate.
●Tradeoff: Reducing α (e.g., to 0.01) decreases Type I error but increases Type II error (lower power) for a fixed sample size.

Think of it like a medical test: Type I = false positive (telling a healthy person they are sick); Type II = false negative (missing a sick person's disease).

Statistical vs. Practical Significance

Statistical significance (p < 0.05) does not imply practical or clinical importance. With a large enough sample, even trivial differences become statistically significant.

●Example: A new drug reduces systolic blood pressure by 1 mmHg (95% CI: 0.2–1.8, p = 0.01). Statistically significant, but 1 mmHg has no clinical relevance.
●Effect sizes (Cohen's d, η², OR) quantify practical significance independently of sample size.
●Always interpret results in context: a 5-minute reduction in surgery time is meaningless; 5 minutes off a cardiac arrest response is life-saving.

The Multiple Testing Problem

If you run 20 statistical tests at α = 0.05, you expect 1 false positive by chance alone. This is the multiple comparisons problem.

●Bonferroni correction: divide α by the number of tests. Testing 10 outcomes → use α = 0.005 per test.
●False Discovery Rate (FDR / Benjamini-Hochberg): less conservative alternative for exploratory research.
●Pre-specify your primary outcome before data collection to avoid this problem.
●In genetic studies (thousands of SNPs), significance threshold is often p < 5×10⁻⁸.