Hypothesis testing is a statistical method used to determine whether there is enough evidence in sample data to draw conclusions about a population. Instead of collecting data from an entire population, you take a sample and test whether the evidence supports or contradicts an assumption about that population.
What Is Null Hypothesis Significance Testing (NHST)?
Null Hypothesis Significance Testing (NHST) refers to one particular procedure: you begin by assuming the null hypothesis is true, collect data, compute a test statistic, and then ask how probable your observed result would be under that assumption.
The word null is key. It does not mean “zero”. It means the hypothesis of no effect, no difference, no relationship. Everything in NHST is organised around building a case against this default position.
In the social and biomedical sciences, we use NHST to ask things like:
- Does a new antidepressant actually reduce symptoms more than a placebo?
- Do boys and girls differ in academic self-efficacy?
- Does social isolation increase mortality risk?
Core Concepts and Key Terms
What are Null and Alternative Hypotheses?
The null hypothesis (H₀) is a statement of “no difference,” “no association,” or “no treatment effect.” The alternative hypothesis (Hₐ) is a statement of “difference,” “association,” or “treatment effect.” H₀ is assumed to be true until proven otherwise. However, Hₐ is the hypothesis the researcher hopes to bolster.
P value
The P value answers the question: “If the null hypothesis were true, what is the probability of observing the current data or data that is more extreme?” Note that the P value is NOT the probability that the hypothesis (or any other hypothesis) is right or wrong.
Significance Level (α)
The significance level (α) represents how sure we want to be before saying the claim is false. Usually, we choose 0.05 (5%). Choosing α = 0.05 means accepting a 5% chance of wrongly rejecting a true null hypothesis, i.e., a false alarm.
In biomedical contexts where a wrong decision could harm patients, researchers often set α = 0.01 or even 0.001.
One-Tailed vs. Two-Tailed Tests
A one-tailed test is used when we expect a change in only one direction: either up or down, but not both. A two-tailed test is used when we want to see if there is a difference in either direction, higher or lower.
| Test Type | When to Use | Example Hypothesis |
| Right-tailed | Expecting an increase | H₁: μ > 50 |
| Left-tailed | Expecting a decrease | H₁: μ < 50 |
| Two-tailed | Any difference, direction unknown | H₁: μ ≠ 50 |
Example 1
A sociologist testing whether immigrants score differently (not just higher or lower) on a civic knowledge test compared to native-born citizens would use a two-tailed test, since the direction of difference is theoretically uncertain.
Example 2
A pharmacologist testing whether a new antihypertensive lowers blood pressure (not raises it) would use a one-tailed (left-tailed) test.
Types of Statistical Tests in NHST
Choosing the wrong test is one of the most common errors in applied research. The decision depends on the type of data (continuous vs. categorical), the number of groups, and whether population variance is known.
| Test | Data Type | Groups | When to Use | Example |
| Z-test | Continuous | 1 or 2 | Large sample (n > 30), known population SD | Comparing national exam mean to a known standard |
| One-sample t-test | Continuous | 1 | Small sample, unknown population SD | Testing if a clinic’s mean wait time differs from 30 min |
| Independent samples t-test | Continuous | 2 | Comparing means of two unrelated groups | Depression scores in therapy group vs. control |
| Paired t-test | Continuous | 2 (related) | Same subjects measured twice | Blood pressure before vs. after drug |
| Chi-square test | Categorical | 2+ | Association between categorical variables | Gender vs. vaccine hesitancy (Yes/No) |
| ANOVA | Continuous | 3+ | Comparing means of ≥3 groups | Anxiety scores across 3 therapy modalities |
| One-tailed tests | Any | Any | Directional hypothesis is pre-specified | New drug expected to reduce tumour size |
Step-by-Step Guide to NHST
- Step 1: State the hypotheses. Define H₀ and H₁ in precise, testable terms before looking at the data.
- Step 2: Choose the significance level (α). Pre-specify α, usually 0.05. Changing it after seeing results invalidates the test.
- Step 3: Select the appropriate statistical test. Match the test to your data structure (see table above).
- Step 4: Collect and organize the data. Gather a representative sample. Poor data quality produces misleading p values regardless of the test.
- Step 5: Compute the test statistic. Calculate how far your sample result lies from what H₀ predicts, in units of standard error.
- Step 6: Determine the p value and make a decision. If p-value ≤ α → reject H₀. If p-value > α → insufficient evidence to reject H₀, which is not proof that H₀ is true.
- Step 7: Interpret results in plain language. Report the effect size, direction of difference, and p value. State the conclusion in the context of the original research question.
What are Type I and Type II Errors?
A Type I error occurs when we reject the null hypothesis although that hypothesis was true. A Type II error occurs when we fail to reject the null hypothesis even though it is false.
- Type I error (α): Concluding a new antidepressant works when it actually doesn’t: leading to unnecessary prescription and costs.
- Type II error (β): Concluding a drug doesn’t work when it actually does: a missed therapeutic opportunity.
- Statistical Power (1 − β): The probability of correctly detecting a real effect. Power is typically set at 0.80 in study planning, meaning researchers accept a 20% chance of missing a real effect.
Limitations of Null Hypothesis Significance Testing
NHST has attracted intense criticism over the past few decades, especially in light of the replication crisis in psychology and biomedicine. Key limitations include:
- Binary thinking: Forcing a rich continuum of evidence into “significant” or “not significant” loses information and encourages all-or-nothing interpretation.
- P-hacking and researcher degrees of freedom: Flexible data collection, analysis choices, and selective reporting inflate the false-positive rate far above the nominal α.
- The file-drawer problem and publication bias: Studies that fail to reject H₀ are less likely to be published, biasing the published literature toward positive findings.
- Conflation of statistical and practical significance: A study of 100,000 patients may find that a drug lowers blood pressure by 0.5 mmHg with p < 0.0001: statistically overwhelming, clinically irrelevant.
- Data quality dependence: The accuracy of the results depends on the quality of the data. Poor-quality or inaccurate data can lead to incorrect conclusions.
- Context limitations: Hypothesis testing doesn’t always consider the bigger picture, which can oversimplify results and lead to incomplete insights.
- Assumption violations: Most standard tests assume normally distributed data, independent observations, and equal variances. Violations can distort p values.
Best practices to mitigate these limitations:
- Pre-register hypotheses and analysis plans (e.g., on OSF or ClinicalTrials.gov)
- Report effect sizes and confidence intervals alongside p values
- Use sufficiently powered studies (plan for ≥80% power)
- Replicate findings before drawing firm conclusions
- Consider Bayesian approaches or equivalence testing where appropriate
Frequently Asked Questions (FAQs)
What is statistical power, and why does it matter?
Statistical power (1 − β) is the probability that a test will correctly detect a true effect when one exists. A study with 50% power has only a coin-flip chance of finding a real effect. Low power wastes resources and produces unreliable findings. Power depends on sample size, effect size, and α. Most disciplines target at least 80% power during study design, requiring formal power calculations before data collection.
How is a confidence interval related to a hypothesis test?
A 95% confidence interval (CI) and a two-tailed test at α = 0.05 convey equivalent information: if the CI excludes the null value (e.g., zero for a mean difference), the corresponding p value will be below 0.05. CIs are often preferred because they communicate both the direction and the magnitude of the effect, not just whether it passed a threshold. Reporting both the p value and the CI is considered best practice.
What does it mean to “pre-register” a study?
Pre-registration means publicly documenting your hypotheses, data collection plan, and analysis strategy before collecting data, typically through platforms like ClinicalTrials.gov (biomedical) or the Open Science Framework (social sciences). This prevents researchers from unconsciously adjusting their hypotheses or analysis methods after seeing results (HARKing: Hypothesising After Results are Known), which inflates the false-positive rate and undermines reproducibility.
When should I use a non-parametric test instead of a t-test or ANOVA?
Parametric tests like t-tests and ANOVA assume the data are approximately normally distributed. When sample sizes are small and data are strongly skewed, heavily bounded (e.g., Likert scales with small n), or contain extreme outliers, non-parametric alternatives are more appropriate. Common examples include the Mann-Whitney U test (instead of independent t-test), Wilcoxon signed-rank test (instead of paired t-test), and Kruskal-Wallis test (instead of one-way ANOVA). Non-parametric tests sacrifice some statistical power in exchange for fewer distributional assumptions.
What is the difference between a one-sample and two-sample test?
A one-sample test compares a single group’s mean (or proportion) to a known or hypothesised population value. Example: testing whether the mean birth weight in a hospital differs from the national standard of 3.2 kg. A two-sample test compares the means (or proportions) of two independent groups.
Example: testing whether mean depression scores differ between patients receiving CBT and those receiving pharmacotherapy. When the two sets of measurements come from the same individuals at different times (e.g., pre- and post-intervention), a paired test is used instead.
Can hypothesis testing be used with observational data, or only with experiments?
NHST applies to both experimental and observational data, but the conclusions that can be drawn differ. Randomised controlled trials (RCTs) allow causal inference: if the test rejects H₀, the intervention is likely the cause. In observational studies (e.g., survey data, cohort studies), NHST can detect associations but cannot establish causation because of potential confounding. A statistically significant association between coffee consumption and reduced Parkinson’s disease risk, for instance, does not by itself prove that coffee is protective because unmeasured lifestyle confounders may explain the association.
Comment