What Is Statistical Significance? Meaning, p-Values & Examples

Statistical significance is one of the most widely used—and widely misunderstood—concepts in data analysis. When researchers run a hypothesis test, they need a way to determine whether the pattern they observe in their sample reflects a genuine effect in the population, or whether it could have arisen by random chance alone. Statistical significance provides that decision rule.

Statistical significance relies on comparing a calculated p-value against a predetermined threshold, typically 0.05. If the p-value falls below this threshold, results are labeled “statistically significant,” suggesting the observed effect is unlikely to be a fluke of sampling variation.

However, statistical significance is not the same as practical importance, and misapplying it can lead to flawed conclusions.

Understanding Hypothesis Testing

Statistical significance cannot be understood without first understanding hypothesis testing, the framework that makes it possible. Hypothesis testing is a formal procedure researchers use to evaluate claims about a population using sample data.

The Two Competing Hypotheses

Every hypothesis test begins with two opposing statements:

Null Hypothesis (H₀): This is the default assumption that there is no effect, no difference, or no relationship. It represents the status quo.$$H_0: \mu_1 = \mu_2$$

Alternative Hypothesis (H₁ or Hₐ): This is the claim the researcher is trying to find support for—that a real effect or difference exists.$$H_1: \mu_1 \neq \mu_2$$

The goal of hypothesis testing is never to “prove” the alternative hypothesis outright. Instead, researchers ask whether the sample data provide enough evidence to reject the null hypothesis in favor of the alternative.

A Worked Example

Suppose a researcher wants to know whether a new study technique improves exam scores compared to a traditional method. Two groups of students are tested:

• Group A (traditional method): mean score = 72, n = 30
• Group B (new technique): mean score = 76, n = 30

The null hypothesis states that there is no true difference between the two methods, and any observed gap of 4 points is simply due to random sampling variation. The alternative hypothesis states that the new technique produces a genuinely different result.

Setting the Significance Level

Before collecting or analyzing data, researchers choose a significance level, denoted alpha (α), which sets the threshold for how much risk they’re willing to accept of rejecting a true null hypothesis. The most common choice is:$$\alpha = 0.05$$

This means researchers accept a 5% chance of concluding there is an effect when none actually exists. Some fields, particularly those involving high-stakes decisions like clinical trials, use a stricter threshold of α = 0.01.

One-Tailed vs. Two-Tailed Tests

Hypothesis tests can be directional or non-directional:

Test Type	Question Asked	Example
Two-tailed	Is there any difference?	Does the technique change scores (higher or lower)?
One-tailed	Is there a difference in a specific direction?	Does the technique increase scores?

Choosing between them depends on the research question. A two-tailed test is more conservative and is the default choice unless there’s a strong theoretical reason to expect an effect in only one direction.

From Hypotheses to Decisions

Once the hypotheses and significance level are set, the researcher calculates a test statistic (such as a t-value or z-value) from the sample data, then determines the corresponding p-value. This p-value is what ultimately gets compared against α to decide whether the null hypothesis can be rejected—the exact mechanics of which are covered in the next section.

What Is a p-Value?

The p-value is the number that ultimately determines whether a result is labeled “statistically significant.” Despite being central to hypothesis testing, it’s also one of the most frequently misinterpreted statistics in research.

Definition

A p-value is the probability of observing a result as extreme as, or more extreme than, the one actually obtained, assuming the null hypothesis is true.$$p = P(\text{observed or more extreme data} \mid H_0 \text{ is true})$$

Importantly, the p-value does not tell you the probability that the null hypothesis is true, nor does it tell you the probability that your results occurred by chance. It only measures how consistent your data are with the null hypothesis.

Continuing the Worked Example

Returning to the exam score comparison from the previous section:

• Group A (traditional method): mean = 72, n = 30
• Group B (new technique): mean = 76, n = 30

Suppose the researcher runs an independent samples t-test and obtains a test statistic of t = 2.15, which corresponds to a p-value of 0.035.

Since this p-value is below the pre-set threshold of α = 0.05, the researcher would reject the null hypothesis and conclude that the new study technique produced a statistically significant improvement in exam scores.

Interpreting the P-Value Correctly

P-Value	Interpretation
p < 0.01	Very strong evidence against H₀
p < 0.05	Strong evidence against H₀ (conventional threshold)
0.05 ≤ p < 0.10	Weak or marginal evidence against H₀
p ≥ 0.10	Little to no evidence against H₀

A smaller p-value indicates stronger evidence against the null hypothesis—but it does not indicate a larger or more important effect. This distinction matters enormously and is covered in more depth later in this article.

Common Misconceptions

Several misunderstandings about p-values persist even among experienced researchers:

• Misconception 1: “A p-value of 0.03 means there’s a 3% chance the null hypothesis is true.” This is incorrect. The p-value is calculated assuming the null hypothesis is true—it cannot simultaneously tell you the probability that the assumption itself is true.
• Misconception 2: “A non-significant result proves the null hypothesis.” A high p-value simply means there isn’t enough evidence to reject H₀; it does not confirm that no effect exists.
• Misconception 3: “A smaller p-value means a bigger effect.” P-values are influenced heavily by sample size, so large datasets can produce tiny p-values even for trivial effects.

Calculating P-Values

P-values can be calculated using statistical software or programming tools, including:

• R using functions like t.test() or pnorm()
• Python via scipy.stats
• SPSS through its built-in hypothesis testing procedures
• Stata using commands like ttest

Due to the widespread misuse of p-values, the American Statistical Association has issued formal guidance cautioning against treating the 0.05 threshold as a rigid marker of scientific truth—a topic explored further in the discussion of p-value limitations later in this article.

Significance Level (Alpha, α)

The significance level, denoted alpha (α), is the threshold a researcher sets before collecting data to determine how much risk they’re willing to accept of making a specific kind of error. It acts as the decision boundary against which the p-value is compared.

Definition

Alpha represents the probability of rejecting the null hypothesis when it is actually true—also known as a Type I error. Researchers choose this threshold in advance, before any data analysis takes place.

$$\alpha = P(\text{reject } H_0 \mid H_0 \text{ is true})$$

The most widely used value across scientific disciplines is:$$\alpha = 0.05$$

This means a researcher is willing to accept a 5% chance of concluding that an effect exists when, in reality, it does not.

Common Alpha Levels

Alpha Level	Risk Accepted	Typical Use Case
α = 0.10	10%	Exploratory research, pilot studies
α = 0.05	5%	Standard threshold across most fields
α = 0.01	1%	Medical research, clinical trials
α = 0.001	0.1%	High-stakes fields (e.g., particle physics)

Stricter alpha levels reduce the chance of false positives but increase the chance of missing a real effect, an important trade-off discussed further in the section on Type I and Type II errors.

Continuing the Worked Example

In the exam score study comparing traditional and new study techniques:

• Group A (traditional method): mean = 72, n = 30
• Group B (new technique): mean = 76, n = 30
• Test statistic: t = 2.15
• P-value: 0.035

Before the study began, the researcher set α = 0.05. Because the calculated p-value (0.035) is smaller than alpha (0.05), the result falls into the rejection region, and the null hypothesis is rejected.

Had the researcher instead chosen a stricter threshold of α = 0.01, the same p-value of 0.035 would not be significant, and the null hypothesis would fail to be rejected. This illustrates why the choice of alpha must be made before analyzing data—selecting it afterward, based on the results obtained, undermines the integrity of the test.

Alpha and the Critical Value

Alpha also determines the critical value, the cutoff point on a test statistic’s distribution beyond which results are considered statistically significant. For a two-tailed test with α = 0.05, the critical z-value is:

$$z_{critical} = \pm 1.96$$

If the calculated test statistic falls beyond this boundary, the result is significant at the 0.05 level. This relationship between alpha, critical values, and test statistics is explored in more detail in the critical values section referenced throughout this article.

Choosing an Appropriate Alpha

The right alpha level depends on the consequences of a false positive:

• Low-risk research (e.g., marketing A/B tests) may tolerate α = 0.10
• Standard academic research typically uses α = 0.05
• High-stakes decisions (e.g., drug approval, safety testing) often require α = 0.01 or stricter

Selecting alpha is ultimately a judgment call that balances the risk of false positives against the risk of missing genuine effects—a balance formalized through the concepts of Type I and Type II errors, covered next.

Type I and Type II Errors

Every hypothesis test carries a risk of reaching the wrong conclusion. Statisticians classify these mistakes into two distinct categories: Type I errors and Type II errors. Understanding the difference is essential for interpreting statistical significance correctly.

The Four Possible Outcomes

Every hypothesis test decision falls into one of four categories, depending on the decision made and the actual state of reality:

	H₀ is Actually True	H₀ is Actually False
Reject H₀	Type I Error (False Positive)	Correct Decision (True Positive)
Fail to Reject H₀	Correct Decision (True Negative)	Type II Error (False Negative)

Type I Error (False Positive)

A Type I error occurs when a researcher rejects the null hypothesis even though it is actually true—concluding an effect exists when it does not.$$P(\text{Type I Error}) = \alpha$$

The probability of committing a Type I error is exactly equal to the significance level chosen before the test. This is why alpha is sometimes described directly as the “Type I error rate.”

Example: In the exam score study, a Type I error would mean concluding the new study technique improves scores when, in reality, the traditional and new methods produce identical results, and the observed 4-point gap was simply due to random sampling variation.

Type II Error (False Negative)

A Type II error occurs when a researcher fails to reject the null hypothesis even though it is actually false—missing a real effect that does exist.$$P(\text{Type II Error}) = \beta$$

The probability of a Type II error is denoted beta (β). Unlike alpha, beta is not directly set by the researcher in advance; instead, it depends on factors such as sample size, effect size, and variability in the data.

Example: If the new study technique truly does raise exam scores, but the researcher’s sample was too small or too noisy to detect this improvement, failing to reject H₀ would be a Type II error.

The Inverse Relationship Between Type I and Type II Errors

Reducing the risk of one error type tends to increase the risk of the other:

• Lowering alpha (e.g., from 0.05 to 0.01) reduces the chance of a Type I error but makes it harder to detect real effects, increasing the chance of a Type II error.
• Raising alpha makes it easier to detect real effects but increases the risk of false positives.

This trade-off is why researchers cannot simply minimize both error types at once — they must balance them based on the consequences of each mistake in their specific context.

Statistical Power

The complement of a Type II error is statistical power, the probability of correctly rejecting a false null hypothesis:

$$\text{Power} = 1 – \beta$$

Higher power means a greater likelihood of detecting a true effect when one exists. Power is influenced by:

• Sample size — larger samples increase power
• Effect size — larger true effects are easier to detect
• Significance level — a higher alpha increases power (at the cost of more Type I errors)
• Variability — less noisy data increases power

Researchers often perform a power analysis before collecting data to determine the sample size needed to reliably detect an effect of a given size. Tools like G*Power and R’s pwr package are commonly used for this purpose.

Weighing the Consequences

Which error is worse depends entirely on context:

Field	Costlier Error	Why
Medical screening	Type II	Missing a real disease can be life-threatening
Criminal justice	Type I	Convicting an innocent person is considered worse than acquitting a guilty one
Drug safety testing	Type I	Approving an unsafe drug can harm many patients
Exploratory research	Type II	Missing a promising early-stage finding may be more costly than a false lead

How to Determine Statistical Significance

With the underlying concepts established, this section walks through the full process of determining whether a result is statistically significant, from raw data to final conclusion.

Step 1: State the Hypotheses

Before analyzing any data, define the null and alternative hypotheses clearly.

$$H_0: \mu_1 = \mu_2 \quad \text{(no difference between groups)}$$$$H_1: \mu_1 \neq \mu_2 \quad \text{(a difference exists)}$$

Step 2: Choose a Significance Level

Select alpha (α) in advance, typically 0.05 unless the field or study design calls for a stricter threshold.

$$\alpha = 0.05$$

Step 3: Select the Appropriate Test

The correct statistical test depends on the type of data, the number of groups, and whether the data meet certain assumptions (such as normality).

Data Situation	Appropriate Test
Comparing two independent group means	Independent samples t-test
Comparing two related/paired means	Paired sample t-test
Comparing three or more group means	ANOVA
Testing relationships between categorical variables	Chi-square test
Testing correlation between two continuous variables	Pearson correlation
Non-normal data, two independent groups	Mann-Whitney U test

Step 4: Calculate the Test Statistic

Using the worked example carried throughout this article:

• Group A (traditional method): mean = 72, SD = 8, n = 30
• Group B (new technique): mean = 76, SD = 7.5, n = 30

An independent samples t-test produces:$$t = \frac{\bar{x}_1 – \bar{x}_2}{\sqrt{\frac{s_1^2}{n_1} + \frac{s_2^2}{n_2}}} = 2.15$$

Step 5: Determine the P-Value

The test statistic is converted into a p-value using the appropriate distribution (t-distribution, in this case) with the relevant degrees of freedom.$$df = n_1 + n_2 – 2 = 58$$

For t = 2.15 with 58 degrees of freedom, the corresponding two-tailed p-value is:

$$p = 0.035$$

Step 6: Compare the P-Value to Alpha

$$p = 0.035 < \alpha = 0.05$$

Since the p-value is smaller than the significance level, the result falls within the rejection region.

Step 7: Make a Decision

Condition	Decision	Conclusion
p < α	Reject H₀	Result is statistically significant
p ≥ α	Fail to reject H₀	Result is not statistically significant

In this example, because p = 0.035 is less than α = 0.05, the researcher rejects the null hypothesis and concludes that the new study technique produced a statistically significant improvement in exam scores compared to the traditional method.

Using Software to Determine Significance

In practice, researchers rarely calculate test statistics and p-values by hand. Common tools include:

• R: t.test(group_a, group_b) returns the t-statistic, degrees of freedom, and p-value directly
• Python: scipy.stats.ttest_ind(group_a, group_b) performs the same calculation
• SPSS: Analyze → Compare Means → Independent-Samples T Test
• Excel: T.TEST() function or the Data Analysis ToolPak
• Stata: ttest score, by(group)

Examples of Statistical Significance

Example 1: Clinical Trial (Independent Samples T-Test)

A pharmaceutical company tests whether a new blood pressure medication is more effective than a placebo.

• Placebo group: mean reduction = 4 mmHg, n = 50
• Treatment group: mean reduction = 9 mmHg, n = 50
• Test used: Independent samples t-test
• Result: t(98) = 3.42, p = 0.001

Since p = 0.001 is less than α = 0.05, the result is statistically significant. The company concludes the medication produces a genuinely greater reduction in blood pressure than the placebo.

Example 2: Marketing A/B Test (Chi-Square Test)

An e-commerce company tests two website layouts to see which produces a higher conversion rate.

	Converted	Did Not Convert	Total
Layout A	120	880	1,000
Layout B	160	840	1,000

• Test used: Chi-square test of independence
• Result: χ²(1) = 5.90, p = 0.015

Because p = 0.015 is below 0.05, the difference in conversion rates between the two layouts is statistically significant, suggesting Layout B genuinely outperforms Layout A rather than the difference being due to chance.

Example 3: Education Research (Paired Sample T-Test)

Returning to the study technique example used throughout this article, suppose the same 30 students are tested before and after using the new technique, rather than comparing two separate groups.

• Pre-test mean: 72
• Post-test mean: 78
• Test used: Paired sample t-test
• Result: t(29) = 2.89, p = 0.007

With p = 0.007 falling well below α = 0.05, the improvement in scores after using the new technique is statistically significant.

Example 4: Psychology Research (Pearson Correlation)

A researcher investigates whether hours of sleep are related to reaction time in a sample of 45 participants.

• Test used: Pearson correlation
• Result: r(43) = -0.31, p = 0.038

Since p = 0.038 is less than 0.05, the negative correlation between sleep and reaction time is statistically significant, indicating that more sleep is associated with faster reaction times in this sample.

Example 5: Manufacturing Quality Control (ANOVA)

A factory compares defect rates across three production shifts to determine whether shift timing affects product quality.

Shift	Mean Defects per 1,000 Units
Morning	12
Afternoon	15
Night	22

• Test used: One-way ANOVA
• Result: F(2, 87) = 6.74, p = 0.002

Because p = 0.002 is below α = 0.05, at least one shift’s defect rate differs significantly from the others, prompting further investigation using post-hoc tests to identify which specific shifts differ.

Example 6: A Non-Significant Result

Not every test produces significance. Suppose a researcher examines whether coffee consumption affects typing speed in a sample of 40 participants.

• Test used: Independent samples t-test
• Result: t(38) = 1.24, p = 0.222

Since p = 0.222 is greater than α = 0.05, the researcher fails to reject the null hypothesis. This does not prove coffee has no effect on typing speed — it simply means this study did not find sufficient evidence of one, a distinction covered further in the discussion of statistical versus practical significance.

Statistical Significance vs Practical Significance

A result can be statistically significant without being meaningful in any real-world sense. This distinction — between statistical significance and practical significance — is one of the most important concepts for correctly interpreting research findings.

Defining the Difference

Statistical significance answers the question: Is this effect likely to be real, or could it be due to chance?

Practical significance (also called clinical or substantive significance) answers a different question: Is this effect large enough to matter in the real world?

A result can fall into any of four combinations:

	Statistically Significant	Not Statistically Significant
Practically Significant	Real and meaningful effect	Possibly a real but undetected effect
Not Practically Significant	Real but trivial effect	No meaningful evidence of an effect

Why This Happens: The Role of Sample Size

P-values are heavily influenced by sample size. With a large enough sample, even a tiny, meaningless difference can become statistically significant.

Example: Suppose a company tests a new website design intended to increase average time spent on a page.

• Original design: mean time on page = 120.0 seconds, n = 50,000
• New design: mean time on page = 120.3 seconds, n = 50,000
• Result: t(99998) = 4.12, p < 0.001

With such a massive sample size, even a 0.3-second difference — likely irrelevant to the business — reaches statistical significance. The effect is real, but not practically meaningful.

Measuring Practical Significance: Effect Size

Because p-values alone cannot indicate whether an effect matters, researchers calculate effect size, which measures the magnitude of a difference independent of sample size.

Cohen’s d is one of the most common effect size measures for comparing two means:

$$d = \frac{\bar{x}_1 – \bar{x}_2}{s_{pooled}}$$

Cohen’s d	Interpretation
0.2	Small effect
0.5	Medium effect
0.8	Large effect

Applying Effect Size to the Worked Example

Returning to the study technique example used throughout this article:

• Group A (traditional method): mean = 72, SD = 8, n = 30
• Group B (new technique): mean = 76, SD = 7.5, n = 30
• Result: t(58) = 2.15, p = 0.035

Calculating Cohen’s d:$$d = \frac{76 – 72}{7.75} \approx 0.52$$

This indicates a medium effect size, suggesting the four-point improvement is not only statistically significant but also practically meaningful — students using the new technique showed a moderately sized, real-world relevant improvement in exam performance.

Other Measures of Practical Significance

Field	Common Effect Size Measure
Social sciences	Cohen’s d, Pearson’s r
Medicine	Number needed to treat (NNT), relative risk
Education	Standardized mean difference
Business/marketing	Percentage lift, revenue impact

The Consumer Behavior Example Revisited

In the earlier coffee-and-typing-speed example, the result was not statistically significant (p = 0.222). But consider a contrasting scenario: a huge sample finds a statistically significant result, yet the effect size is negligible.

Example: A company with 200,000 app users finds that a new notification style increases daily engagement from 45.0% to 45.4% of users, with p = 0.003. While technically significant, a 0.4 percentage point lift may not justify the cost of implementing the change — a judgment call that depends on business context rather than statistics alone.

Best Practice: Report Both

Because p-values and effect sizes answer different questions, well-designed research reports both:

$$t(58) = 2.15, \, p = 0.035, \, d = 0.52$$

This combination tells readers not just whether an effect exists, but how large it is — allowing for more informed, balanced conclusions than statistical significance alone can provide. This distinction becomes especially important when evaluating the limitations of p-values, covered in the next section.

Factors That Affect Statistical Significance

Tools and Tests Used

Common Statistical Tests

Test	Used For	Data Type
Independent samples t-test	Comparing means of two unrelated groups	Continuous, normally distributed
Paired sample t-test	Comparing means of two related measurements	Continuous, normally distributed
One-way ANOVA	Comparing means across three or more groups	Continuous, normally distributed
Chi-square test	Testing relationships between categorical variables	Categorical
Pearson correlation	Measuring linear relationships between two continuous variables	Continuous
Mann-Whitney U test	Comparing two independent groups when data are not normal	Ordinal or non-normal continuous
Wilcoxon Signed Rank Test	Comparing paired data when normality is violated	Ordinal or non-normal continuous

Worked Example Across Multiple Tools

Returning to the study technique example used throughout this article:

• Group A (traditional method): mean = 72, SD = 8, n = 30
• Group B (new technique): mean = 76, SD = 7.5, n = 30
• Test used: Independent samples t-test
• Result: t(58) = 2.15, p = 0.035

Here is how this same test would be run across the most widely used statistical tools.

R

r

group_a <- c(...)  # traditional method scores
group_b <- c(...)  # new technique scores
t.test(group_a, group_b)

R returns the t-statistic, degrees of freedom, p-value, and confidence interval in a single output, making it a popular choice for academic research.

Python

python

from scipy import stats
t_stat, p_value = stats.ttest_ind(group_a, group_b)

Python’s SciPy library is widely used in data science workflows, particularly when statistical testing is combined with broader data cleaning, visualization, or machine learning pipelines.

SPSS

In SPSS, the same test is run through the menu system:

Analyze → Compare Means → Independent-Samples T Test

SPSS is common in social science and psychology research due to its accessible point-and-click interface, which requires no coding.

Stata

stata

ttest score, by(group)

Stata is frequently used in economics and health research, particularly for its strong handling of panel data and survey-weighted analyses.

Excel

Excel offers two approaches:

• The T.TEST() function: =T.TEST(range1, range2, 2, 2)
• The Data Analysis ToolPak: Data → Data Analysis → t-Test: Two-Sample Assuming Equal Variances

Excel is often the entry point for students and business analysts, though it offers less flexibility than dedicated statistical software for advanced tests.

Choosing the Right Tool

Tool	Best Suited For	Learning Curve
R	Academic research, custom analyses, visualization	Moderate to steep
Python	Data science, automation, large datasets	Moderate to steep
SPSS	Social sciences, psychology, no-code workflows	Low
Stata	Economics, epidemiology, panel data	Moderate
Excel	Quick calculations, business contexts, teaching	Low

Specialized Tools for Power Analysis

Beyond running the significance test itself, researchers often need to determine adequate sample size beforehand. Common tools for this include:

• G*Power: A free, widely used tool for calculating required sample sizes across many test types
• R’s pwr package: Provides power analysis functions integrated directly into R workflows
• Python’s statsmodels: Includes power and sample size calculation functions alongside its broader statistical modeling capabilities

FAQs