• Sign Up

Hypothesis Testing, P Values, Confidence Intervals, and Significance

  • Article Library
  • Hypothesis Testing, P Values, Confidence Intervals, and Significance

Hypothesis Testing, P Values, Confidence Intervals, and Significance

Article Author:
Jacob Shreffler
Article Editor:
Martin Huecker
Updated:
5/6/2020 9:40:41 AM
For CME on this topic:
Hypothesis Testing, P Values, Confidence Intervals, and Significance CME
PubMed Link:
Hypothesis Testing, P Values, Confidence Intervals, and Significance

Definition/Introduction

Medical providers often rely on evidence-based medicine to guide decision-making in practice. Often a research hypothesis is tested with results provided, typically with p values, confidence intervals, or both. Additionally, statistical or research significance is estimated or determined by the investigators. Unfortunately, healthcare providers may have different comfort levels in interpreting these findings, which may affect the adequate application of the data.

Issues of Concern

Without having a foundational understanding of hypothesis testing, p values, confidence intervals as well as the difference between statistical and clinical significance, it may affect healthcare providers' ability to make clinical decisions without relying purely on the research investigators deemed level of significance. Therefore, an overview of these concepts is provided to allow medical professionals to use their expertise to determine if results are reported sufficiently and if the study outcomes are clinically appropriate to be applied in healthcare practice.

Hypothesis Testing

Investigators conducting studies need research questions and hypotheses to guide analyses. Starting with broad research questions (RQs), investigators then identify a gap in current clinical practice or research. Any research problem or statement is grounded in a better understanding of relationships between two or more variables. For this article, we will use the following research question example:

Research Question: Is Drug 23 an effective treatment for Disease A?

Research questions do not directly imply specific guesses or predictions; we must formulate research hypotheses. A hypothesis is a predetermined declaration regarding the research question in which the investigator(s) makes a precise, educated guess about a study outcome. This is sometimes called the alternative hypothesis and ultimately allows the researcher to take a stance based on experience or insight from medical literature. An example of a hypothesis is below.

Research Hypothesis: Drug 23 will significantly reduce symptoms associated with Disease A compared to Drug 22.

The null hypothesis states that there is no statistical difference between groups based on the stated research hypothesis.

Researchers should be aware of journal recommendations in considering how to report p values, and manuscripts should remain internally consistent.

Regarding p values, as the number of individuals enrolled in a study (the sample size) increases, the likelihood of finding a statistically significant effect increases. With very large sample sizes, the p-value can be very low significant differences in the reduction of symptoms for Disease A between Drug 23 and Drug 22. The null hypothesis is deemed true until a study presents significant data to support rejecting the null hypothesis. Based on results, the investigators will either reject the null hypothesis (if they found significant differences or associations), or they will fail to reject the null hypothesis (they were not able to provide proof that there were significant differences or associations).

In order to test a hypothesis, researchers obtain data on a representative sample to determine whether to reject or fail to reject a null hypothesis. In most research studies, it is not feasible to obtain data for an entire population. The use of a sampling procedure allows for statistical inference, though this involves a certain possibility of error.[1] When determining whether to reject or fail to reject the null hypothesis, mistakes can be made: Type I and Type II errors. Though it is impossible to assure that these errors have not occurred, researchers should limit the possibilities of these faults.[2]

Significance

Significance is a term to describe the substantive importance of medical research. Statistical significance is the likelihood of results due to chance.[3] Healthcare providers should always delineate statistical significance from clinical significance, a common error when reviewing biomedical research.[4] When conceptualizing findings reported as either significant or not significant, healthcare providers should not simply accept researchers' results or conclusions without considering the clinical significance. Healthcare professionals should consider the clinical importance of findings and have an understanding of both p values and confidence intervals, so they do not have to rely on the researchers to determine the level of significance.[5] One criterion often used to determine statistical significance is the utilization of p values.

P Values

P values are used in research to determine whether the sample estimate is significantly different from a hypothesized value. The p-value is the probability that the observed effect within the study would have occurred by chance if, in reality, there was no true effect. Conventionally, data yielding a p<0.05 or p<0.01 is considered statistically significant. While some have debated that the 0.05 level should be lowered, it is still universally practiced.[6] Hypothesis testing allows us to determine the size of the effect.

An example of findings reported with p values are below:

Statement: Drug 23 reduced patients' symptoms compared to Drug 22. Patients who were provided Drug 23 (n=100) were 2.1 times less likely than patients who received Drug 22 (n = 100) to experience symptoms of Disease A, p<0.05.

Or

Statement:Individuals who were prescribed Drug 23 experienced fewer symptoms (M = 1.3, SD = 0.7) compared to individuals who were prescribed Drug 22 (M = 5.3, SD = 1.9). This finding was statistically significant, p= 0.02.

For either statement, if the threshold had been set at 0.05, the null hypothesis (that there was no relationship) should be rejected, and we should conclude significant differences. Noticeably, as can be seen in the two statements above, some researchers will report findings with < or > and others will provide an exact p-value (0.000001) but never zero [6]. When examining research, readers should understand how p values are reported. The best practice is to report all p values for all variables within a study design, rather than only providing p values for variables with significant findings.[7] The inclusion of all p values provides evidence for study validity and limits suspicion for selective reporting/data mining.  

While researchers have historically used p values, experts who find p values problematic encourage the use of confidence intervals.[8]. P-values alone do not allow us to understand the size or the extent of the differences or associations.[3] In March 2016, the American Statistical Association (ASA) released a statement on p values, noting that scientific decision-making and conclusions should not be based on a fixed p-value threshold (e.g., 0.05). They recommend focusing on the significance of results in the context of study design, quality of measurements, and validity of data. Ultimately, the ASA statement noted that in isolation, a p-value does not provide strong evidence.[9]

When conceptualizing clinical work, healthcare professionals should consider p values with a concurrent appraisal study design validity. For example, a p-value from a double-blinded randomized clinical trial (designed to minimize bias) should be weighted higher than one from a retrospective observational study [7]. The p-value debate has smoldered since the 1950s[10], and replacement with confidence intervals has been suggested since the 1980s.[11]

Confidence Intervals

A confidence interval provides a range of values within given confidence (e.g., 95%), which includes the accurate value of the statistical constraint within a targeted population.[12] Most research uses a 95% CI, but investigators can set any level (e.g., 90% CI, 99% CI).[13] A CI provides a range with the lower bound and upper bound limits of a difference or association that would be plausible for a population.[14] Therefore, a CI of 95% indicates that if a study were to be carried out 100 times, the range will contain the true value in 95,[15] confidence intervals provide more evidence regarding the precision of an estimate compared to p-values.[6]

In consideration of the similar research example provided above, one could make the following statement with 95% CI:

Statement: Individuals who were prescribed Drug 23 had no symptoms after three days, which was significantly faster than individuals who were prescribed Drug 22; there was a mean difference between the two groups of days to the recovery of 4.2 days (95% CI: 1.9 – 7.8).

It is important to note that the width of the CI is affected by the standard error and the sample size; reducing a study sample number will result in less precision of the CI (increase the width).[14] A larger width indicates a smaller sample size or a larger variability.[16] A researcher would want to increase the precision of the CI. For example, a 95% CI of 1.43 – 1.47 is much more precise than the one provided in the example above. In research and clinical practice, CIs provide valuable information on whether the interval includes or excludes any clinically significant values.[14]

Null values are sometimes used for differences with CI (zero for differential comparisons and 1 for ratios). However, CIs provide more information than that.[15] Consider this example: A hospital implements a new protocol that resulted in the reduction of wait time for patients in the emergency department by an average of 25 minutes (95% CI: -2.5 – 41 minutes). Because the range crosses zero, implementation of this protocol in different populations could result in longer wait times; however, the range is much higher on the positive side. Thus, while the p-value used to detect statistical significance for this may result in "not significant" findings, individuals should examine this range, consider the study design, and weigh whether or not it is still worth piloting in their workplace.

Similarly to p-values, 95% CIs cannot control for researchers' errors (eg. study bias or improper data analysis).[14] In consideration of whether to report p-values or CIs, researchers should examine journal preferences. When in doubt, reporting both may be beneficial.[13] An example is below:

Reporting both: Individuals who were prescribed Drug 23 had no symptoms after three days, which was significantly faster than individuals who were prescribed Drug 22, p = 0.009. There was a mean difference between the two groups of days to the recovery of 4.2 days (95% CI: 1.9 – 7.8).

Clinical Significance

Recall that clinical significance and statistical significance are two different concepts. Healthcare providers should remember that a study with statistically significant differences and large sample size may be of no interest to clinicians, whereas a study with lower sample size and statistically non-significant results could impact clinical practice.[14] Additionally, as previously mentioned, a non-significant finding may be a reflection of the study design itself rather than relationships between variables.

Healthcare providers using evidence-based medicine to inform practice should use clinical judgment to determine practical importance of studies through careful evaluation of the design, sample size, power, likelihood of type I and type II errors, data analysis and reporting of statistical findings (p values, 95% CI or both).[4] Interestingly, some experts have called for "statistically significant" or "not significant" to be excluded from work as statistical significance has never and will never be equivalent to clinical significance.[17]

The decision on what is clinically significant can be challenging, depending on the providers' experience and especially the severity of the disease. Providers should use their knowledge and experiences to determine the meaningfulness of study results and make inferences based not only on significant or not significant results by researchers but by their understanding of study limitations and practical implications.

Nursing, Allied Health, and Interprofessional Team Interventions

All physicians, nurses, pharmacists, and other healthcare professionals should strive to understand the concepts in this chapter. These individuals should maintain the ability to review and incorporate new literature for evidence-based and safe care. 


References

[1] Jones M,Gebski V,Onslow M,Packman A, Statistical power in stuttering research: a tutorial. Journal of speech, language, and hearing research : JSLHR. 2002 Apr;     [PubMed PMID: 12003508]
[2] Sedgwick P, Pitfalls of statistical hypothesis testing: type I and type II errors. BMJ (Clinical research ed.). 2014 Jul 3;     [PubMed PMID: 24994622]
[3] Fethney J, Statistical and clinical significance, and how to use confidence intervals to help interpret both. Australian critical care : official journal of the Confederation of Australian Critical Care Nurses. 2010 May;     [PubMed PMID: 20347326]
[4] Hayat MJ, Understanding statistical significance. Nursing research. 2010 May-Jun;     [PubMed PMID: 20445438]
[5] Ferrill MJ,Brown DA,Kyle JA, Clinical versus statistical significance: interpreting P values and confidence intervals related to measures of association to guide decision making. Journal of pharmacy practice. 2010 Aug;     [PubMed PMID: 21507834]
[6] Infanger D,Schmidt-Trucksäss A, P value functions: An underused method to present research results and to promote quantitative reasoning. Statistics in medicine. 2019 Sep 20;     [PubMed PMID: 31270842]
[7] Dorey F, Statistics in brief: Interpretation and use of p values: all p values are not equal. Clinical orthopaedics and related research. 2011 Nov;     [PubMed PMID: 21918804]
[8] Liu XS, Implications of statistical power for confidence intervals. The British journal of mathematical and statistical psychology. 2012 Nov;     [PubMed PMID: 22026811]
[9] Tijssen JG,Kolm P, Demystifying the New Statistical Recommendations: The Use and Reporting of p Values. Journal of the American College of Cardiology. 2016 Jul 12;     [PubMed PMID: 27386779]
[10] Spanos A, Recurring controversies about P values and confidence intervals revisited. Ecology. 2014 Mar;     [PubMed PMID: 24804448]
[11] Freire APCF,Elkins MR,Ramos EMC,Moseley AM, Use of 95% confidence intervals in the reporting of between-group differences in randomized controlled trials: analysis of a representative sample of 200 physical therapy trials. Brazilian journal of physical therapy. 2019 Jul - Aug;     [PubMed PMID: 30366845]
[12] Dorey FJ, In brief: statistics in brief: Confidence intervals: what is the real result in the target population? Clinical orthopaedics and related research. 2010 Nov;     [PubMed PMID: 20532716]
[13] Porcher R, Reporting results of orthopaedic research: confidence intervals and p values. Clinical orthopaedics and related research. 2009 Oct;     [PubMed PMID: 19565303]
[14] Gardner MJ,Altman DG, Confidence intervals rather than P values: estimation rather than hypothesis testing. British medical journal (Clinical research ed.). 1986 Mar 15;     [PubMed PMID: 3082422]
[15] Cooper RJ,Wears RL,Schriger DL, Reporting research results: recommendations for improving communication. Annals of emergency medicine. 2003 Apr;     [PubMed PMID: 12658257]
[16] Doll H,Carney S, Statistical approaches to uncertainty: P values and confidence intervals unpacked. Equine veterinary journal. 2007 May;     [PubMed PMID: 17520981]
[17] Colquhoun D, The reproducibility of research and the misinterpretation of {i}p{/i}-values. Royal Society open science. 2017 Dec;     [PubMed PMID: 29308247]