If the p-value is large, it would be nice to be able to conclude that the null hypothesis is true. Large p-values can occur if the null hypothesis is true or close enough to being true. However, they can also occur if the study is not sufficiently well-designed to have a reasonable chance of generating a low p-value if an important difference from the null hypothesis actually exists. Even a p-value of 1.0, which is the highest value it can possibly be, does not necessarily provide strong evidence that the null is true because large p-values can also be obtained if the null is false but the study is poorly designed.
The quality of a study is measured by its statistical power, and p-values need to be interpreted in its light. Power is the probability of obtaining a statistically significant result given that the null hypothesis is not true. Statistical power can help determine necessary sample sizes and assist the planning of data collection and subsequent analysis. The only problem is that this is rarely done in ecology. Power is almost never reported by ecologists, but in approximately half of all cases authors interpret their non-significant results as evidence that the null hypothesis is true (Peterman, 1990; Taylor and Gerrodette, 1993; Johnson, 1999; Anderson et al., 2000; Fidler et al., 2004). This is despite the fact that power must be known if we are to interpret the importance of non-significant results (Fidler et al., 2004).
When calculating power it is necessary to specify both the difference one wishes to detect and the variance of the data. Both values can be difficult to determine, but any calculation of power is conditional on the values that are used. Smaller differences may go undetected, and power will be less than expected if the variance is underestimated.
Was this article helpful?