P values

What Are P values?

P values, or probability values, are a fundamental concept in statistical analysis used to quantify the evidence against a null hypothesis in hypothesis testing. In essence, a P value indicates the probability of observing a test statistic as extreme as, or more extreme than, the one calculated from the sample data, assuming that the null hypothesis is true. A smaller P value suggests stronger evidence against the null hypothesis, making it less likely that the observed data occurred by random chance alone if the null hypothesis were indeed correct. This measure aids researchers and analysts in data analysis to decide whether to reject or fail to reject the null hypothesis.

History and Origin

The concept of the P value has roots in the 18th century with figures like Pierre-Simon Laplace, but its modern application and popularization are largely attributed to Sir Ronald A. Fisher in the early 20th century. Fisher, a British statistician and geneticist, formalized the P value as a tool for statistical inference in his 1925 work, Statistical Methods for Research Workers. He proposed the widely adopted (P = 0.05) (or 1 in 20) as a conventional threshold for statistical significance¹⁶, ¹⁷. Fisher's methodology was designed to provide a measure of evidence against a null hypothesis, allowing researchers to gauge the strength of their findings. The P value, as defined by Fisher, represents the probability of obtaining an effect equal to or more extreme than the one observed, under the assumption that the null hypothesis is true¹⁵.

Key Takeaways

• P values quantify the evidence against a null hypothesis.
• A lower P value suggests stronger evidence against the null hypothesis.
• They are a probability, ranging from 0 to 1.
• P values are often compared to a predetermined significance level (alpha) to make decisions in hypothesis testing.
• Misinterpretation of P values can lead to flawed conclusions in research and analysis.

Formula and Calculation

The calculation of a P value depends on the specific statistical test being conducted and the distribution of the test statistic under the null hypothesis. While there isn't a single universal formula for P values themselves, they are derived from the test statistic (e.g., t-statistic, F-statistic, chi-square statistic) and the associated probability distribution.

For example, in a simple z-test for a population mean, the test statistic is calculated as:

Where:

• (\bar{x}) = sample mean
• (\mu_0) = hypothesized population mean (under the null hypothesis)
• (\sigma) = population standard deviation
• (n) = sample size

Once the Z-score is calculated, the P value is the probability of observing a Z-score as extreme as or more extreme than the calculated one, based on the standard normal distribution. This probability is typically found using statistical tables or software.

Interpreting the P values

Interpreting P values involves comparing the calculated P value to a predefined significance level, often denoted as alpha ((\alpha)). Common alpha levels are 0.05 (5%), 0.01 (1%), or 0.10 (10%).

• If P value (\le \alpha): The result is considered statistically significant. This means there is sufficient evidence to reject the null hypothesis in favor of the alternative hypothesis. The observed data are considered unlikely to have occurred if the null hypothesis were true.
• If P value (> \alpha): The result is not considered statistically significant. There is insufficient evidence to reject the null hypothesis. This does not mean the null hypothesis is true, but rather that the observed data do not provide strong enough evidence against it.

For instance, a P value of 0.03 (3%) with an alpha level of 0.05 (5%) would lead to the rejection of the null hypothesis. In contrast, a P value of 0.07 (7%) with the same alpha level would lead to a failure to reject the null hypothesis. It is crucial for professionals engaged in quantitative research to understand that a P value does not indicate the magnitude or importance of an effect, nor the probability that the hypothesis is true¹⁴.

Hypothetical Example

Consider a financial analyst wanting to determine if a new algorithmic trading strategy generates a higher average daily return than a benchmark index, which historically has an average daily return of 0.05%.

1. Formulate Hypotheses:
   • Null Hypothesis ((H_0)): The algorithmic strategy's average daily return is equal to or less than 0.05%.
   • Alternative Hypothesis ((H_A)): The algorithmic strategy's average daily return is greater than 0.05%.
2. Collect Data: The analyst runs the algorithm for 100 trading days and records the daily returns. The average daily return for this period is 0.07%, with a standard deviation of 0.02%.
3. Choose Significance Level: The analyst sets an alpha ((\alpha)) level of 0.05.
4. Calculate Test Statistic: Using a one-sample t-test (since the population standard deviation is unknown and sample size is relatively large), a t-statistic is calculated based on the sample mean, hypothesized mean, sample standard deviation, and sample size.
5. Determine P value: Based on the calculated t-statistic and degrees of freedom, the P value is derived from the t-distribution. Let's assume the calculation yields a P value of 0.02.
6. Make a Decision: Since the P value (0.02) is less than the chosen alpha level (0.05), the analyst rejects the null hypothesis. This suggests that there is statistically significant empirical evidence to conclude that the new algorithmic strategy's average daily return is indeed greater than 0.05%. This step is a critical part of model validation for the strategy.

Practical Applications

P values are widely used across various domains within finance and economics for drawing conclusions from data. In econometrics, they help assess the significance of coefficients in regression models, indicating whether certain economic variables have a statistically reliable impact on an outcome. For instance, an analyst might use P values to determine if a country's GDP growth significantly influences its stock market performance.

In risk management, P values can be employed in testing the effectiveness of new risk models or confirming if observed deviations from expected losses are statistically meaningful. For the evaluation of investment strategies, P values are frequently used in backtesting to ascertain if a strategy's historical performance is genuinely anomalous or merely a result of chance. However, it's vital to recognize the potential for misuse, as extensive backtesting without proper controls can lead to "p-hacking" and misleadingly low P values¹³. Financial institutions also use them in fraud detection, where a low P value might indicate an unusual transaction that warrants further investigation. The National Institute of Standards and Technology (NIST) also utilizes P values in their statistical test suites for evaluating the randomness of cryptographic applications, where a P value helps assess if a generated sequence of numbers deviates significantly from true randomness¹².

Limitations and Criticisms

Despite their widespread use, P values have faced substantial criticism, particularly regarding their interpretation and potential for misuse. A key limitation is the common misinterpretation that a P value represents the probability that the null hypothesis is true, or that a low P value indicates a large or important effect¹⁰, ¹¹. In reality, a P value merely indicates the compatibility of the data with a specified statistical model, often the null hypothesis⁹. It does not convey the size of an effect or its practical significance⁸.

Another significant critique revolves around the arbitrary nature of the conventional 0.05 significance threshold. Relying solely on this binary "significant/not significant" distinction can obscure important information and lead to a false sense of certainty or a dismissal of potentially meaningful findings⁶, ⁷. This over-reliance can lead to practices like "p-hacking," where researchers manipulate data analysis to achieve a statistically significant P value, thereby increasing the risk of false positives and distorting research findings⁵.

Leading statistical organizations, such as the American Statistical Association (ASA), have issued statements cautioning against the misuse of P values, emphasizing that scientific conclusions should not be based solely on whether a P value crosses a specific threshold³, ⁴. They advocate for considering other factors, such as confidence interval estimates, study design, data collection methods, and external evidence, to provide a more comprehensive interpretation of results¹, ².

P values vs. Statistical Significance

While often used interchangeably in common parlance, "P values" and "statistical significance" represent distinct but related concepts. A P value is the specific probability calculated from data that quantifies the evidence against the null hypothesis. It is a continuous measure ranging from 0 to 1. Statistical significance, on the other hand, is a binary conclusion reached by comparing the P value to a predetermined threshold, known as the alpha level ((\alpha)). If the P value falls below this threshold, the result is declared statistically significant; otherwise, it is not. The confusion often arises because the P value is the basis for determining statistical significance, but it is not the significance itself. Statistical significance is a judgment made about the P value in the context of a chosen alpha level, acting as a decision rule rather than a direct measure of evidence.

FAQs

What does a high P value mean?

A high P value (e.g., greater than 0.05) indicates that the observed data are consistent with the null hypothesis. It suggests that there isn't enough evidence to reject the null hypothesis, meaning the observed effect could reasonably be due to random chance. It does not prove that the null hypothesis is true.

Can a P value be 0?

In theory, a P value can approach zero, indicating an extremely low probability that the observed results occurred by chance under the null hypothesis. In practice, due to computational precision and the nature of continuous distributions, a P value is rarely exactly zero but can be reported as <0.001 or a similar small number.

Is a low P value always good?

Not necessarily. While a low P value indicates statistical significance, it does not imply that the finding is practically important or that the effect size is large. A statistically significant result, especially with very large sample sizes, can correspond to a very small and practically irrelevant effect. Focus should also be placed on the magnitude of the effect and its real-world implications, not just the P value.

What is the typical alpha level used with P values?

The most commonly used alpha level is 0.05 (or 5%). However, the appropriate alpha level depends on the specific context of the research, the field of study, and the consequences of making a Type I error (false positive) or a Type II error (false negative). Some studies use stricter levels like 0.01, while others might use 0.10.