Type 1 Error: Rejecting a True Null Hypothesis

Concerning the null hypothesis H_o that the ACE training graduates do not have higher VAST scores than non-graduates, there are two possible decisions we can make based upon our observed sample mean: “reject H_o” or “fail to reject H_o.”

Question: If there is no treatment effect, which decision regarding the null hypothesis is correct and which is in error?

When there is no treatment effect but we reject the null hypothesis and conclude that there is an effect, we are making an error. This is called a Type 1 error, falsely concluding that there is an effect, by rejecting the null, when there is no effect (top purple cell). On the other hand, if we fail to reject the null hypothesis, our conclusion correctly matches the actual situation (bottom purple cell).

Alpha, Type 1 Error, and Critical Values

Whether or not we reject the null hypothesis is determined by whether the observed sample mean exceeds a critical value. The critical value is defined on the sampling distribution for means of samples of size N drawn from the null distribution, where the probability of obtaining a sample mean as large as or larger than the critical value is equal to a pre-defined alpha, α.

We can compute a critical value in four easy steps:

1. Identify the p-value for the critical value. Here the p-value is .05 on the upper tail.
2. Convert the p-value to a z-score. Here the p–z converter tells us that a z-score of 1.645 cuts off .05 on the upper tail.
3. Multiply the z-score by the standard error to find the distance the critical value is from the mean.
4. Add this distance to the mean to find the numerical value of the critical value.

In the figure below, the sampling distributions of the mean are shown for N = 5 (top) and N = 25 (bottom) and critical values (573.56 and 532.9 respectively, represented by dotted red lines) are both set at α = .05. The light blue regions indicate the parts of the sampling distribution where the sample mean is below the critical value, thus leading to the decision to fail to reject the null hypothesis. The dark blue regions show areas of rejection. Thus, when there is no treatment effect (and we are sampling from the null population), the light blue regions represent the probability of making the correct decision, and the dark blue regions represent the probability of making a Type 1 error, that is α.

• Alpha, α, is the probability of making a Type 1 error when H_o is true. This value is specified by the researcher before looking at the data.
• In the two sampling distributions below, the dark blue regions both represent 5% of the distribution, corresponding to the value of α = .05 that was specified.

Notice that with a bigger sample size, the critical value is closer to the null mean because of reduced sample mean variability. Thus, the size of the sample mean needed to reject the null hypothesis, in our example, is smaller when the the sample size is larger.

We can now update the decision matrix by labeling the cells and inserting their probabilities when the actual situation is that there is no treatment effect:

Next