HookThe radar operator's dilemma

In 1940 a Chain Home radar operator on the English coast stared at a flickering cathode-ray tube and had to make the same decision hundreds of times a night: is that blip a Luftwaffe raid, or atmospheric noise? Call it a raid when it is noise and squadrons scramble for nothing — fuel burned, pilots exhausted, credibility spent. Call it noise when it is a raid and bombers reach their target unopposed. The two mistakes are not symmetric, they cannot both be driven to zero, and making one rarer makes the other more common. Postwar engineers turned this dilemma into signal detection theory, but statisticians had formalised it a decade earlier: Jerzy Neyman and Egon Pearson's 1933 framework gave the two mistakes the names this section runs on — the false alarm is a Type I error, the missed raid a Type II error.

Every hypothesis test you have ever run makes exactly these two mistakes at calculable rates, and this section is where you finally calculate both. A Type I error is rejecting a null hypothesis that is actually true; its probability is the actual significance level of the test — which, for the discrete distributions, is almost never the 5% on the label. A Type II error is failing to reject a null hypothesis that is actually false; its probability \(\beta\) depends on what the truth really is, and \(1-\beta\) — the probability the test catches a real effect — is called the power of the test. You will compute both for binomial, Poisson and normal tests, and meet the trade-off the radar operator lived nightly: tighten the test against false alarms and you blunt its power to detect anything.

ModelType I errors — the price on the label versus the price you pay

A Type I error occurs when \(H_0\) is true but your observation lands in the critical region, so you reject it anyway. Since the critical region was built precisely to have small probability under \(H_0\),

\[P(\text{Type I error})=P(X\in\text{critical region}\mid H_0\text{ true}),\]

which is the actual significance level of the test. For a test on a normal mean this equals the nominal level exactly: a continuous distribution lets you place the critical value wherever you like, so a test 'at the 5% level' really does have \(P(\text{Type I error})=0.05\). For binomial and Poisson tests the distribution is lumpy — probability arrives in discrete chunks — so you cannot usually hit 5% on the nose. The convention is to choose the critical region so that its probability under \(H_0\) is as large as possible without exceeding the nominal level, and then the actual level is whatever that region's probability turns out to be: often 2% or 3% when the label says 5%. Examiners lean on this distinction hard: 'the test is carried out at the 5% level' and 'the probability of a Type I error is 5%' are different claims, and for discrete distributions the second is usually false. Always state the error in context too — not 'we reject a true \(H_0\)' but 'the company concludes the defect rate has fallen when in fact it has not'.

Worked example

A machine produces components of which 30% are, historically, rejects. After a service, the manager tests \(H_0\colon p=0.3\) against \(H_1\colon p<0.3\) at the 5% level using the number of rejects \(X\) in a sample of 20, where \(X\sim\mathrm{B}(20,0.3)\) under \(H_0\). Step 1 — build the critical region: cumulative binomial values give \(P(X\le 2)=0.0355\) and \(P(X\le 3)=0.1071\). Since \(0.1071>0.05\), the critical region is \(X\le 2\). Step 2 — read off the Type I probability: \(P(\text{Type I error})=P(X\le 2\mid p=0.3)=0.0355\), an actual level of 3.55%, not 5%. Step 3 — context: there is a 3.55% chance the manager concludes the service reduced the reject rate when really nothing changed. The same logic runs for a Poisson test: with \(H_0\colon\lambda=9\), \(H_1\colon\lambda<9\) and critical region \(X\le 3\), \(P(\text{Type I error})=P(X\le 3\mid\lambda=9)=e^{-9}\big(1+9+\tfrac{81}{2}+\tfrac{729}{6}\big)=172e^{-9}=0.0212\) — a 5%-label test whose real false-alarm rate is 2.1%.

ModelType II errors and power — how often the test misses

A Type II error is the opposite failure: \(H_0\) is false, but the observation lands in the acceptance region and the test fails to reject. Here is the awkward asymmetry that organises every calculation in this section: 'the null is false' is not one hypothesis but infinitely many — the true parameter could be anywhere in \(H_1\) — so \(P(\text{Type II error})\) has no single value. It can only be computed given a specific alternative value, which is why every exam question supplies one ('given that in fact \(p=0.1\)…'). Writing \(\beta\) for this probability,

\[\beta=P(X\notin\text{critical region}\mid\text{specified true value}),\qquad \text{power}=1-\beta,\]

and the power is the probability that the test correctly detects that particular departure from \(H_0\). The two-step recipe never varies. Step one: find the critical region using the \(H_0\) distribution — the region is fixed before the truth enters. Step two: throw the \(H_0\) distribution away, recompute the probability of the acceptance region under the specified true parameter, and that is \(\beta\). The single most common exam error is smearing these steps together — using the true parameter to build the region, or the null parameter to evaluate \(\beta\). The region belongs to \(H_0\); the error probability belongs to the truth.

Worked example

Continue the components example: critical region \(X\le 2\), built under \(H_0\colon p=0.3\). Suppose the service genuinely worked and the true reject rate is \(p=0.1\). Step 1 — the acceptance region is \(X\ge 3\). Step 2 — evaluate it under the truth, \(X\sim\mathrm{B}(20,0.1)\): \(P(X\le 2)=P(0)+P(1)+P(2)=0.9^{20}+20(0.1)(0.9)^{19}+190(0.01)(0.9)^{18}=0.1216+0.2702+0.2852=0.6769\). So \(\beta=P(X\ge 3\mid p=0.1)=1-0.6769=0.3231\) and the power is \(0.677\). Read that number honestly: even when the service halves-and-more the reject rate from 30% to 10%, this test only notices about two times in three — a 32% chance of a Type II error. Small samples buy weak tests, and computing the power is how you find out just how weak.

MechanismThe normal case — where everything is exact and the trade-off is visible

For a test on a normal mean the calculations are cleaner because the critical region boundary is a formula rather than a search. Testing \(H_0\colon\mu=\mu_0\) against \(H_1\colon\mu>\mu_0\) at level \(\alpha\), with known \(\sigma\) and sample size \(n\), the test rejects when the sample mean exceeds

\[c=\mu_0+z_\alpha\frac{\sigma}{\sqrt{n}},\]

where \(z_\alpha\) is the upper-tail critical value (1.6449 for 5%, 2.3263 for 1%). Because \(\bar{X}\) is continuous, \(P(\text{Type I error})=\alpha\) exactly. For the Type II calculation the boundary \(c\) stays put but the distribution of \(\bar{X}\) slides across to sit over the true mean \(\mu_1\): \(\beta=P(\bar{X}<c\mid\mu=\mu_1)\), one standardisation away. Sketch the two normal curves — the \(H_0\) curve with area \(\alpha\) shaded to the right of \(c\), the true curve with area \(\beta\) shaded to the left of it — and the whole politics of testing is in one picture. Drag \(c\) rightward (demand a stricter significance level) and \(\alpha\) shrinks while \(\beta\) swells: you cannot reduce both errors by moving the boundary. The only ways to shrink both at once are a larger sample, which narrows both curves around their means, or a bigger true effect \(|\mu_1-\mu_0|\), which pulls the curves apart — and the effect size is nature's choice, not yours.

Worked example

Battery lifetimes are normal with standard deviation \(\sigma=4\) hours. A manufacturer claims a mean of 30 hours; a retailer suspects it is higher and tests \(H_0\colon\mu=30\) against \(H_1\colon\mu>30\) at the 5% level with \(n=25\). Step 1 — critical value: \(c=30+1.6449\times\frac{4}{\sqrt{25}}=30+1.6449\times 0.8=31.316\) hours, and \(P(\text{Type I error})=0.05\) exactly. Step 2 — suppose the true mean is \(\mu=32\). Then \(\bar{X}\sim\mathrm{N}(32,0.8^2)\) and \(\beta=P(\bar{X}<31.316)=P\!\left(Z<\frac{31.316-32}{0.8}\right)=P(Z<-0.855)=0.196\). Step 3 — power \(=1-0.196=0.804\): the test detects a genuine 2-hour improvement about four times in five. Step 4 — feel the trade-off: rerun at the 1% level and \(c=30+2.3263\times 0.8=31.861\), so \(\beta=P(Z<\frac{31.861-32}{0.8})=P(Z<-0.174)=0.431\). Halving nothing and quartering nothing — cutting the false-alarm rate from 5% to 1% more than doubled the miss rate, from 19.6% to 43.1%.

CaseWhich error matters? Context decides, and the examiner asks

The mathematics treats the two errors symmetrically; the world does not, and the highest-skill mark in this topic is matching the error to its real cost. In a criminal trial with \(H_0\colon\) 'the defendant is innocent', a Type I error convicts an innocent person and a Type II error acquits a guilty one — English law's Blackstone ratio ('better that ten guilty persons escape than that one innocent suffer') is an explicit judgement that \(\alpha\) should be tiny even at the cost of large \(\beta\). In medical screening the weights flip: with \(H_0\colon\) 'no disease', a Type I error is a false positive that costs a follow-up test and a bad week, while a Type II error is a missed cancer — so screening programmes run deliberately trigger-happy tests and mop up the false alarms downstream. Drug regulation splits the difference with a convention you should quote: trials are typically designed for \(\alpha=0.05\) and power \(0.8\) — an explicit declaration that missing a real effect (\(\beta=0.2\)) is treated as four times more tolerable than a false alarm.

When an exam question asks 'explain, in context, which error is more serious', it wants exactly this reasoning: name each error as a concrete wrong decision in the scenario ('the airline grounds a safe fleet' versus 'the airline keeps flying a faulty one'), attach the consequence to each, and conclude. And when it asks how to reduce both errors simultaneously, there is only one honest answer on a fixed test statistic: increase the sample size. Everything else — moving the significance level, widening the region — just slides cost from one error to the other, radar-operator style.

VocabularyKey terms the mark scheme pays for

Type I error

Rejecting \(H_0\) when it is true — the false alarm. Its probability is the actual significance level: \(P(X\in\text{critical region}\mid H_0)\).

Type II error

Failing to reject \(H_0\) when it is false — the miss. Its probability \(\beta\) can only be computed for a specified true parameter value.

Actual significance level

The true probability of a Type I error. Equals the nominal level for normal tests; for binomial and Poisson tests it is the probability of the discrete critical region, usually below the nominal level.

Critical region

The set of outcomes that triggers rejection of \(H_0\). Built from the \(H_0\) distribution so its probability is at most the nominal significance level — never rebuilt using the alternative.

Acceptance region

The complement of the critical region. A Type II error is precisely the event that the observation lands here while \(H_0\) is false, so \(\beta\) is its probability under the true parameter.

Power of a test

\(1-\beta\): the probability the test rejects \(H_0\) when a specified alternative is true. Grows with sample size and with the distance between the null and true parameter values.

Nominal versus actual level

The nominal level is the advertised \(\alpha\) (e.g. 5%); the actual level is the achieved \(P(\text{Type I error})\). They differ whenever the test statistic is discrete.

Error trade-off

For a fixed sample, shrinking \(\alpha\) (a stricter test) enlarges \(\beta\), and vice versa. Only a larger sample — or a larger true effect — reduces both together.

TrapsMisconceptions that cost marks

“A test at the 5% significance level has a 5% probability of a Type I error.”

Actually: Only when the test statistic is continuous (the normal tests). For binomial and Poisson tests the critical region's probability jumps in discrete steps, so the actual level is the largest achievable value not exceeding 5% — in the worked example it was 3.55%, and quoting 5% there loses the mark.

“P(Type II error) is a single number, like the significance level.”

Actually: It depends entirely on what the true parameter is: \(\beta\) is a function of the alternative value, near \(1-\alpha\) for truths barely different from \(H_0\) and near 0 for distant ones. That is why every question must specify the true value before \(\beta\) can be found — and why your answer should say 'given that \(p=0.1\)' rather than presenting \(\beta\) as a property of the test alone.

“The critical region should be recalculated with the true parameter when finding the Type II error.”

Actually: Never. The critical region is fixed by the \(H_0\) distribution and the significance level before any truth is revealed; the true parameter enters only when you evaluate the probability of the acceptance region. Building the region from the alternative distribution is the single most heavily penalised structural error in this topic.

“A non-significant result means \(H_0\) is true.”

Actually: It means the test failed to detect a difference — which, when power is low, is weak evidence of anything. The components test had power 0.677 against \(p=0.1\): a full third of the time it 'accepts' \(H_0\) even though the reject rate has collapsed. Insufficient evidence is not evidence of absence.

ExamWhat examiners want

Definitions earn their marks only in context. 'A Type I error is rejecting a true null hypothesis' is the textbook line; the examiner wants 'the inspector concludes the mean fill volume has changed when it has not'. Train the translation both ways for every scenario you meet, and keep the direction straight with one anchor: Type I is the false alarm (an error about a true \(H_0\)), Type II is the miss (an error about a false \(H_0\)).

Structure every calculation as the two-step recipe and show the seam. Step one, under \(H_0\): state the distribution, build or quote the critical region, and give \(P(\text{Type I error})\) as the actual probability of that region — for discrete tests, show the cumulative comparison that fixes the boundary (\(P(X\le 2)=0.0355\le 0.05\) but \(P(X\le 3)=0.1071>0.05\)), because that comparison is usually a mark on its own. Step two, under the stated truth: write the new distribution explicitly (\(X\sim\mathrm{B}(20,0.1)\)), evaluate the acceptance region's probability, and label it \(\beta\); power is \(1-\beta\), stated as a separate line. For normal tests, keep the critical value \(c\) in raw units (hours, grams) rather than \(z\)-units — the Type II standardisation then reads cleanly as \((c-\mu_1)\) over the standard error, and sign errors become visible. Finally, expect the interpretive coda: 'comment on the power of the test' wants a sentence linking the number to the design — low power means a real effect of this size will usually go undetected, and the remedy is a larger sample, not a different significance level.