Conditional Probability, Independence and Bayes’ Rule

STA 211 · Lecture 04

Introduction

Conditional probability revises the probability of an event when additional information is available. It is the foundation of diagnostic testing, classification, reliability analysis, risk assessment and Bayesian inference.

Learning Objectives

You should be able to:

• calculate conditional probabilities;
• use multiplication and chain rules;
• test event independence;
• apply the law of total probability; and
• use Bayes’ theorem to reverse conditioning.

Pre-Test

1. What does $P(A \cap B)$ represent?
2. If a card is drawn from a standard deck and is known to be red, how many possible cards remain?

Conditional Probability

For events $A$ and $B$ with $P(B) > 0$,

$$P(A \mid B) = \frac{P(A \cap B)}{P(B)}.$$

Conditioning restricts the sample space to $B$ and then measures the proportion of $B$ that also lies in $A$.

Example 4.1: Two coin tosses

A fair coin is tossed twice. Given that at least one head occurred, find the probability that the second toss was a head.

The restricted outcomes are $HH,HT,TH$. Two of these have a head on the second toss, so

$$P\left( \text{second is H} \mid \text{at least one H} \right) = \frac{2}{3}.$$

Multiplication and Chain Rules

Rearranging the definition gives

$$P(A \cap B) = P(B)P(A \mid B) = P(A)P(B \mid A).$$

For three events,

$$P(A \cap B \cap C) = P(A)P(B \mid A)P(C \mid A \cap B).$$

More generally,

$$P\left( \underset{i = 1}{\bigcap^{n}}A_{i} \right) = P\left( A_{1} \right)\prod_{i = 2}^{n}P\left( A_{i} \mid A_{1} \cap \cdots \cap A_{i - 1} \right).$$

Example 4.2: Sampling without replacement

A box contains 10 red and 10 green balls. Three balls are drawn without replacement. The probability that all are red is

$$\frac{10}{20} \cdot \frac{9}{19} \cdot \frac{8}{18} = \frac{2}{19} \approx 0.1053.$$

Independence

Events $A$ and $B$ are independent if

$$P(A \cap B) = P(A)P(B).$$

When $P(B) > 0$, this is equivalent to

$$P(A \mid B) = P(A).$$

Independence means that knowledge of one event does not alter the probability of the other.

Mutual versus pairwise independence

Events $A_{1},A_{2},A_{3}$ are mutually independent if all pairwise factorisations hold and

$$P\left( A_{1} \cap A_{2} \cap A_{3} \right) = P\left( A_{1} \right)P\left( A_{2} \right)P\left( A_{3} \right).$$

Pairwise independence alone does not guarantee mutual independence.

Mutually exclusive is not independent

If $A$ and $B$ are mutually exclusive with positive probabilities, then $P(A \cap B) = 0$ but $P(A)P(B) > 0$. Therefore, they are dependent.

Probability Trees

A tree diagram makes sequential conditioning explicit. Probabilities multiply along a path and add across mutually exclusive paths.

Probability tree for two-stage events

Probability tree for two-stage events

Law of Total Probability

If $A_{1},\ldots,A_{k}$ form a partition of $\Omega$ and $P\left( A_{i} \right) > 0$, then for any event $B$,

$$P(B) = \sum_{i = 1}^{k}P\left( B \mid A_{i} \right)P\left( A_{i} \right).$$

Example 4.3: Damaged goods

A company receives 70% of its goods from supplier A and 30% from supplier B. The damage rates are 4% and 7%, respectively. Then

$$P(D) = 0.04(0.70) + 0.07(0.30) = 0.049.$$

Bayes’ Theorem

For a partition $A_{1},\ldots,A_{k}$ and evidence $B$,

$$P\left( A_{j} \mid B \right) = \frac{P\left( B \mid A_{j} \right)P\left( A_{j} \right)}{\sum_{i = 1}^{k}P\left( B \mid A_{i} \right)P\left( A_{i} \right)}.$$

Bayes’ theorem combines a prior probability $P\left( A_{j} \right)$ with the likelihood $P\left( B \mid A_{j} \right)$ to obtain the posterior probability $P\left( A_{j} \mid B \right)$.

Example 4.4: Medical screening

Suppose disease prevalence is 2%, sensitivity is 95%, and specificity is 90%. Let $D$ denote disease and $+$ a positive test.

$$P(D) = 0.02,\quad\quad P( + \mid D) = 0.95,$$

$$P\left( + \mid D^{c} \right) = 1 - 0.90 = 0.10.$$

The total probability of a positive test is

$$P( + ) = 0.95(0.02) + 0.10(0.98) = 0.117.$$

Therefore,

$$P(D \mid + ) = \frac{0.95(0.02)}{0.117} \approx 0.1624.$$

Although the test is sensitive, most positive results are false positives because the disease is uncommon.

Post-Test

1. If $P(A) = 0.6$, $P(B) = 0.5$ and $P(A \cap B) = 0.3$, are $A$ and $B$ independent?
2. A bag has 6 white and 4 black balls. Two are drawn without replacement. Find the probability that both are white.
3. A factory has three machines producing 50%, 30% and 20% of output, with defect rates 1%, 2% and 4%. Find the overall defect rate.
4. For Question 3, find the probability that a defective item came from machine 3.
5. Explain why mutually exclusive positive-probability events cannot be independent.

Selected answers: 1. Yes; 2. $\frac{6}{10}\frac{5}{9} = \frac{1}{3}$; 3. $0.019$; 4. $0.008/0.019 \approx 0.4211$.

Previous lecture

Course home

Next lecture