Basic Probability Theory – YourStatsGuru

Probability is something we live with everyday, but it is often misinterpreted. It can be, sometimes, counter-intuitive and make us think that we’ve made a mistake, but if we follow the rules we can’t go wrong.

Probability Rules!

Everything that we want to describe using probabilities is called an event, whether we are interested in the probability of winning the lottery or the probability of contracting a particular disease. The collection of events that make up all the possible outcomes is called the sample space (denoted as $S$ ).

The Sample Space

The probability that any event occurs is one, i.e.:

$Pr(S) = 1$

For example, at the start of most sporting matches, a coin is tossed to decide how the match starts, e.g. in tennis, the person who wins the toss can elect to serve or receive and the other player selects an end to start from. The coin has two sides, typically called "Heads" ( $H$ ) and "Tails" ( $T$ ), therefore the sample space ( $S$ ) is:

$S = \left\{H,T\right\}$

When the coin is tossed, it must land on one side or the other and hence the probability that an event occurs is 1. Note that we are not saying anything about the probability of the events "Heads" and "Tails" except that one of them occurs when we toss the coin.

Probability that an event occurs

The probability that event $A$ occurs is found by taking the number of outcomes where $A$ occurs and dividing it by the total number of outcomes, i.e.:

$Pr\left(A\right) = \dfrac{\textrm{Number of outcomes where A occurs}}{\textrm{Total number of outcomes}}$

We will use the notation $Pr\left(A\right)$ to indicate the "probability of event $A$ ". Some texts use an alternate notation of $P\left(A\right)$ , both notations are perfectly acceptable and which you use is really a matter of preference.

Boundary values for any probability

For any event, the event must never occur, always occur or sometimes occur. The probability of any event also reflects this and so we can see that all probabilities must be bounded between "it never happens" and "it always happens". From above it follows that if the event $A$ never occurs then $Pr\left(A\right) = 0$ (since the number of times $A$ occurs is zero). Equally, if $A$ always occurs then $Pr\left(A\right) = 1$ because the number of times $A$ occurs is equal to the total number of outcomes. Thus we get the following definition:

The probability of any event must be between zero and one, i.e.:

$0 \leq Pr\left(A\right) \leq 1$

Complementary event

Sometimes we are not really interested in the probability that a particular event occurs, rather we want to know what is the probability that it doesn’t occur, e.g. the probability of not rolling a 12. The event " $A$ does not occur" is the complementary event which we typically denote as $A^{c}$ . The probability that event $A$ does not occur (called the complement of $A$ ) is given by:

$Pr\left(A^{c}\right) = 1 - Pr\left(A\right)$

Note: Some texts use the alternate notation of $A^{\prime}$ to denote the complement of $A$ . I prefer not to use this notation to avoid accidental confusion with the derivative of $A$ (from calculus shorthand), especially given we often rely on the techniques of calculus to find probabilities.

Typically we want to know something about the probability of two events occurring. We have to be very careful to make sure that we define what we want to know correctly. For example, when we say “Bill and Mary passed the course” we mean that both Bill and Mary passed the course. When we say “Bill or Mary passed the course” we mean that just Bill passed, or just Mary passed or both passed the course. Unfortunately, many people end up using the word “and” when they mean “or” and vice versa. To help you get it right, the following tables illustrate the difference between the meanings:

We can represent this visually using the good old Venn diagram:

The event $A$ and $B$ (symbolically written as $A \cap B$ ).

The event $A$ or $B$ (symbolically written as $A \cup B$ ).

From these diagrams we can see the following:

Union (Or) of events

The probability that either event $A$ or event $B$ (or both) occur is given by:

$Pr\left(A \cup B\right) = Pr\left(A\right) + Pr\left(B\right) - Pr\left(A \cap B\right)$

Intersection (And) of events

The probability that both event $A$ and event $B$ occur is given by:

$Pr\left(A \cap B\right) = Pr\left(A\right) \times Pr\left(B|A\right)$

Oops! "What’s this $Pr\left(B|A\right)$ thing?", I hear you ask. Well, $Pr\left(B|A\right)$ is the probability that $B$ occurs given that $A$ has occurred, what we call a conditional probability. The following should help you to get the idea.

Conditional Probability

Suppose we were to ask a person to choose a number between 1 and 10 (inclusive). Thus our sample space would be $S = \left\{1,2,3,4,5,6,7,8,9,10\right\}$ . We might define $A$ to be the event that the person selects a number less than 5 (i.e. $A = \left\{1,2,3,4\right\}$ ) and $B$ to be the event that the person selects an even number (i.e. $B = \left\{2,4,6,8,10\right\}$ ). Now consider the following questions:

What would be the probability that the person selects an even number given they choose a number less than 5, i.e. $Pr\left(B|A\right) =$ ?

We can see that if the person has chosen a number less than 5 then there are only two even numbers out of the four possible numbers in $A$ , so $Pr\left(B|A\right)$ should be $\frac{2}{4}$ or 0.5. Rearranging the intersection equation we can see that:

$Pr\left(B|A\right) = \dfrac{Pr\left(A \cap B\right)}{Pr\left(A\right)}$

and we can easily see that $Pr\left(A\right) = \frac{4}{10}$ and $Pr\left(A \cap B\right) = \frac{2}{10}$ (since $A$ and $B$ only have 2 common elements out of the sample space of 10 elements). Using these values in the rearranged equation, we find that

$Pr\left(B|A\right) = \dfrac{\frac{2}{10}}{\frac{4}{10}}=\dfrac{2}{4}$

as expected.

What would be the probability that the person selects a number less than 5 given they choose an even number, i.e. $Pr\left(A|B\right) =$ ?

We can see that if the person has chosen an even number then there are only two numbers out of the five possible numbers in $B$ , so $Pr\left(A|B\right)$ should be $\frac{2}{5}$ or 0.4. We know that $Pr\left(B\right) = \frac{5}{10}$ and $Pr\left(A \cap B\right) = \frac{2}{10}$ . Using these values in the rearranged equation, we find that

$Pr\left(A|B\right) = \dfrac{\frac{2}{10}}{\frac{5}{10}}=\dfrac{2}{5}$

as expected.

Now that we know how to find a conditional probability, we can use this in determining if two events are independent or mutually exclusive, but we must be careful not to confuse these definitions. Two events that are independent cannot be mutually exclusive, just as mutually exclusive events cannot be independent.

Independent events

Two events ( $A$ , $B$ ) are said to be independent when knowing that one event ( $B$ ) has occurred tells you no additional information about the probability of the other event ( $A$ ) occurring, i.e.:

$Pr\left(A|B\right) = Pr\left(A\right)$ and $Pr\left(B|A\right) = Pr\left(B\right)$

Mutually exclusive events

Two events ( $A$ , $B$ ) are said to be mutually exclusive when knowing that one event ( $B$ ) has occurred tells you that the probability of the event ( $A$ ) occurring is zero, i.e.:

$Pr\left(A|B\right) = 0$ or $Pr\left(A \cap B\right) = 0$

The following examples and exercise should help you get these definitions clear in your mind.

Example of Independent Events

Suppose event $A$ represents the outcome from the roll of a six sided die and event $B$ represents the outcome from tossing a coin, thus $A = \left\{1,2,3,4,5,6\right\}$ and $B = \left\{Heads, Tails\right\}$ . It should be evident that knowing that you rolled a 3 on the dice tells you nothing about the outcome of the coin toss and similarly, knowing that you got a tail from the coin toss tells you nothing about the outcome of the roll of the die. As such, events $A$ and $B$ would be called independent events.

Example of Mutually Exclusive Events

Suppose that we look at the final result for a student in a subject, caring only if the student passes or fails. Since a student cannot both pass and fail at the same time, it follows that the events are mutually exclusive, i.e.:

$Pr\left(Pass|Fail\right) = 0$ or $Pr\left(Pass \cap Fail\right) = 0$

In simple cases, like those above, finding a conditional probability is fairly straight forward, however, we typically are not dealing with such simple cases. Often we know $Pr\left(B|A\right)$ but we really want to know $Pr\left(A|B\right)$ . For example, we might know the probability of getting a positive test result if you have a particular disease, but what we would really like to know is the probability that you have the disease. Thankfully, a man called Reverend Thomas Bayes had a way of reversing the conditioning so that now we can find what we really want.

Bayes’ Rule

The conditional probability $Pr\left(A|B\right)$ can be found by:

$Pr\left(A|B\right) = \dfrac{Pr\left(B|A\right) \times Pr\left(A\right)}{Pr\left(B|A\right) \times Pr\left(A\right)+Pr\left(B|A^{c}\right) \times Pr\left(A^{c}\right)} = \dfrac{Pr\left(B|A\right) \times Pr\left(A\right)}{Pr\left(B\right)}$

It is important to note that Bayes’ Rule does not always apply. For example, in Null Hypothesis Significance Testing, one cannot compute $Pr\left(H_{0}|Data\right)$ from the p-value ( $Pr\left(Data|H_{0}\right)$ ).

CITE THIS AS:
Ovens, Matthew. “Basic Probability Theory” Retrieved from YourStatsGuru.

First published 2010 | Last updated: 6 November 2024