15.3 The logistic regression model¶

Throughout this session we will assume that the outcome $Y$ is binary. Further, we assume that $Y$ takes a value of 0 (“failure”) or 1 (“success”). As we discussed earlier, the terminology of success and failure does not imply success is a good thing; in health applications “success” often refers to a bad outcome such as death.

We will initially consider the simple situation with a single independent variable of interest, $X$. We assume that conditional on $X$, the outcome $Y$ follows a Bernoulli distribution:

\[ Y | X=x \sim Bernoulli(\pi_x) \]

Then $\pi_x$ is the conditional probability of sucess, given $X=x$. It also represents the conditional expectation of the outcome, given $X=x$.

\[ \pi_x = E[Y | X=x] = P(Y=1 | X=x) \]

Typically, our research question involves relating this probability to the covariate(s).

15.3.1 Components of the model¶

The logit function¶

As we have discussed, we do not wish to directly model $\pi$, because fitted values can lie outside the possible range of values. Instead, we will first transform $\pi$. In other words, we will model a function of $\pi$. We want a one-to-one function (so we can back-transform to the original scale, if we wish) that maps a probability $\pi$ to the whole real line.

The function that we use in logistic regression is called the logit function. Specifically,

\[ logit(\pi) = log\left(\frac{\pi}{1-\pi}\right) \]

The probability $\pi$ lies in the interval $[0,1]$ but the transformed value, $logit(\pi)$ lies in the range $(-\infty, \infty)$.

It will also be useful to know how to back-transform. If $logit(\pi) = L$ then

\[ \pi = \frac{exp(L)}{1 + exp(L)} \]

This relationship will allow us to obtain fitted probabilities from our logistic regression model.

Odds¶

Suppose we have a binary outcome, where the probability of success is $\pi$, i.e. $P(Y=1) = \pi$. Then the odds of success are given by

\[ \frac{\pi}{1-\pi} \]

Therefore, $logit(\pi)$ is the logarithm of the odds, or the log-odds. We will see below that using the logit function leads to the parameters of the regression model being interpreted in terms of odds and odds ratios.

The linear predictor¶

Just as for linear regression models, the linear predictor is an additive function of the independent variables. With a single covariate, it is simply:

\[ \beta_0 + \beta_1 X \]

15.3.2 The basic logistic regression model¶

The equation for a logistic regression model with, relating $X$ to a binary outcome $Y$ is:

\[ logit(\pi_x) = \beta_0 + \beta_1 X \]

Note that, unlike linear regression, there is no explicit error term in the logistic regression model.

Interpreting the parameters¶

Suppose that our single covariate $X$ is binary, taking values 1 (exposed, say) and 0 (unexposed). Our model is then:

\[\begin{split} logit(\pi_x) = \begin{cases} \beta_0 &\text{when $X$=0 (unexposed group)} \\ \beta_0 + \beta_1 &\text{when $X$=1 (exposed group)} \end{cases} \end{split}\]

In other words, we have:

\[\begin{split} \begin{align*} \beta_0 &\qquad \text{is the log-odds of the outcome in the unexposed group} \\ \beta_0 + \beta_1 &\qquad \text{is the log-odds of the outcome in the exposed group} \end{align*} \end{split}\]

Taking the exponential, we have

\[\begin{split} \begin{align*} e^{\beta_0} &\qquad \text{is the odds of the outcome in the unexposed group} \\ e^{\beta_0 + \beta_1} &\qquad \text{is the odds of the outcome in the exposed group} \end{align*} \end{split}\]

Now we have that $e^{\beta_0 + \beta_1} = e^{\beta_0} \times e^{\beta_1}$. Therefore, $e^{\beta_1}$ also represents the multiplicative increase in the odds, going from the unexposed group to the exposed group. This multiplicative increase is known as the odds ratio. Therefore, we can also write:

\[\begin{split} \begin{align*} e^{\beta_0} &\qquad \text{is the odds of the outcome in the unexposed group} \\ e^{\beta_1} &\qquad \text{is the odds ratio of the outcome, comparing the exposed group to the unexposed group} \end{align*} \end{split}\]

General interpretation¶

This leads us to the following interpretation of the model: