Welcome to a new series on interpretability where I’ll go through methods to interpret models, starting from the linear regression to so called black-box ensembles. I wanted to start this series for a while, due to two reasons:
I feel like anything I write is helpful for me, like I’m explaining to my future self. So, anytime I can’t recall something my own blog becomes my go-to source.
There seems to be a lot of confusions within the ML field in terms of interpretability (at least among the new comers), where most of the time wording suggest causality when it shouldn’t. I think I learned a few approaches to be helpful for that as well.
If you want a book to find everything in one place your go-to source should be Interpretable ML by Christoph Molnar. Most of the approaches that I learned is from there. I should also give credit to Regression and Other Stories.
By the way, throughout the series I won’t dive much into how to fit models. So, I assume some familiarity with GLMs, decision tree, and bagging and boosting methods in general. For example, I won’t walk through the summary table below, assuming you already know how to read it.
Let’s start with a simple linear regression where we only have predictor show a simple trick that aids the interpretability.
model = smf.ols('kid_score ~ mom_iq', data=cognitive)results = model.fit()print(results.summary())
What the table above tells us is that: \(\displaystyle \text{E[kid's score | mom's iq]} = 25.7998 + 0.61 \cdot \text{mom's iq}\)
Which implies the intercept, 25.7998, is the expected value of kid’s score when the mom’s iq is 0. That doesn’t make sense, right? Nobody has an IQ score of 0.
So, a way to make it make sense is via centering the predictor so that the intercept corresponds to the mean. Let me show.
As you can see, intercept has changed since now it corresponds to the expected value of kid’s score when the mom_iq is the average.
print(np.round(cognitive['kid_score'].mean(), 4))
86.7972
What about the slope? Isn’t it interpreted as one unit increase in mom’s IQ is associated with 0.61 increase of kid’s score?
I suggest you to stay away from such interpretation. The model does not provide information about changes within units but about comparisons between units. So, the most secure way of interpretation is as follows: Under the model, a kid with a mom’s score of x+1 will be, on average, 0.61 IQ higher compared to a kid with a mom’s score of x.
Let’s extend this to a more general setting where there’s more than 1 variable.
model = smf.ols('kid_score ~ mom_iq + mom_hs', data=cognitive)results = model.fit()print(results.summary())
In this setting, the coefficient corresponds to the expected difference between two observations, or group of observations, that only differ on the interested predictor by 1. Very simple way of showing it is like below:
if I take the difference: \(\displaystyle (\beta_0 + \beta_1X_1 + \beta_1 + \beta_2X_2) - (\beta_0 + \beta_1X_1 + \beta_2X_2) = \beta_1\)
This also points out the problem with collinearity. If \(X_1\) and \(X_2\) changes together, how am I gonna be left with \(\beta_1\) alone since others won’t cancel each other. Hence, we won’t be able to attribute the change to a single variable.
With these interpretations from the table in mind, we can also make use of plots which are more intuitive.
With plots, it’s very easy to understand the model and its assumptions. By looking at the plot, for example, it’s apparent that coefficient of mom’s iq does not depend on high school variable — the slope is the same for both levels of high school.
We can include an interaction term in the model specification, so that it can depend on the level of high school.
Plots show that the lines are not parallel, so coefficients are not the same for both levels of high school variable.
I want to emphasize this, interaction effect is ADDITIVE to the main effect. Since interpretation of “while holding all other variables constant” is meaningless - we can’t hold mom_iq constant while increasing the interaction variable.
What it means to be additive? Well, think of it this way. The coefficient of interaction is negative. When the high school level is yes, we add -0.4843 to the slope of the mom_iq. Since the coefficient of mom_iq positive, it means that mom_iq matters less when the level of high school is “yes”.
I hope this post clarified some aspects of interpretation of linear models. Of course, there are extensions of the classic linear regression, called Generalized Linear Models (GLMs). Actually, linear regression itself is a special case in GLM framework where the link function is identity function.
Anyway, I believe I’ll follow this up with model agnostic methods since most of you go for ML models. However, make no mistake, you may actually need these parametric approaches since you don’t always end up with a lot of data.
