In addition to the accuracy of predictions made by an ML model, it’s also important to understand exactly how the model makes its predictions. Understanding how a model makes predictions provides better insights into the nature of the data and patterns within it, offers the possibility for further model refinement, and increases confidence when using the obtained predictions.

One of the most widespread algorithms used to explain model predictions is SHAP (SHapley Additive exPlanations).

Concept

The SHAP algorithm was developed in 2017 (link to the original paper) and is derived from the concept of Shapley values in game theory. The basic idea behind this concept is that, in order to determine the individual contribution of each player to the outcome of the game, it is necessary to consider the game’s outcome for every possible combination of players. This concept is translated to ML models by substituting players with variables and the game with the model’s predictions.

The SHAP algorithm belongs to the class of algorithms that can be applied to any type of ML model and can explain variable importance at the model level, as well as provide explanations for individual predictions.

Calculating SHAP Values

To determine SHAP values for a model, it is first necessary to create a set of all possible combinations of variables used in the final model (a partition set). Then, models are trained for each of these combinations (with all models being otherwise identical). These trained models are grouped according to the number of variables used during training.

The described process can be visually represented using an example of predicting real estate prices. Let’s say the prediction uses three variables: age of the property, number of rooms, and area. For one specific property, it might look like this:

The image shows all possible combinations of variables and the predicted price for each combination.

Based on the results of the trained models, two elements that determine the final SHAP values are calculated: marginal contribution and weight.

The marginal contribution for a variable is calculated between all models, where in one model the given variable is included, and in the other, it is not. The marginal contribution then represents the difference in results between these paired models.

The weight assigned to each marginal contribution is determined according to two rules. The first rule is that the sum of the weights for a variable must equal 1, and the second rule is that all marginal contributions for models with the same number of variables must be equal. The first rule ensures that the entire contribution of the variable to the final model is captured, while the second rule ensures that each marginal contribution for variables at the same level is equal. For example, in the real estate price prediction model, the weights for marginal contributions would be 1/3 for models with one variable, 2/3 for models with two variables, and 1/3 for models with all three variables.

The obtained marginal contributions and weights are then used to calculate the weighted average, which results in the SHAP value.

According to this procedure, the SHAP value for the variable “area” in the real estate price prediction example would be calculated as follows:

• Marginal contribution relative to the average value is 108,000 – 100,000 = 8,000 with a weight of 1/3
• Marginal contribution relative to the model with the variable “age” is 135,000 – 120,000 = 15,000 with a weight of 1/6
• Marginal contribution relative to the model with the variable “rooms” is 98,000 – 95,000 = 3,000 with a weight of 1/6
• Marginal contribution relative to the model with the variables “age” and “rooms” is 124,000 – 118,000 = 6,000 with a weight of 1/3

The final SHAP value is:

1/3 * 8,000 + 1/6 * 15,000 + 1/6 * 3,000 + 1/3 * 6,000 = 7,666.67

This means that, for the specific property, the variable “area” increased the price relative to the average by 7,666.67.

Interpreting SHAP Values

For models solving regression problems, SHAP values show how much each variable shifts the prediction from the average value of the target variable.

For classification problems, SHAP values indicate the change in the probability of belonging to one of the classes of the target variable.

SHAP values are the most widely used algorithm for interpreting ML model results, and due to their ability to globally and locally present the reasons for predictions, they provide additional information for making decisions based on ML models.