A Responsible Machine Learning Workflow with Focus on Interpretable Models, Post-hoc Explanation, and Discrimination Testing

Abstract

This manuscript outlines a viable approach for training and evaluating machine learning systems for high-stakes, human-centered, or regulated applications using common Python programming tools. The accuracy and intrinsic interpretability of two types of constrained models, monotonic gradient boosting machines and explainable neural networks, a deep learning architecture well-suited for structured data, are assessed on simulated data and publicly available mortgage data. For maximum transparency and the potential generation of personalized adverse action notices, the constrained models are analyzed using post-hoc explanation techniques including plots of partial dependence and individual conditional expectation and with global and local Shapley feature importance. The constrained model predictions are also tested for disparate impact and other types of discrimination using measures with long-standing legal precedents, adverse impact ratio, marginal effect, and standardized mean difference, along with straightforward group fairness measures. By combining interpretable models, post-hoc explanations, and discrimination testing with accessible software tools, this text aims to provide a template workflow for machine learning applications that require high accuracy and interpretability and that mitigate risks of discrimination.

1. Introduction

Responsible artificial intelligence (AI) has been variously conceptualized as AI-based products or projects that use transparent technical mechanisms, that create appealable decisions or outcomes, that perform reliably and in a trustworthy manner over time, that exhibit minimal social discrimination, and that are designed by humans with diverse experiences, both in terms of demographics and professional backgrounds (e.g., Responsible Artificial Intelligence, Responsible AI: A Framework for Building Trust in Your AI Solutions, PwC’s Responsible AI, Responsible AI Practices). Although responsible AI is today a somewhat broad and amorphous notion, at least one aspect is becoming clear. Machine learning (ML) models, a common application of AI, can present serious risks. ML models can be inaccurate and unappealable black-boxes, even with the application of newer post-hoc explanation techniques [1] (e.g., When a Computer Program Keeps You in Jail). ML models can perpetuate and exacerbate discrimination [2,3,4], and ML models can be hacked, resulting in manipulated model outcomes or the exposure of proprietary intellectual property or sensitive training data [5,6,7,8]. This manuscript makes no claim that these interdependent issues of ML opaqueness, discrimination, privacy harms, and security vulnerabilities have been resolved, even as singular entities, and much less as complex intersectional phenomena. However, Section 2, Section 3 and Section 4 do propose some specific technical countermeasures, mostly in the form of interpretable models, post-hoc explanation, and disparate impact (DI) and discrimination testing, that responsible practitioners can use to address a subset of these vexing problems. (In the United States (US), interpretable models, explanations, DI testing, and the model documentation they enable may also be required under the Civil Rights Acts of 1964 and 1991, the Americans with Disabilities Act, the Genetic Information Nondiscrimination Act, the Health Insurance Portability and Accountability Act, the Equal Credit Opportunity Act (ECOA), the Fair Credit Reporting Act (FCRA), the Fair Housing Act, Federal Reserve Supervision and Regulation (SR) Letter 11-7, the European Union (EU) General Data Protection Regulation (GDPR) Article 22, or other laws and regulations [9]. Note that this text and associated software are not, and should not be construed as, legal advice or regulatory compliance requirements.)

Section 2 describes methods and materials, including training datasets, interpretable and constrained models, post-hoc explanations, tests for DI and other social discrimination, and public and open source software resources associated with this text. In Section 3, interpretable and constrained modeling results are compared to less interpretable and unconstrained models, and post-hoc explanation and discrimination testing results are also presented for interpretable models. Of course, an even wider array of tools and techniques are likely helpful to fully minimize discrimination, inaccuracy, privacy, and security risks associated with ML models. Section 4 puts forward a more holistic responsible ML modeling workflow, and addresses the burgeoning Python ecosystem for responsible AI, along with appeal and override of automated decisions, and discrimination testing and remediation in practice. Section 5 closes this manuscript with a brief summary of the outlined methods, materials, results, and discussion.

2. Materials and Methods

The simulated data (see Section 2.1) are based on the well-known Friedman datasets. Its known feature importance and augmented discrimination characteristics are used to gauge the validity of interpretable modeling, post-hoc explanation, and discrimination testing techniques [10,11]. The mortgage data (see Section 2.2) are sourced from the Home Mortgage Disclosure Act (HMDA) database, a fairly realistic data source for demonstrating the template workflow [12] (see Mortgage data (HMDA)). To provide a sense of fit differences, performance is compared on simulated data and collected mortgage data between the more interpretable constrained ML models and the less interpretable unconstrained ML models. Because the unconstrained ML models, gradient boosting machines (GBMs, e.g., [13,14]) and artificial neural networks (ANNs, e.g., [15,16,17,18]), do not exhibit convincing accuracy benefits on the simulated or mortgage data and can also present the unmitigated risks discussed above, further explanation and discrimination analyses are applied only to the constrained, interpretable ML models [1,19,20]. Here, monotonic gradient boosting machines (MGBMs, as implemented in XGBoost or h2o, see Section 2.3) and explainable neural networks (XNNs, e.g., [21,22], see Section 2.4) will serve as those more interpretable models for subsequent explanatory and discrimination analyses. MGBM and XNN interpretable model architectures are selected for the example workflow because they are straightforward variants of popular unconstrained ML models. If practitioners are working with GBM and ANN models, it should be relatively uncomplicated to also evaluate the constrained versions of these models.

The same can be said of the selected explanation methods and discrimination tests. Due to their post-hoc nature, they can often be shoe-horned into existing ML workflows and pipelines. Presented explanation techniques include partial dependence (PD) and individual conditional expectation (ICE) (see Section 2.5) and Shapley values (see Section 2.6) [14,23,24,25]. PD, ICE, and Shapley values provide direct, global, and local summaries and descriptions of constrained models without resorting to the use of intermediary and approximate surrogate models. Discrimination testing methods discussed (see Section 2.7) include adverse impact ratio (AIR, see Part 1607—Uniform Guidelines on Employee Selection Procedures (1978) §1607.4), marginal effect (ME), and standardized mean difference (SMD) [2,26,27]. Accuracy and other confusion matrix measures are also reported by demographic segment [28]. All outlined materials and methods are implemented in open source Python code, and are made available on GitHub (see Section 2.8).

2.1. Simulated Data

Simulated data are created based on a signal-generating function, f, applied to input data, $\mathbf{X}$, first proposed in Friedman [10] and extended in Friedman et al. [11]:

$$\begin{array}{c}\hfill f\left(\mathbf{X}\right)=10sin\left(\pi X_{\mathrm{Friedman},1}{X}_{\mathrm{Friedman},2}\right)+20{(X_{\mathrm{Friedman},3}-0.5)}^2+10X_{\mathrm{Friedman},4}+5X_{\mathrm{Friedman},5}\end{array}$$

(1)

where each $X_{\mathrm{Friedman},j}$ is a random uniform feature in $[0,1]$. In Friedman’s texts, a Gaussian noise term was added to create a continuous output feature for testing spline regression methodologies. In this manuscript, the signal-generating function and input features are modified in several ways. Two binary features, a categorical feature with five discrete levels, and a bias term are introduced into f to add a degree of complexity that may more closely mimic real-world settings. For binary classification analysis, the Gaussian noise term is replaced with noise drawn from a logistic distribution and coefficients are re-scaled to be $\frac{1}{5}$ of the size of those used by Friedman, and any $f\left(\mathbf{X}\right)$ value above 0 is classified as a positive outcome, while $f\left(\mathbf{X}\right)$ values less than or equal to zero are designated as negative outcomes. Finally, f is augmented with two hypothetical protected class-control features with known dependencies on the binary outcome to allow for discrimination testing. The simulated data are generated to have eight input features, twelve after numeric encoding of categorical features, and a binary outcome, two class-control features, and 100,000 instances. The simulated data are then split into a training and test set, with 80,000 and 20,000 instances, respectively. Within the training set, a five-fold cross-validation indicator is used for training all models. For an exact specification of the simulated data, see the software resources referenced in Section 2.8.

2.2. Mortgage Data

The mortgage dataset analyzed here is a random sample of consumer-anonymized loans from the HDMA database. These loans are a subset of all originated mortgage loans in the 2018 HMDA data that were chosen to represent a relatively comparable group of consumer mortgages. A selection of features is used to predict whether a loan is high-priced, i.e., the annual percentage rate (APR) charged was 150 basis points (1.5%) or more above a survey-based estimate of other similar loans offered around the time of the given loan. After data cleaning and preprocessing to encode categorical features and create missing markers, the mortgage data contain ten input features and the binary outcome, high-priced. The data are split into a training set with 160,338 loans and a marker for 5-fold cross-validation and a test set containing 39,662 loans. While lenders would almost certainly use more information than the selected features to determine whether to offer and originate a high-priced loan, the selected input features (loan to value (LTV) ratio, debt to income (DTI) ratio, property value, loan amount, introductory interest rate, customer income, etc.) are likely to be some of the most influential factors that a lender would consider. See the resources put forward in Section 2.8 and Appendix A for more information regarding the HMDA mortgage data.

2.3. Monotonic Gradient Boosting Machines

MGBMs constrain typical GBM training to consider only tree splits that obey user-defined positive and negative monotonicity constraints, with respect to each input feature, $X_j$, and a target feature, $\mathbf{y}$, independently. An MGBM remains an additive combination of B trees trained by gradient boosting, $T_b$, and each tree learns a set of splitting rules that respect monotonicity constraints, ${\mathsf{\Theta }}_b^{\mathrm{mono}}$. For an instance, $\mathbf{x}$, a trained MGBM model, $g^{\mathrm{MGBM}}$, takes the form:

$$\begin{array}{cc}\hfill g^{\mathrm{MGBM}}\left(\mathbf{x}\right)& =\sum _{b=0}^{B-1}{T}_b\left(\mathbf{x};{\mathsf{\Theta }}_b^{\mathrm{mono}}\right)\hfill \end{array}$$

(2)

As in unconstrained GBM, ${\mathsf{\Theta }}_b^{\mathrm{mono}}$ is selected in a greedy, additive fashion by minimizing a regularized loss function that considers known target labels, the predictions of all subsequently trained trees in $g^{\mathrm{MGBM}}$, and the b-th tree splits applied to $\mathbf{x}$, $T_b(\mathbf{x};{\mathsf{\Theta }}_b^{\mathrm{mono}})$, in a numeric loss function (e.g., squared loss, Huber loss), and a regularization term that penalizes complexity in the current tree. See  Appendix B.1 and Appendix B.2 for details pertaining to MGBM training.

Herein, two $g^{\mathrm{MGBM}}$ models are trained. One on the simulated data and one on the mortgage data. In both cases, positive and negative monotonic constraints for each $X_j$ are selected using domain knowledge, random grid search is used to determine other hyperparameters, and five-fold cross-validation and test partitions are used for model assessment. For exact parameterization of the two $g^{\mathrm{MGBM}}$ models, see the software resources referenced in Section 2.8.

2.4. Explainable Neural Networks

XNNs are an alternative formulation of additive index models in which the ridge functions are neural networks [21]. XNNs also bear a strong resemblance to generalized additive models (GAMs) and so-called explainable boosting machines (EBMs or GA²Ms), which consider main effects and a small number of two-way interactions and may also incorporate boosting into their training [14,29]. XNNs enable users to tailor interpretable neural network architectures to a given prediction problem and to visualize model behavior by plotting ridge functions. A trained XNN function, $g^{\mathrm{XNN}}$, applied to some instance, $\mathbf{x}$, is defined as:

$$\begin{array}{c}\hfill g^{\mathrm{XNN}}\left(\mathbf{x}\right)={\mu }_0+\sum _{k=0}^{K-1}{\gamma }_k{n}_k\left(\sum _{j=0}^{J-1}{\beta }_{k,j}{x}_j\right)\end{array}$$

(3)

where ${\mu }_0$ is a global bias for K individually specified ANN subnetworks, $n_k$, with weights ${\gamma }_k$. The inputs to each $n_k$ are themselves a linear combination of the J modeling inputs and their associated ${\beta }_{k,j}$ coefficients in the deepest, i.e., projection, layer of $g^{\mathrm{XNN}}$.

Two $g^{\mathrm{XNN}}$ models are trained by mini-batch stochastic gradient descent (SGD) on the simulated data and mortgage data. Each $g^{\mathrm{XNN}}$ is assessed in five training folds and in a test data partition. $L_1$ regularization is applied to network weights to induce a sparse and interpretable model, where each $n_k$ and corresponding ${\gamma }_k$ are ideally associated with an important $X_j$ or combination thereof. $g^{\mathrm{XNN}}$ models appear highly sensitive to weight initialization and batch size. Be aware that $g^{XNN}$ architectures may require manual and judicious feature selection due to long training times. For more details regarding $g^{\mathrm{XNN}}$ training, see the software resources in Section 2.8 and Appendix B.1 and Appendix B.3.

2.5. One-Dimensional Partial Dependence and Individual Conditional Expectation

PD plots are a widely-used method for describing and plotting the estimated average prediction of a complex model, g, across some partition of data, $\mathbf{X}$, for some interesting input feature, $X_j\in \mathbf{X}$ [14]. ICE plots are a newer method that describes the local behavior of g with regard to values of an input feature in a single instance, $x_j$. PD and ICE can be overlaid in the same plot to create a holistic global and local portrait of the predictions for some g and $X_j$ [23]. When $\mathrm{PD}(X_j,g)$ and $\mathrm{ICE}(x_j,g)$ curves diverge, such plots can also be indicative of modeled interactions in g or expose flaws in PD estimation, e.g., inaccuracy in the presence of strong interactions and correlations [23,30]. For details regarding the calculation of one-dimensional PD and ICE, see the software resources in Section 2.8 and  Appendix B.1 and Appendix B.4.

2.6. Shapley Values

Shapley explanations are a class of additive, locally accurate feature contribution measures with long-standing theoretical support [24,31]. Shapley explanations are the only known locally accurate and globally consistent feature contribution values, meaning that Shapley explanation values for input features always sum to the model’s prediction, $g\left(\mathbf{x}\right)$, for any instance $\mathbf{x}$, and that Shapley explanation values should not decrease in magnitude for some instance of $x_j$ when g is changed such that $x_j$ truly makes a stronger contribution to $g\left(\mathbf{x}\right)$ [24,25]. Shapley values can be estimated in different ways, many of which are intractable for datasets with large numbers of input features. Tree Shapley Additive Explanations (SHAP) is a specific implementation of Shapley explanations that relies on traversing internal decision tree structures to efficiently estimate the contribution of each $x_j$ for some $g\left(\mathbf{x}\right)$ [25]. Tree SHAP has been implemented in popular gradient boosting libraries such as h2o, LightGBM, and XGBoost, and Tree SHAP is used to calculate accurate and consistent global and local feature importance for MGBM models in Section 3.2.2 and Appendix E.1. Deep SHAP is an approximate Shapley value technique that creates SHAP values for ANNs [24]. Deep SHAP is implemented in the shap package and is used to generate SHAP values for the two $g^{XNN}$ models discussed in Section 3.2.2 and Appendix E.1. For more information pertaining to the calculation of Shapley values, see Appendix B.1 and Appendix B.5.

2.7. Discrimination Testing Measures

Because many current technical discussions of fairness in ML appear inconclusive (e.g., Tutorial: 21 Fairness Definitions and Their Politics), this text will draw on regulatory and legal standards that have been used for years in regulated, high-stakes employment and financial decisions. The discussed measures are also representative of fair lending analyses and pair well with the mortgage data. (See Appendix C for a brief discussion regarding different types of discrimination in US legal and regulatory settings, and Appendix D for remarks on practical vs. statistical significance for discrimination measures.) One such common measure of DI used in US litigation and regulatory settings is ME. ME is simply the difference between the percent of the control group members receiving a favorable outcome and the percent of the protected class members receiving a favorable outcome:

$$\begin{array}{c}\hfill \mathrm{ME}\equiv 100·(\mathrm{Pr}(\widehat{\mathbf{y}}=1|X_c=1)-\mathrm{Pr}(\widehat{\mathbf{y}}=1|X_p=1))\end{array}$$

(4)

where $\widehat{\mathbf{y}}$ are the model decisions, $X_p$ and $X_c$ represent binary markers created from some demographic attribute, c denotes the control group (often whites or males), p indicates a protected group, and $\mathrm{Pr}(·)$ is the operator for conditional probability. ME is a favored DI measure used by the US Consumer Financial Protection Bureau (CFPB), the primary agency charged with regulating fair lending laws at the largest US lending institutions and for various other participants in the consumer financial market (see Supervisory Highlights, Issue 9, Fall 2015). Another important DI measure is AIR, more commonly known as a relative risk ratio in settings outside of regulatory compliance.

$$\begin{array}{c}\hfill \mathrm{AIR}\equiv \frac{\mathrm{Pr}(\widehat{\mathbf{y}}=1|X_p=1)}{\mathrm{Pr}(\widehat{\mathbf{y}}=1|X_c=1)}\end{array}$$

(5)

AIR is equal to the ratio of the proportion of the protected class that receives a favorable outcome and the proportion of the control class that receives a favorable outcome. Statistically significant AIR values below 0.8 can be considered prima facie evidence of discrimination. An additional long-standing and pertinent measure of DI is SMD. SMD is often used to assess disparities in continuous features, such as income differences in employment analyses, or interest rate differences in lending. It originates from work on statistical power, and is more formally known as Cohen’s d. SMD is equal to the difference in the average protected class outcome, ${\overline{\widehat{\mathbf{y}}}}_p$, minus the control class outcome, ${\overline{\widehat{\mathbf{y}}}}_c$, divided by a measure of the standard deviation of the population, ${\sigma }_{\widehat{\mathbf{y}}}$. (There are several measures of the standard deviation of the score that are typically used: 1. the standard deviation of the population, irrespective of protected class status, 2. a standard deviation calculated only over the two groups being considered in a particular calculation, or 3. a pooled standard deviation, using the standard deviations for each of the two groups with weights.) Cohen defined values of this measure to have small, medium, and large effect sizes if the values exceeded 0.2, 0.5, and 0.8, respectively.

$$\begin{array}{c}\hfill \mathrm{SMD}\equiv \frac{{\overline{\widehat{\mathbf{y}}}}_p-{\overline{\widehat{\mathbf{y}}}}_c}{{\sigma }_{\widehat{\mathbf{y}}}}\end{array}$$

(6)

The numerator in the SMD is roughly equivalent to ME but adds the standard deviation divisor as a standardizing factor. Because of this standardization factor, SMD allows for a comparison across different types of outcomes, such as inequity in mortgage closing fees or inequities in the interest rates given on certain loans. In this, one may apply definitions in Cohen [26] of small, medium, and large effect sizes, which represent a measure of practical significance, which is described in detail in Appendix D. Finally, confusion matrix measures in demographic groups, such as accuracy, false positive rate (FPR), false negative rate (FNR), and their ratios, are also considered as measures of DI in Section 3.2.3 and Appendix E.2.

2.8. Software Resources

Python code to reproduce discussed results is available at: https://github.com/h2oai/article-information-2019. The primary Python packages employed are: numpy 1.14.5 and pandas 0.22.0 for data manipulation, h2o 3.26.0.9, Keras 2.3.1, shap 0.31.0, and tensorflow 1.14.0 for modeling, explanation, and discrimination testing, and typically matplotlib 2.2.2 for plotting.

3. Results

Results are laid out for the simulated and mortgage datasets. Accuracy is compared for unconstrained, less interpretable $g^{\mathrm{GBM}}$ and $g^{\mathrm{ANN}}$ models and constrained, more interpretable $g^{\mathrm{MGBM}}$ and $g^{\mathrm{XNN}}$ models. Then, for the $g^{\mathrm{MGBM}}$ and $g^{\mathrm{XNN}}$ models, intrinsic interpretability, post-hoc explanation, and discrimination testing results are explored.

3.1. Simulated Data Results

Fit comparisons between unconstrained and constrained models and XNN interpretability results are discussed in Section 3.1.1 and Section 3.1.2. As model training and assessment on the simulated data is a rough validation exercise meant to showcase expected results on data with known characteristics, and given that most of the techniques in the proposed workflow are already used widely or have been validated elsewhere, reporting of simulated data results in the main text will focus mostly on fit measures and the more novel $g^{\mathrm{XNN}}$ interpretability results. The bulk of the post-hoc explanation and discrimination testing results for the simulated data are left to Appendix E.

3.1.1. Constrained vs. Unconstrained Model Fit Assessment

Table 1. Fit measures for $g^{\mathrm{GBM}}\left(\mathbf{X}\right)$, $g^{\mathrm{MGBM}}\left(\mathbf{X}\right)$, $g^{\mathrm{ANN}}\left(\mathbf{X}\right)$, and $g^{\mathrm{XNN}}\left(\mathbf{X}\right)$ on the simulated test data. Arrows indicate the direction of improvement for each measure and the best result in each column is displayed in bold font.

Model	Accuracy ↑	AUC ↑	F1 ↑	Logloss ↓	MCC ↑	RMSE ↓	Sensitivity ↑	Specificity ↑
$g^{\mathrm{GBM}}$	0.757	0.847	0.779	0.486	0.525	0.400	0.858	0.657
$g^{\mathrm{MGBM}}$	0.744	0.842	0.771	0.502	0.504	0.407	0.864	0.625
$g^{\mathrm{ANN}}$	0.757	0.850	0.779	0.480	0.525	0.398	0.858	0.657
$g^{\mathrm{XNN}}$	0.758	0.851	0.781	0.479	0.528	0.397	0.867	0.648

presents a variety of fit measures for $g^{\mathrm{GBM}}$, $g^{\mathrm{MGBM}}$, $g^{\mathrm{ANN}}$, and $g^{\mathrm{XNN}}$ on the simulated test data. $g^{\mathrm{XNN}}$ exhibited the best performance, but the models exhibited only a fairly small range of fit results. Interpretability and explainability benefits of the constrained models appeared to come at little cost to overall model performance, or in the case of $g^{\mathrm{ANN}}$ and $g^{\mathrm{XNN}}$, no cost at all. For the displayed measures, $g^{\mathrm{MGBM}}$ performed ∼2% worse on average than $g^{\mathrm{GBM}}$. $g^{\mathrm{XNN}}$ performed ∼0.5% better on average than $g^{\mathrm{XNN}}$, and $g^{\mathrm{XNN}}$ actually showed slightly better fit than $g^{\mathrm{ANN}}$ across all fit measures except specificity. Fit measures that required a probability cutoff were taken at the best F1 threshold for each model.

3.1.2. Interpretability Results

For $g^{\mathrm{XNN}}$, inherent interpretability manifested as plots of sparse ${\gamma }_k$ output layer weights, $n_k$ subnetwork ridge functions, and sparse ${\beta }_{j,k}$ weights in the bottom projection layer. Figure 1 provides detailed insights into the structure of $g^{\mathrm{XNN}}$ (also described in Equation (3)). Figure 1a displays the sparse ${\gamma }_k$ weights of the output layer, where only $n_k$ subnetworks with $k\in \{1,4,7,8,9\}$ were associated with large magnitude weights. The $n_k$ subnetwork ridge functions appear in Figure 1b as simplistic but distinctive functional forms. Color-coding between Figure 1a,b visually reinforces the direct feed-forward relationship between the $n_k$ subnetworks and the ${\gamma }_k$ weights of the output layer.

$n_k$ subnetworks were plotted across the output values of their associated ${\sum }_j{\beta }_{k,j}{x}_j$ projection layer hidden units, and color-coding between Figure 1b,c link the ${\beta }_{j,k}$ weights to their $n_k$ subnetworks. Most of the heavily utilized $n_k$ subnetworks had sparse weights in their ${\sum }_j{\beta }_{k,j}{x}_j$ projection layer hidden units. In particular, subnetwork $n_1$ appeared to be almost solely a function of $X_{\mathrm{Friedman},3}$ and appeared to exhibit the expected quadratic behavior for $X_{\mathrm{Friedman},3}$. Subnetworks $n_7$, $n_8$, and $n_9$ appeaed to be most associated with the globally important $X_{\mathrm{Friedman},1}$ and $X_{\mathrm{Friedman},2}$ features, likely betraying the effort required for $g^{\mathrm{XNN}}$ to model the nonlinear $\sin \left(\right)$ function of the $X_{\mathrm{Friedman},1}$ and $X_{\mathrm{Friedman},2}$ product, and these subnetworks, especially $n_7$ and $n_8$, appeared to display some noticeable sinusoidal characteristics. Subnetwork $n_4$ seemed to be a linear combination of all the original input $X_j$ features, but did weigh the linear $X_{\mathrm{Friedman},4}$ and $X_{\mathrm{Friedman},5}$ terms roughly in the correct two-to-one ratio. As a whole, Figure 1a–c exhibited evidence that $g^{\mathrm{XNN}}$ learned about the signal-generating function in Equation (1) and the displayed information should help practitioners understand which original input $X_j$ features were weighed heavily in each $n_k$ subnetwork, and which $n_k$ subnetworks have a strong influence on $g^{\mathrm{XNN}}\left(\mathbf{X}\right)$ output. See Appendix B.3 for additional details regarding general XNN architecture.

3.2. Mortgage Data Results

Results for the mortgage data are presented in Section 3.2.1, Section 3.2.2 and Section 3.2.3 to showcase the example workflow. $g^{\mathrm{ANN}}$ and $g^{\mathrm{XNN}}$ outperformed $g^{\mathrm{GBM}}$ and $g^{\mathrm{MGBM}}$ on the mortgage data, but as in Section 3.1.1, the constrained variants of both model architectures did not show large differences in model fit with respect to unconstrained variants. Assuming that in high-stakes applications small fit differences on static test data did not outweigh the need for enhanced model debugging facilitated by high interpretability, only $g^{\mathrm{MGBM}}$ and $g^{\mathrm{XNN}}$ interpretability, post-hoc explainability, and discrimination testing results are presented.

3.2.1. Constrained vs. Unconstrained Model Fit Assessment

Table 2. Fit measures for $g^{\mathrm{GBM}}\left(\mathbf{X}\right)$, $g^{\mathrm{MGBM}}\left(\mathbf{X}\right)$, $g^{\mathrm{ANN}}\left(\mathbf{X}\right)$, and $g^{\mathrm{XNN}}\left(\mathbf{X}\right)$ on the mortgage test data. Arrows indicate the direction of improvement for each measure and the best result in each column is displayed in bold font.

Model	Accuracy ↑	AUC ↑	F1 ↑	Logloss ↓	MCC ↑	RMSE ↓	Sensitivity ↑	Specificity ↑
$g^{\mathrm{GBM}}$	0.795	0.828	0.376	0.252	0.314	0.276	0.634	0.813
$g^{\mathrm{MGBM}}$	0.765	0.814	0.362	0.259	0.305	0.278	0.684	0.773
$g^{\mathrm{ANN}}$	0.865	0.871	0.474	0.231	0.418	0.262	0.624	0.891
$g^{\mathrm{XNN}}$	0.869	0.868	0.468	0.233	0.409	0.263	0.594	0.898

shows that $g^{\mathrm{ANN}}$ and $g^{\mathrm{XNN}}$ noticeably outperformed $g^{\mathrm{GBM}}$ and $g^{\mathrm{MGBM}}$ on the mortgage data for most of the fit measures. This is at least partially due to the preprocessing required to present directly comparable post-hoc explainability results and to use neural networks and TensorFlow, e.g., numerical encoding of categorical features and missing values. This preprocessing appears to hamstring some of the tree-based models’ inherent capabilities. $g^{\mathrm{GBM}}$ models trained on non-encoded data with missing values repeatedly produced receiver operating characteristic area under the curve (AUC) values of ∼0.81 (not shown, but available in resources discussed in Section 2.8).

Regardless of the fit differences between the two families of models, the difference between the constrained and unconstrained variants within the two types of models is small for the GBMs and smaller for the ANNs, ∼3.5% and ∼1% worse fit respectively, averaged across the measures in Table 2.

3.2.2. Interpretability and Post-hoc Explanation Results

For $g^{\mathrm{MGBM}}\left(\mathbf{X}\right)$, intrinsic interpretability was evaluated with PD and ICE plots of mostly monotonic prediction behavior for several important $X_j$, and post-hoc Shapley explanation analysis was used to create global and local feature importance. Global Shapley feature importance for $g^{\mathrm{MGBM}}\left(\mathbf{X}\right)$ on the mortgage test data is reported in Figure 2. $g^{\mathrm{MGBM}}$ placed high importance on LTV ratio, perhaps too high, and also weighed DTI ratio, property value, loan amount, and introductory rate period heavily in many of its predictions. Tree SHAP values are reported in the margin space, prior to the application of the logit link function, and the reported numeric values can be interpreted as the mean absolute impact of each $X_j$ on $g^{\mathrm{MGBM}}\left(\mathbf{X}\right)$ in the mortgage test data in the $g^{\mathrm{MGBM}}\left(\mathbf{X}\right)$ margin space. The potential over-emphasis of LTV ratio, and the de-emphasis of income, likely an important feature from a business perspective, and the de-emphasis of the encoded no introductory rate period flag feature may also contribute to the decreased performance of $g^{\mathrm{MGBM}}\left(\mathbf{X}\right)$ as compared to $g^{\mathrm{XNN}}\left(\mathbf{X}\right)$.

Domain knowledge was used to positively constrain DTI ratio and LTV ratio and to negatively constrain income and the loan term flag under $g^{\mathrm{MGBM}}$. The monotonicity constraints for DTI ratio and LTV ratio were confirmed for $g^{\mathrm{MGBM}}\left(\mathbf{X}\right)$ on the mortgage test data in Figure 3. Both DTI ratio and LTV ratio displayed positive monotonic behavior at all selected percentiles of $g^{\mathrm{MGBM}}\left(\mathbf{X}\right)$ for ICE and on average with PD. Because PD curves generally followed the patterns of the ICE curves for both features, it is also likely that no strong interactions were at play for DTI ratio and LTV ratio under $g^{\mathrm{MGBM}}$. Of course, the monotonicity constraints themselves could have dampened the effects of non-monotonic interactions under $g^{\mathrm{MGBM}}$, even if they did exist in the training data (e.g., LTV ratio and the no introductory rate period flag, see Figure 6). This rigidity could also have played a role in the performance differences between $g^{\mathrm{MGBM}}\left(\mathbf{X}\right)$ and $g^{\mathrm{XNN}}\left(\mathbf{X}\right)$ in the mortgage data not observed for the simulated data, wherein strong interactions appeared to be between features with the same monotonicity constraints (e.g., $X_{\mathrm{Friedman},1}$ and $X_{\mathrm{Friedman},2}$, see Figure 1).

PD and ICE are displayed with a histogram to highlight any sparse regions in an input feature’s domain. Because most ML models will always issue a prediction on any instance with a correct schema, it is crucial to consider whether a given model learned enough about an instance to make an accurate prediction. Viewing PD and ICE along with a histogram is a convenient method to visually assess whether a prediction is reasonable and based on sufficient training data. The DTI ratio and LTV ratio do appear to have had sparse regions in their univariate distributions. The monotonicity constraints likely play to the advantage of $g^{\mathrm{MGBM}}$ in this regard, as $g^{\mathrm{MGBM}}\left(\mathbf{X}\right)$ appears to carry reasonable predictions learned from dense domains into the sparse domains of both features.

Figure 3 also displays PD and ICE for the unconstrained feature property value. Unlike the DTI ratio and LTV ratio, PD for property value did not always follow the patterns established by ICE curves. While PD showed monotonically increasing prediction behavior on average, apparently influenced by large predictions at extreme $g^{\mathrm{MGBM}}\left(\mathbf{X}\right)$ percentiles, ICE curves for individuals at the 40th percentile of $g^{\mathrm{MGBM}}\left(\mathbf{X}\right)$ and lower exhibited different prediction behavior with respect to property value. Some individuals at these lower percentiles displayed monotonically decreasing prediction behavior, while others appeared to show fluctuating prediction behavior. Property value was strongly right-skewed, with little data regarding high-value property from which $g^{\mathrm{MGBM}}$ can learn. For the most part, reasonable predictions did appear to be carried from more densely populated regions to more sparsely populated regions. However, prediction fluctuations at lower $g^{\mathrm{MGBM}}\left(\mathbf{X}\right)$ percentiles were visible, and appeared in a sparse region of property value. This divergence of PD and ICE could be indicative of an interaction affecting property value under $g^{\mathrm{MGBM}}$ [23], and analysis by surrogate decision tree did show evidence of numerous potential interactions in lower predictions ranges of $g^{\mathrm{MGBM}}\left(\mathbf{X}\right)$ [32] (not shown, but available in resources discussed in Section 2.8). Fluctuations in ICE could also have been caused by overfitting or by leakage of strong non-monotonic signal from important constrained features into the modeled behavior of non-constrained features.

In Figure 4, local Tree SHAP values are displayed for selected individuals at the 10th, 50th, and 90th percentiles of $g^{\mathrm{MGBM}}\left(\mathbf{X}\right)$ in the mortgage test data. Each Shapley value in Figure 4 represents the difference in $g^{\mathrm{MGBM}}\left(\mathbf{x}\right)$ and the average of $g^{\mathrm{MGBM}}\left(\mathbf{X}\right)$ associated with this instance of some input feature $x_j$ [33]. Accordingly, the logit of the sum of the Shapley values and the average of $g^{\mathrm{MGBM}}\left(\mathbf{X}\right)$ is $g^{\mathrm{MGBM}}\left(\mathbf{x}\right)$, the prediction in the probability space for any $\mathbf{x}$.

The selected individuals showed an expected progression of mostly negative Shapley values at the 10th percentile, a mixture of positive and negative Shapley values at the 50th percentile, mostly positive Shapley values at the 90th percentile, and with globally important features driving most local model decisions. Deeper significance for Figure 4 lies in the ability of Tree SHAP to accurately and consistently summarize any single $g^{\mathrm{MGBM}}\left(\mathbf{x}\right)$ prediction in this manner, which is generally important for enabling logical appeal or override of ML-based decisions, and is specifically important in the context of lending, where applicable regulations often require lenders to provide consumer-specific reasons for denying credit to an individual. In the US, applicable regulations are typically ECOA and FCRA, and the consumer-specific reasons are commonly known as adverse actions codes.

Figure 5 displays global feature importance for $g^{\mathrm{XNN}}\left(\mathbf{X}\right)$ on the mortgage test data. Deep SHAP values are reported in the probability space, after the application of the logit link function. They are also calculated from the projection layer of $g^{\mathrm{XNN}}$. Thus, the Deep SHAP values in Figure 5 are the estimated average absolute impact of each input, $X_j$, in the projection layer and probability space of $g^{\mathrm{XNN}}\left(\mathbf{X}\right)$ for the mortgage test data. $g^{\mathrm{XNN}}$ distributes importance more evenly across business drivers and puts stronger emphasis on the no introductory rate period flag feature than does $g^{\mathrm{MGBM}}$. Like $g^{\mathrm{MGBM}}$, $g^{\mathrm{XNN}}$ puts little emphasis on the other flag features. Unlike $g^{\mathrm{MGBM}}$, $g^{\mathrm{XNN}}$ assigned higher importance to property value, loan amount, and income, and lower importance on LTV ratio and DTI ratio.

The capability of $g^{\mathrm{XNN}}$ to model nonlinear phenomenon and high-degree interactions, and to do so in an interpretable manner, is on display in Figure 6. Figure 6a presents the sparse ${\gamma }_k$ weights of the $g^{\mathrm{XNN}}$ output layer in which the $n_k$ subnetworks with $k\in \{0,1,2,3,5,8,9\}$ had large magnitude weights and $n_k$ subnetworks, $k\in \{4,6,7\}$, had small or near-zero magnitude weights. Distinctive ridge functions that fed into those large magnitude ${\gamma }_k$ weights are highlighted in Figure 6b and color-coded to pair with their corresponding ${\gamma }_k$ weight. As in the Section 3.1.2, $n_k$ ridge function plots varied with the output of the corresponding projection layer ${\sum }_j{\beta }_{k,j}{x}_j$ hidden unit, with weights displayed in matching colors in Figure 6c. In both the simulated and mortgage data, $n_k$ ridge functions appeared to be elementary functional forms that the output layer learned to combine to generate accurate predictions, reminiscent of basis functions for the modeled space. Figure 6c displays the sparse ${\beta }_{j,k}$ weights of the projection layer ${\sum }_j{\beta }_{k,j}{x}_j$ hidden units that were associated with each $n_k$ subnetwork ridge function. For instance, subnetwork $n_3$ was influenced by large weights for LTV ratio, no introductory rate period flag, and introductory rate period, whereas subnetwork $n_9$ was nearly completely dominated by the weight for income. See Appendix B.3 for details regarding general XNN architecture.

To complement the global interpretability of $g^{\mathrm{XNN}}$, Figure 7 displays local Shapley values for selected individuals, estimated from the projection layer using Deep SHAP in the $g^{\mathrm{XNN}}$ probability space. Similar to Tree SHAP, local Deep SHAP values should sum to $g^{\mathrm{XNN}}\left(\mathbf{x}\right)$. While the Shapley values appeared to follow the roughly increasing pattern established in Figure A4, Figure A6 and Figure 4, their true value was their ability to be calculated for any $g^{\mathrm{XNN}}\left(\mathbf{x}\right)$ prediction, as a means to summarize model reasoning and allow for appeal and override of specific ML-based decisions.

3.2.3. Discrimination Testing Results

Table 3a,b show the results of the discrimination tests using the mortgage data for two sets of class-control groups: blacks as compared to whites, and females as compared to males. As with the simulated data in Table A1, several measures of disparities are shown, with the SMDs calculated using the probabilities from $g^{\mathrm{MGBM}}\left(\mathbf{X}\right)$ and $g^{\mathrm{XNN}}\left(\mathbf{X}\right)$, and the accuracy, FPRs, and FPR ratios, MEs, and AIRs calculated using a binary outcome based on a cutoff of 0.20 (anyone with probabilities of 0.2 or less receives the favorable outcome; see Appendix F for comments pertaining to discrimination testing and cutoff selection). Since $g^{\mathrm{MGBM}}$ and $g^{\mathrm{XNN}}$ were predicting the likelihood of receiving a high-priced loan, $g^{\mathrm{MGBM}}$ and $g^{\mathrm{XNN}}$ assume that a lower score was favorable. Thus, one might consider FPR ratios as a measure of the class-control disparities. FPR ratios were higher under $g^{\mathrm{XNN}}$ than $g^{\mathrm{MGBM}}$ (2.45 vs. 2.10) in Table 3b, but overall FPRs were lower for blacks under $g^{\mathrm{XNN}}$ (0.295 vs. 0.315) in Table 3a. This is the same pattern seen in the simulated data results in Appendix E.2, leading to the question of whether a fairness goal should not only consider class-control relative rates, but also intra-class improvements in the chosen fairness measure. Similar results were found for the female-male comparison, but the relative rates are less stark: 1.15 for $g^{\mathrm{MGBM}}\left(\mathbf{X}\right)$ and 1.21 for $g^{\mathrm{XNN}}\left(\mathbf{X}\right)$.

Both ME and AIR showed higher disparities for blacks under $g^{\mathrm{XNN}}$ than $g^{\mathrm{MGBM}}$. Blacks receive high-priced loans 21.4% more frequently using $g^{\mathrm{XNN}}$ vs. 18.3% for $g^{\mathrm{MGBM}}$. Both $g^{\mathrm{MGBM}}$ and $g^{\mathrm{XNN}}$ showed AIRs that were statistically significantly below parity (not shown, but available in resources discussed in Section 2.8), and which were also below the EEOC’s 0.80 threshold. This would typically indicate need for further review to determine the cause and validity of these disparities, and a few relevant remediation techniques for such discovered discrimination are discussed in Section 4.3. On the other hand, women improved under $g^{\mathrm{XNN}}$ vs. $g^{\mathrm{MGBM}}$ (MEs of 3.6% vs. 4.1%; AIRs of 0.955 vs. 0.948). The AIRs, while statistically significantly below parity, were well above the EEOC’s threshold of 0.80. In most situations, the values of these measures alone would not likely flag a model for further review. Black SMDs for $g^{\mathrm{XNN}}\left(\mathbf{X}\right)$ and $g^{\mathrm{MGBM}}\left(\mathbf{X}\right)$ were similar: 0.621 and 0.628, respectively. These exceeded Cohen’s guidelines of 0.5 for a medium effect size and would likely trigger further review. Female SMDs were well below Cohen’s definition of small effect size: 0.105 and 0.084 for $g^{\mathrm{XNN}}\left(\mathbf{X}\right)$ and $g^{\mathrm{MGBM}}\left(\mathbf{X}\right)$, respectively. Similar to results for female AIR, these values alone are unlikely to prompt further review.

4. Discussion

4.1. The Burgeoning Python Ecosystem for Responsible Machine Learning

Figure 8 displays a holistic approach to ML model training, assessment, and deployment meant to decrease discrimination, inaccuracy, and privacy and security risks for high-stakes, human-centered, or regulated ML applications. (See Toward Responsible Machine Learning for details regarding Figure 8.) While all the methods mentioned in Figure 8 play an important role in increasing human trust and understanding of ML, a few pertinent references and Python resources are highlighted below as further reading to augment this this text’s focus on certain interpretable models, post-hoc explanation, and discrimination testing techniques.

Any discussion of interpretable ML models would be incomplete without references to the seminal work of the Rudin group at Duke University and EBMs, or GA²Ms, pioneered by researchers at Microsoft and Cornell [29,34,35]. In keeping with a major theme of this manuscript, models from these leading researchers and several other kinds of interpretable ML models are now available as open source Python packages. Among several types of currently available interpretable models, practitioners can now use Python to evaluate EBM in the interpret package, optimal sparse decision trees, GAMs in the pyGAM package, a variant of Friedman’s RuleFit in the skope-rules package, monotonic calibrated interpolated lookup tables in tensorflow/lattice, and this looks like that interpretable deep learning [34,35,36,37] (see Optimal Sparse Decision Trees, ProtoPNet (this looks like that)). Additional, relevant references and Python functionality include:

• Exploratory data analysis (EDA): H2OAggregatorEstimator in h2o [38].
• Sparse feature extraction: H2OGeneralizedLowRankEstimator in h2o [39].
• Preprocessing and models for privacy: diffprivlib, tensorflow/privacy [40,41,42,43].
• Causal inference and probabilistic programming: dowhy, PyMC3 [44].
• Post-hoc explanation: structured data explanations with alibi and PDPbox, image classification explanations with DeepExplain, and natural language explanations with allennlp [45,46,47].
• Discrimination testing: aequitas, Themis.
• Discrimination remediation: Reweighing, adversarial de-biasing, learning fair representations, and reject option classification with AIF360 [48,49,50,51].
• Model debugging: foolbox, SALib, tensorflow/cleverhans, and tensorflow/model- analysis [52,53,54,55].
• Model documentation: models cards [56], e.g., GPT-2 model card, Object Detection model card.

See Awesome Machine Learning Interpretability for a longer, community-curated metalist of related software packages and resources.

4.2. Appeal and Override of Automated Decisions

Interpretable models and post-hoc explanations can play an important role in increasing transparency into model mechanisms and predictions. As seen in Section 3, interpretable models often enable users to enforce domain knowledge-based constraints on model behavior, to ensure that models obey reasonable expectations, and to gain data-derived insights into the modeled problem domain. Post-hoc explanations generally help describe and summarize mechanisms and decisions, potentially yielding an even clearer understanding of ML models. Together they can allow for human learning from ML, certain types of regulatory compliance, and crucially, human appeal or override of automated model decisions [32]. Interpretable models and post-hoc explanations are likely good candidates for ML uses cases under the FCRA, ECOA, GDPR and other regulations that may require explanations of model decisions, and they are already used in the financial services industry today for model validation and other purposes. (For examples uses in financial services, see Deep Insights into Explainability and Interpretability of Machine Learning Algorithms and Applications to Risk Management. Also note that many non-consistent explanation methods can result in drastically different global and local feature importance values across different models trained on the same data or even for refreshing a model with augmented training data [33]. Consistency and accuracy guarantees are perhaps a factor in the growing momentum behind Shapley values as a candidate technique for generating consumer-specific adverse action notices for explaining and appealing automated ML-based decisions in highly-regulated settings, such as credit lending [57].) In general, transparency in ML also facilitates additional responsible AI processes such as model debugging, model documentation, and logical appeal and override processes, some of which may also be required by applicable regulations (e.g., US Federal Reserve Bank SR 11-7: Guidance on Model Risk Management). Among these, providing persons affected by a model with the opportunity to appeal ML-based decisions may deserve the most attention. ML models are often wrong (“All models are wrong, but some are useful.”—George Box, Statistician (1919–2013)) and appealing black-box decisions can be difficult (e.g., When a Computer Program Keeps You in Jail). For high-stakes, human-centered, or regulated applications that are trusted with mission- or life-critical decisions, the ability to logically appeal or override inevitable wrong decisions is not only a possible prerequisite for compliance, but also a failsafe procedure for those affected by ML decisions.

4.3. Discrimination Testing and Remediation in Practice

A significant body of research has emerged around exploring and fixing ML discrimination [58]. Methods can be broadly placed into two groups: more traditional methods that mitigate discrimination by searching across possible algorithmic and feature specifications, and many approaches that have been developed in the last 5–7 years that alter the training algorithm, preprocess training data, or post-process predictions in order to diminish class-control correlations or dependencies. Whether these more recent methods are suitable for a particular use case depends on the legal environment where a model is deployed and on the use case itself. For comments on why some recent techniques could result in regulatory non-compliance in certain scenarios, see Appendix G.

Of the newer class of fairness enhancing interventions, within-algorithm discrimination mitigation techniques that do not use class information may be more likely to be acceptable in highly regulated settings today. These techniques often incorporate a loss function where more discriminatory paths or weights are penalized and only used by the model if improvements in fit overcome some penalty. (The relative level of fit-to-discrimination penalty is usually determined via hyperparameter.) Other mitigation strategies that only alter hyperparameters or algorithm choice are also likely to be acceptable. Traditional feature selection techniques (e.g., those used in linear models and decision trees) are also likely to continue to be accepted in regulatory environments. For further discussion of techniques that can mitigate DI in US financial services, see Schmidt and Stephens [59].

Regardless of the methodology chosen to minimize disparities, advances in computing have enhanced the ability to search for less discriminatory models. Prior to these advances, only a small number of alternative algorithms could be tested for lower levels of disparity without causing infeasible delays in model implementation. Now, large numbers of models can be quickly tested for lower discrimination and better predictive quality. An additional opportunity arises as a result of ML itself: the well-known Rashomon effect, or the multiplicity of good ML models for most datasets. It is now feasible to train more models, find more good models, and test more models for discrimination, and among all those tested models, there are likely to be some with high predictive performance and low discrimination.

4.4. Intersectional and Non-Static Risks in Machine Learning

The often black-box nature of ML, the perpetuation or exacerbation of discrimination by ML, or the privacy harms and security vulnerabilities inherent in ML are each serious and difficult problems on their own. However, evidence is mounting that these harms can also manifest as complex intersectional challenges, e.g., the fairwashing or scaffolding of biased models with ML explanations, the privacy harms of ML explanations, or the adversarial poisoning of ML models to become discriminatory [8,19,20] (e.g., Tay, Microsoft’s AI chatbot, gets a crash course in racism from Twitter). (While the focus of this paper is not ML security, proposed best-practices from that field do point to transparency of ML systems as a mitigating factor for some ML attacks and hacks [55]. High system complexity is sometimes considered a mitigating influence as well [60]. This is sometimes known as the transparency paradox in data privacy and security, and it likely applies to ML security as well, especially in the context of interpretable ML models and post-hoc explanation (see The AI Transparency Paradox.)) Practitioners should of course consider the discussed interpretable modeling, post-hoc explanation, and discrimination testing approaches as at least partial remedies to the black-box and discrimination issues in ML. However, they should also consider that explanations can ease model stealing, data extraction, and membership inference attacks, and that explanations can mask ML discrimination. Additionally, high-stakes, human-centered, or regulated ML systems should generally be built and tested with robustness to adversarial attacks as a primary design consideration, and specifically to prevent ML models from being poisoned or otherwise altered to become discriminatory. Accuracy, discrimination, and security characteristics of a system can change over time as well. Simply testing for these problems at training time, as presented in Section 3, is not adequate for high-stakes, human-centered, or regulated ML systems. Accuracy, discrimination, and security should be monitored in real-time and over time, as long as a model is deployed.

5. Conclusions

This text puts forward results on simulated data to provide some validation of constrained ML models, post-hoc explanation techniques, and discrimination testing methods. These same modeling, explanation, and discrimination testing approaches are then applied to more realistic mortgage data to provide an example of a responsible ML workflow for high-stakes, human-centered, or regulated ML applications. The discussed methodologies are solid steps toward interpretability, explanation, and minimal discrimination for ML decisions, which should ultimately enable increased fairness and logical appeal processes for ML decision subjects. Of course, there is more to the responsible practice of ML than interpretable models, post-hoc explanation, and discrimination testing, even from a technology perspective, and Section 4 also points out numerous additional references and open source Python software assets that are available to researchers and practitioners today to increase human trust and understanding in ML systems. While the complex and messy problems of racism, sexism, privacy violations, and cyber crime can probably never be solved by technology alone, this work and many others illustrate numerous ways for ML practitioners to mitigate such risks.