An Empirical Survey on Explainable AI Technologies: Recent Trends, Use-Cases, and Categories from Technical and Application Perspectives

Abstract

In a wide range of industries and academic fields, artificial intelligence is becoming increasingly prevalent. AI models are taking on more crucial decision-making tasks as they grow in popularity and performance. Although AI models, particularly machine learning models, are successful in research, they have numerous limitations and drawbacks in practice. Furthermore, due to the lack of transparency behind their behavior, users need more understanding of how these models make specific decisions, especially in complex state-of-the-art machine learning algorithms. Complex machine learning systems utilize less transparent algorithms, thereby exacerbating the problem. This survey analyzes the significance and evolution of explainable AI (XAI) research across various domains and applications. Throughout this study, a rich repository of explainability classifications and summaries has been developed, along with their applications and practical use cases. We believe this study will make it easier for researchers to understand all explainability methods and access their applications simultaneously.

1. Introduction

In the last decade, eXplainable Artificial Intelligence (XAI) has seen a significant increase in articles, conferences, and symposia [1,2]. Artificial Intelligence (AI)-based algorithms, particularly deep neural networks, are changing how humans approach real-world tasks. Machine Learning (ML) algorithms have in recent years increased in automating various aspects of science, business, and social workflow. It is partly due to a rise in research in a branch of ML known as Deep Learning (DL), where thousands (even billions) of neuronal parameters are trained to generalize how to perform a specific task. As a result, a plethora of domain-dependent and context-specific methods for interpreting ML models and forming explanations for humans have emerged. The importance of comprehending trust [3], ethics [4], bias [5], and the effect of adversarial examples [6] in deceiving AI classifier decisions is highlighted by the recent interest in XAI.

According to Miller et al., one of the main motives for seeking justifications for particular choices is curiosity [7]. Better learning facilitation might be another factor that would strengthen the model’s design and yield better outcomes. Over time, each explanation should result in consistent or comparable explanations for the same data point and consistent or comparable explanations for related data points [8]. Explanations should make the AI algorithm expressive to increase human comprehension and confidence in decision-making and support impartial and just decisions. As a result, ML systems must provide an explanation or workable solution to maintain transparency, trust, and fairness in decision-making.

An explainable AI (XAI) system aims to make its behavior more understandable to humans by providing explanations. The terms interpretability and explainability are often used interchangeably, but they have different meanings [9]. Explainability is more concerned with explaining the inner mechanism of the system (black-box model) or presenting the solution in human-understandable terms. However, interpretability concerns finding causality between input and output or understanding the cause of a decision.

A few general principles can be followed when developing more human-friendly AI systems: XAI technology should explain its capabilities and features to improve its usability [10]. Moreover, it explains what it has done, what it is doing now, and what it will do next, and it identifies the critical information on which it relies. Despite this, every explanation is contextualized according to the user’s task, abilities, and expectations [11]. Interpretability and explainability are, therefore, domain-dependent and cannot be defined independently. Full or partial explanations are acceptable. Fully interpretable models provide complete and transparent explanations. Partially interpretable models reveal essential aspects of their reasoning process. Certain “interpretability constraints” are associated with interpretable models, such as monotonicity for some variables and correlation between them. Variable importance measures, local models that approximate global models at specific points and saliency maps are examples of partial explanations.

There are still many open issues, and difficulties at the nexus of machine learning and explanation [12]. They consist of, but are not restricted to:

• Contrasting interpretability and accuracy: In XAI research on explanation, the methods and limitations of explainability are explored as a major theme. Accuracy, explainability, and tractability must be balanced based on accuracy and fidelity trade-offs.
• Describing abilities as opposed to describing decisions: High-level expertise is demonstrated by the capacity to analyze novel situations. It is crucial to aid end users in understanding an AI system’s capabilities. They must learn how to gauge a particular AI system’s capabilities and whether it has any blind spots or cannot solve certain types of problems.
• The use of abstractions to clarify explanations: Large plans are described in large steps using high-level patterns. The discovery and sharing of abstractions in learning and explanation have long been challenging, and today’s cutting-edge XAI research focuses on what makes abstractions helpful in learning and explanation [11].

From another perspective, machine learning is becoming increasingly important in modern life, from entertainment to various industries. Many businesses are eager to use machine learning models because they are faster and more accurate than legacy systems. Furthermore, it will be difficult to ignore that machine learning models will eventually take over human decision-making tasks in various fields; however, a lack of transparency about their decision-making procedures, inner mechanisms, or unexpected behaviors may frustrate and confuse users. It may also negatively impact human life, particularly in critical decision-making domains such as financial loans, insurance quotes, and cancer detection in health care. It could negatively impact other applications such as face recognition [13], game theory-based predictive analytics [14,15], recidivism risk assessment, and self-driving cars.

From the standpoint of human-centered research, research on competencies and knowledge could take XAI beyond explaining a specific system and assisting its users in determining appropriate trust. In the future, XAI may be significant in many social roles. These roles include learning and demonstrating knowledge to individuals, working with other agents to connect the knowledge, developing cross-disciplinary insights and common ground, collaborating with others to teach people, and drawing on existing knowledge to advance knowledge discovery and application. The future of XAI is just beginning from such a social perspective of knowledge understanding and generation. This trend is regrettable because there is a plethora of information in the sector that needs to be arranged. This article aims to conduct a systematic review of research in the field of XAI and define field boundaries. Additionally, it presents empirical examples and a selection of fundamental, commonly used XAI methods.

The contributions of this research are four-fold:

• We demonstrated the significance and utility of XAI research by providing examples of how it positively shapes several fields of research, including AI;
• We provided various examples using open-source data sets to compare various XAI techniques and tools;
• We demonstrated how XAI can be integrated into an AI pipeline at every stage;
• Finally, we compiled a rich collection of XAI techniques, categories, and applications inspired by well-known XAI researchers.

In Section 2, we explore recent XAI research through bibliometrics and network analysis. Section 3 provides an overview of the primary categories of XAI methods. Section 4 describes the different XAI methods that belong to the broad category Self-Explainable Modeling including intrinsic explainability, ad-hoc explainability, and attention mechanism. Section 5 explains the different post-hoc XAI methods, and provides various examples from different XAI methods for numeric, textual, and image data types. The advancements in research areas such as language modeling, human-AI partnerships, and sequential analysis of big data inspired us to focus on XAI applications in Section 6. The article is concluded with some remarks on the future direction of XAI in Section 7.

2. Bibliometric and Statistical Analysis of XAI Research

According to De Bellis [16], bibliometric is a scientometric discipline that studies the quantitative features and characteristics of scientific research, such as the number of citations, authors information (name, affiliation, city, country), title, abstract, and keywords of an article. This scientific journal published that article, references, and so on. Statistical, mathematical, and visualization techniques are typically used to treat bibliographic data in the form of tables and graphs. For example, Vargas-Quesada and Moya-Anegon [17] proposed a methodology for developing visual representations of scientific domains. Citations and co-citations are used to demonstrate interactions between authors and papers. For the following analysis, we used bibliometric data from Scopus, which includes many different sources such as IEEE, ACM, etc. The query used to retrieve data from the previously mentioned sources contains all types of methods, related tasks, and synonyms and can be found in Appendix A. The queries performed on 24 August 2022, and 48,060 conference and journal articles are listed in the above-mentioned database; 8180 articles are related to Computer Vision and 6204 articles belong to NLP.

Figure 1 demonstrated that the number of publications began to rise in the early 2000s; however, the field of explainability has grown exponentially since 2017, and the importance of explainability in the future of artificial intelligence and related fields is reflected in this exponential growth. Further, many organizations, including the ACM [18] and the European General Data Protection Regulation (EGDPR) [19], have issued statements about algorithmic transparency and regulations that encourage systems and institutions to take explainability into account when designing and using algorithmic decision-making tools. As a result of these points, it is indisputable that explainability must increase.

The XAI field is as vast as artificial intelligence, and explainability is possible whenever we have algorithmic AI. As a result, this broadness makes it challenging to investigate all aspects of explainability. We run two queries to determine the proportion of XAI papers related to Natural Language Processing and Computer Vision. Appendix A contains two queries that we used to search the Scopus databases for research on the intersection of XAI and computer vision or natural language processing. This method categorizes 48,060 articles into 3 categories: 6204 articles in NLP, 8180 articles in CV, and the rest related to other computer science and engineering.

Additionally, there are several learning tasks in XAI-related disciplines, including speech recognition, named entity recognition, and image reconstruction. The most popular tasks in these domains are also analyzed to determine which tasks draw the most attention. Figure 2 depicts the top 25 learning tasks with the highest frequency of publications in the domain of XAI in NLP, Computer Vision, and other related fields. The study reveals that, among various tasks, text classification and image classification receive the greatest attention from the natural language processing and computer vision fields, respectively.

It is also possible to analyze the distribution of publications across various research areas. Figure 3 demonstrates the distribution of publications in the top-10 ranking of research areas. It is clear that Computer Science, Engineering, and Mathematics have the highest number of publications among various subject areas.

Figure 4 is a collection of synonymous terms and their frequencies used in the XAI publications. This graph gives an overview of the main keywords authors use in their titles. $Interpretability$ is the main term that appeared in all the collected documents.

Finally, Figure 5 illustrates the distribution of algorithms used by researchers in their publications. The algorithms that received the most attention are deep learning, machine learning, convolutional neural networks, and deep neural networks.

Statistical Analysis on XAI Research

A decision tree, RuleFit model, linear model, or naive Bayes model are intrinsically interpretable XAI models that can be used for solving problems [20]. For example, in linear models, coefficients can be used to summarize the contribution of variables to the final prediction, where bigger coefficients are more important. Decision Trees are close to human logic because they use feature importance to select features; intuitively, the most important features are chosen first. The Naive Bayes classifier uses Bayes’ theorem of conditional probabilities [21]. For each feature, it calculates the probability for a class depending on the value of the feature. The Naive Bayes classifier uses conditional probabilities based on the Bayes theorem. For every feature, the probability is calculated for a class depending on the value of the feature. The Naive Bayes classifier calculates the class probabilities for each feature independently, and this independence assumption shows the contribution of each feature to the final prediction. The attention mechanism is undoubtedly one of the great revolutions in deep neural networks [22]. The attention mechanism helps the network take into account short and long dependencies by taking a weighted sum of hidden states as a context vector to overcome the memorization issue of neural networks [22].

The attention mechanism was introduced in the study by Bahdanau et al. [23] when they were working on neural machine translation. It was soon extended into computer vision, Natural Language Processing, and other fields. Various attention families have been created since the Bahdanau et al. paper in 2014.

Table 1. Different types of attention mechanisms applied in the publications under study.

Attention Type	Frequency
Attention Mechanism	221
Self-Attention Mechanism	19
Neural Attention Mechanism	6
Visual-Attention Mechanism	5
Co-Attention Mechanism	3
Hierarchical Attention Network	2
Inner-Attention Mechanism	2
Mutual Attention Mechanism	2
Interactive Attention Network	1
Compositional Attention Network	1
Channel Attention	1
Deep Attention Network	1
Gated Self-Attention	1
Sequence To Sequence Attention	1
Simple Neural Attention Meta-Learner	1
Intra-Attention Mechanism	1
Cross-Attention	1
Cross-Layer Attention	1
Sequence Generative Adversarial Network	1

illustrates the different attention types and their frequencies used by researchers in the explainability domain. While searching the articles, words, including attention, were erased to prevent the dominance of the attention mechanism in our work and help other explainability methods be seen.

The authors’ co-occurrence network can help us cluster the content and keywords of academic articles. Figure 6 demonstrates an undirected co-occurrence network with the popular authors’ keywords in the publications under study. Each node represents a keyword, and its size is proportional to the number of documents containing the keyword. We also find that “deep learning” is the main keyword since it is associated with the larger node. “Representation learning” and “attention mechanisms” are the second and third main keywords. Links between nodes are related to keywords usually appearing in the same documents. Through further analysis of these networks, we find that “deep learning” bridges “representation learning”, “attention mechanisms”, and “artificial intelligence.” In addition, we can see that machine learning algorithms are mainly located in orange nodes. The green nodes are representative of data mining and AI, which connect the levels between deep learning and machine learning.

3. Explainable AI Categories

Doshi and Kim [24] defined interpretability (explainability) in a machine learning context as “the ability to explain or present in understandable terms to a human”. Miller [7] defines interpretability as “the degree to which a human can understand the reasons for a decision”. XAI’s ultimate goal is to make a black-box model more transparent or use a transparent model to help end-users better understand model behaviors. For example, an end-user can be an expert in AI, and XAI must assist that user in comprehending the model structure and behavior in greater depth. The end-user can also be a person with no or limited knowledge of AI, and XAI needs to help the person better understand the model results.

XAI incorporates a vast group of techniques, and this broadness makes applying the right XAI tool to a specific problem difficult. Several categorizations in Molnar’s Interpretable Machine Learning book can provide insight into this issue [25]. His idea heavily influenced our XAI categories summary (see Figure 7), but we provided more examples and application-based definitions, resulting in a valuable repository of XAI resources.

3.1. Self-Explainable Modeling

XAI techniques consist of two major categories: self-explainable and post-hoc explainable modeling, depending on when explainability is demanded. Self-explainable modeling concerns constructing self-explanatory models at the beginning of training, e.g., building a decision tree or applying interpretability directly to the structure of the model, e.g., adding an attention layer to deep learning models.

3.2. Post-Hoc Explainable Modeling

In contrast, post-hoc explainable modeling requires creating a secondary model or a secondary technique to provide explanations for an existing model after training the black box model. The main difference between self-explainable, especially intrinsic and post-hoc methods, comes from the trade-off between fidelity and accuracy. An intrinsically explainable model could provide an accurate and undistorted explanation of the cost of sacrificing prediction performance [20]. For example, a company needs to predict the next month’s sales. They use a black-box model such as a Recurrent Neural Network (RNN) to fit the data. Suppose it predicts low sales for next month; the company would need a reasonable explanation for the low sales prediction of the model to decide on further actions. In addition, explaining an RNN model could be more intuitive and require an interpretable surrogate model. On the other hand, if we fitted an intrinsic method like a generalized linear model, the variables could be sorted by the coefficient weights of the model as feature importance, and it would be more precise which variables contributed more to next month’s low sales. There are two types of self-explainable methods intrinsic and ad-hoc explainability methods. Intrinsic explainability can be achieved by Linear, and Logistic Regression, Decision Trees, K-Nearest Neighbors, Rule-Based Learners, General Additive Models, and Bayesian Models as stated in the table given in Ref. [26]. The second category of self-explainable modeling belongs to ad-hoc explainability methods that employ the explainability methods during the training of the black-box models. On the other hand, We can see the use of post-hoc explainability tools in Ref. [27]. Here, the authors are using Logic Programming (LP) to justify the decision made by the model. Graphical visualizations, as well as other mathematical tools, have been used to explain the result. It can list global as well as local explanations for the output.

3.3. Global and Local Explainability

To enhance transparency in machine learning models, users can examine all data instances and understand how the model works globally by inspecting all data instances. The two ways to construct globally interpretable models are using all data to fit the model or using all instances of a few features. On the other hand, local interpretability examines a model’s individual prediction, figures out why the model makes its decision locally, and helps uncover the causal relations between a specific input and its corresponding model prediction. Global and local interpretability methods help users trust a model and prediction. The survey paper examines global and local interpretability techniques used in different models [28].

Before going further, we should mention that this survey has not included the pre-modeling explainability because the goal of this stage is to gain more useful insights from data and use them for model development. Moreover, this can be seen as a set approach from classical statistics for a better understanding of data rather than the model itself. Khaleghi [29] provides intriguing information regarding pre-modeling explainability methods. To add more information regarding pre-modeling explainability, the authors also provide a brief overview of feature selection by various mutual information techniques in Appendix B and dependency measure between features in Appendix C.

Explainable AI is a fast-growing area that applies various mathematical, statistical, and visual tools to discover explainability for a black-box algorithm. To help audiences have a holistic view of explainability methods, the authors constructed a hierarchical tree to relate various techniques in XAI’s literature. The succeeding tree and the following contents draw heavily from ideas in Refs. [20,25,30,31] (see Figure 2).

4. Self-Explainable Modeling Methods

Self-explainable modeling is based on two approaches: intrinsic explainability and ad-hoc explainability. In Figure 8, we show all the categories and subcategories under this classification.

4.1. Intrinsic Explainability

The first approach concerns the model’s explainability rather than its high learning performance. In this approach, an intrinsic and simple model is more desirable than complicated black-box models. For example, in social sciences and psychology, researchers would like to use Structural Equation Modeling (SEM), which can better capture the relationship between latent factors than a complex machine learning model with poor explainability. The Intrinsic explainability approach consists of three main categories as linear models (including generalized linear models, logistic regression, etc.), tree-based models (including Decision Tree, Rule-Fit, etc.), and finally, graphical models such as Bayesian Networks, Naive Bayes, cyclic directed models, etc.

4.1.1. Graphical Models

Currently, new techniques have been coming up in the field of XAI. A graphical explainability tool named gLIME was introduced in Ref. [32] in which the significant features were identified, and their direct and indirect impact on the model’s decision was depicted using graphical methods. An application is seen in Ref. [33], where graphical models are being applied in the healthcare sector. They have presented a neural network that provides visual interpretability for various biology-related datasets. Deep-learning-based algorithms have started being implemented in businesses where predictive models are used. Ref. [34] discussed using Bayesian Networks in predicting activities, whose results are explained using intrinsic explainability methods. The Naive Bayes classifier is considered an intrinsic interpretability model because it calculates the class probabilities for each feature independently, and this independence assumption can demonstrate each feature’s contribution to the final prediction. In other words, it is evident for each feature how much it contributes towards a particular class prediction since we can interpret the conditional probability [25].

4.1.2. Linear Models

The reason behind linear models’ interpretability is that they tend to learn a monotonic linear relationship between features and targets [35]. The linear model’s coefficients can be sorted by feature importance to summarize the whole model (except for when there is a strong interaction between the features).

4.1.3. Tree-Based Models

The tree-based models, such as decision trees, are also highly interpretable models because they are close to human logic and can learn the non-linear relationship between features and targets, even when features are interacting with each other. The decision trees provide feature importance based on impurity criteria such as entropy, or the Gini index [36]. Similar to decision trees, other tree-based ensemble models, such as Random Forest, can provide the feature importance ranking. A feature’s relative rank (depth) can be used to assess its importance as a decision node in a tree. During prediction, the top features contribute a greater fraction of the input samples to the final prediction. By analyzing the fraction of samples that decrease in impurity after splitting, one can estimate the relative importance of the features. The variance of such an estimate can be reduced by averaging predictions over several randomized trees. It is called the Mean Decrease in Impurity (MDI) [37]. It should note that the impurity-based feature importance computed on tree-based models suffers from two flaws that can lead to misleading conclusions. First, they are computed on statistics derived from the training dataset and do not necessarily inform us of which features are most important to make good predictions on the held-out dataset. Secondly, they favor high cardinality features, which are features with many unique values. Permutation feature importance is an alternative to impurity-based feature importance, which does not suffer from these flaws.

4.2. Ad-Hoc Explainability Methods

As mentioned previously, these models are intrinsically interpretable; however, they usually cannot provide high accuracy or minimum error for most practical, real-world problems. Thus the ad-hoc explainability approach has drawn much attention recently [31]. In ad-hoc explainability, the methods apply or add to a black-box model during the training. Researchers have proposed various ad-hoc explainability methods to enhance the transparency of the black box that can be categorized into adding interpretability constraints, explainable hybrid models, joint prediction and explanation, architectural adjustments, and information flow.

4.2.1. Adding Interpretability Constraints

Interpretability constraints can be seen in Zhang et al. [38] where the authors introduced constraints such as regularization terms or dropout in Convolutional Neural Network’s (CNN) convolution layers [38]. Another example of an ad-hoc explainability method can be seen in Ref. [26], where a CNN is used to identify skin cancer and detect plant disease. Here, feature selection and cleaning procedures are applied in the layers to provide a meaningful explanation of the model’s decision. A model’s explainability can be enhanced by using various regularization techniques. The authors of Ref. [39] explained how to train a multi-layer perceptron using a new penalty function called tree regularization. They also proved that this method is more explainable without compromising predictive performance. Saliency learning, another strategy used in Ref. [40], teaches the model where to focus its attention to provide explainability.

4.2.2. Hybrid Explainable Models

Combining an intrinsically explainable model with a black box method can result in an explainable hybrid model that is both high performing and explainable. Using K-nearest neighbor inference on the hidden representation of the training dataset, the deep K-nearest neighbors (DKNN) method [41] has been demonstrated to be effective and reliable in providing example-based explanations. However, it could be impractical for a huge dataset due to storing a hidden representation of the whole training dataset. In order to gain accuracy on a variety of forms of data while maintaining the model’s interpretability, deep, weighted averaging classifiers are another attempt that integrates nonparametric kernel regression and employs a weighted sum of training instances to predict its label [42]. Alvarez et al. [43] developed a self-explaining model by generalizing linear classifiers and utilizing the regularization principle. They argued that their suggested model better satisfies explicitness, fidelity, and stability—three principles for an explanation—than the earlier research. Similar to SENNs, Contextual Explanation Networks (CENs) also learn to predict by encoding the context into a probabilistic intermediate space that serves as an explanation, then feeding the input to a model [44]. BagNet is another effort to yield an explainable hybrid model in image processing tasks by utilizing bags of features learned from deep networks and classifying an image based on the occurrences of local image patches obtained by dividing the original image [45].

4.2.3. Joint Prediction and Explanation

Another XAI approach involves using a model to simultaneously make a prediction and provide the explanation that goes with it. To establish this explainable model, Teaching Explanations for Decisions (TED) [46] fed the model a decision and its explanation as a single label during the training phase and drove the model to make a decision with its associated explanation during the testing phase. Park et al. [47] combined visual and textual explanations to create a multimodal explanation and showed that visual explanations are more insightful. However, it should be noted that adding explanations during the training phase is not always possible, and the explanation employed may be subjective to what humans desire to see, making prediction inefficient. To overcome this limitation, Lei et al. [48] used a combined generator and encoder in which the generator provides a distribution across text fragments as potential rationales, which are then fed through the encoder for prediction.

4.2.4. Explainability through Architectural Adjustments

Some studies are looking deeper into the architecture of deep networks to increase model explainability. In order to create interpretable convolutional networks, Zhang et al. incorporated a specific loss function into the feature maps of the filters of conventional CNN [38]. This allowed the high Conv-layer filters to contain more semantically meaningful knowledge. In another attempt at an image classification task, Chen et al. [49] included a prototype layer between the convolutional layers and the fully connected layer to consider the prototypical aspects of one class or another.

4.2.5. Attention Mechanism

Visualizing the attention weight matrix helps users interpret which parts of the input are attended to for individual predictions. The attention mechanism was indeed revolutionary in neural machine translation (NMT), and it has spread to other parts of NLP and computer vision fields. We will be using the attention mechanism more frequently, so we will delve deeper into it by defining the background theory in the following.

Bahdanau et al. [23] proposed the attention mechanism in Neural Machine Translation (NMT) to address two disadvantages of the seq2seq model (NMT) [50] to remember long dependencies. It takes long dependencies into account for better memorization by taking a weighted sum of intermediate hidden states as a context vector. Various types of attention families have been born since the Bahdanau et al. [23] paper published in 2015, such as Global, Local, and Self Attention Mechanism. In Figure 4, we can see a general MTN that usually consists of two encoder and decoder networks, and a fixed-length context vector. Bahdanau et al. [23] used a bidirectional RNN encoder and an RNN decoder where the final hidden state of the RNN is trying to encode the entire sentence into a fixed-length context vector. Attention Mechanism [23] was proposed to address two disadvantages of the seq2seq model (NMT) [50] to remember long dependencies. The first disadvantage was using the fixed-length context vector, and the second one was NMT’s encoder, which only used the last hidden state of the LSTM and ignored the intermediate states of the encoder. Figure 9 demonstrates how long LSTM sequence fails to translate sentences to Chinese [51].

4.2.6. Attention Definition and Formula

The attention mechanism (see Figure 10) is a vector of importance weights used to predict the target element based on how strongly this element is correlated with other context elements. We can approximate the context vector [52] by taking their sum weighted by the attention vector.

The model by Bahdanau et al. [23] can be understood by using the following example, where $x=[x_1,x_2,\dots ,x_n]$ is a source sequence, e.g., English text, $y=[y_1,y_2,\dots ,y_m]$ is the target sequence, e.g., French translation, and

$$h_i={[{\overrightarrow{h_i}}^T;{\overleftarrow{h_i}}^T]}^T,i=1,\dots ,n$$

(1)

$h_i$ is the concatenation of the two hidden encoder states described in Equation (1). The encoder is a bidirectional RNN with a forward hidden state $\overrightarrow{h_i}$ and a backward hidden state $\overleftarrow{h_i}$. The decoder network has a hidden state $s_t=f(s_{t-1},y_{t-1},c_t)$ for the output word at position $t,t=1,\dots ,m$, where the context vector $c_t$ is composed of alignment scores weighted by hidden states in the input sequence, as seen in Equation (2) and the alignment scores are calculated using Equations (3) and (4) where $y_t$ is the context vector for the output.

$$c_t=\sum _{i=1}^n{\alpha }_{t,i}{\mathit{h}}_i$$

(2)

$${\alpha }_{t,i}=\mathrm{align}(y_t,x_i)=\frac{exp\left(\mathrm{score}({\mathit{s}}_{t-1},{\mathit{h}}_i)\right)}{{\sum }_{i^{\prime }=1}^{n}exp\left(\mathrm{score}({\mathit{s}}_{t-1},{\mathit{h}}_{i^{\prime }})\right)}$$

(3)

$$score(s_t,h_i)=v_a^{T}tanh\left(W_a[s_t;h_i]\right)$$

(4)

The alignment model assigns a score ${\alpha }_{t,i}$ to the pair of input at position i and output at position t, $(y_t,x_i)$, based on how well they match. The set of ${\alpha }_{t,i}$ are weights defining how much of each source hidden state should be considered for each output. In the alignment model, $v_a$ and $W_a$ are weight matrices learned in addition to the alignment score $\alpha$. Feed-forward networks are jointly trained with other parts of the model.

Since publishing the attention mechanism, researchers have applied it for various task. In addition, various type of attentions with distinct architectures have been built recently [22]. The attention mechanism not only helps the model to exclude the irrelevant information, and prevent information overload by putting more weight on limited relevant information, but also it helps to bring explainbility to neural networks models by visualizing the attention weights [53].

Locally intrinsic interpretable models are usually achieved by designing more justified model architectures that could explain why a specific decision is made locally. For instance, in neural machine translation (NMT) with local attention mechanism, the model only uses a local fixed number of intermediate hidden states instead of all (Figure 11). Moreover, users could understand how words locally in one language (English) depend on words in another language (French) for correct translation [23] (Figure 12).

Cheng et al. [55] used self-attention to relate different positions of a sequence in machine reading task by replacing the NMT’s target sequence with the same input (source) sequence. In other words, the self-attention mechanism enables a model to learn that current words are correlated with preceding words (see Figure 13). Moreover, the self-attention architecture helps the researcher to substitute the recurrent layers with multi-head self-attention ones for encoding the input sequence. It is a major move that helped not only accelerate the training by removing recurrent layers and parallelizing training but also enhance the learning performance by providing a wider range of context [56]. Galassi et al. [57] surveyed and proposed a taxonomy of attention types and models related to learning tasks, input representations, compatibility (score) functions, distribution functions, and multiplicity.

In the following, there are a few examples of works that pushed explainability into the model’s architecture. For example, Shah et al. [58] proposed a new model that integrates tree-based and CNN architectures for hyperspectral image classification. They use joint prediction and explanation by adding spatial attention for choosing appropriate salient features at each decision step, this enables explainability and efficiency in learning. In another example, Angle et al. [59] developed a method for explaining the Gradient Boosting Trees (GBT) more explainable. Although the ensemble trees are quite transparent, the result which comes at the end is an aggregate of the output of different trees. Hence, they have introduced a local intrinsic explainability tool that allows us to understand the contribution of each feature in the output.

5. Post-Hoc Explainable Modeling Methods

5.1. Global Post-Hoc Interpretability Methods

Examples provided in the Global and Local model-agnostic sections are all based on the California Housing dataset

Table 2. California housing dataset summary of continuous variables.

Features	Median_Income	House_Median_Age	Total_Rooms	Total_Bedrooms	Population	Household	Latitude	Longitude
count	20,640	20,640	20,640	20,640	20,640	20,640	20,640	20,640
mean	3.87	28.64	5.43	1.10	1425.48	3.07	35.63	−119.57
std	1.90	12.59	2.47	0.47	1132.46	10.39	2.14	2.00
min	0.50	1.00	0.85	0.33	3.00	0.69	32.54	−124.35
25%	2.56	18.00	4.44	1.01	787	2.43	33.93	−121.80
50%	3.53	29.00	5.23	1.05	1166	2.82	34.26	−118.49
75%	4.74	37.00	6.05	1.10	1725	3.28	37.71	−118.01
max	15.00	52.00	141.91	34.07	35682	1243.33	41.95	−114.31

in the sklearn package.

Due to the huge difference in the internal mechanism of traditional machine learning and highly black-box deep learning models (See Figure 14), we categorize Global post-hoc interpretability methods into two distinct branches: (1) Global Post-hoc Explanations for Traditional Machine Learning Models and (2) Global Post-hoc Explanations for Explaining internals of Deep Learning models (e.g., XAI representations of the neurons for intermediate layers in DL).

5.2. Traditional Machine Learning Models

Global Model-agnostic techniques cannot capture a particular feature’s actual effect on the black-box model’s prediction if there is a strong linear or non-linear correlation between features. So we need to explore the correlation of features and response variable before employing these techniques (for more details about correlation methods, see Appendix B).

Here, we describe six types of model-agnostic explanations for traditional machine learning models: (i) visualization, (ii) perturbation-based XAI, (iii) model summary, (iv) global surrogate models, (v) prototypes and criticism explanations, and (vi) influential instances.

(i) Visualization: Partial Dependence Plot (PDP)-PD plot [60] can show the marginal effect one or two features on the predicted outcome of a machine learning model where, a PDP can show whether the relationship between the target and a feature is linear, monotonic, or complicated. This is a global method because it does not focus on specific instances but an overall average (See Figure 15) For example, when applied to a linear regression model, PDPs always show a linear relationship.

$${\widehat{f}}_{x_s}\left(x_s\right)=E_{x_c}\left[\widehat{f}(x_s,x_c)\right]=\int \widehat{f}(x_s,x_c)dP\left(x_c\right)$$

(5)

$x_s$ is set of selected features, $x_c$ other features, and $\widehat{f}$ prediction function of machine learning model.

We trained and fine tuned a Random Forest Regressor on California Housing dataset and demonstrated the effect of a single variable, such as Longitude, on the model’s prediction using SKLearn Inspection package in Figure 15. Figure 16 demonstrates the effects of household and housing_median_age on the prediction model in 2D and 3D using SKLearn.

Individual Conditional Expectation (ICE) plot displays one line per instance that shows how the instance’s prediction changes when a feature changes. The equivalent to a PDP for individual data instances is called ICE plot [62]. A PDP is the average of the lines of an ICE plot, and the values for a line (and one instance) can be computed by keeping all other features the same, creating variants of this instance by replacing the feature’s value with values from a grid and making predictions with the black box model for these newly created instances. Therefore, the result is a set of points for an instance with the grid’s feature value and the respective predictions [25].

Accumulated Local Effects (ALE) Plot describes how features influence the prediction of a machine learning model on average, and it is an unbiased alternative to PDPs. PDP and ALE only differ in whether averages of predictions (PDP) or of differences in predictions (ALE) are calculated and whether averaging is done over the marginal (PDP) or conditional distribution (ALE). ALE plots average the changes in the predictions and accumulate them over the grid [63] Figure 17.

$$\begin{array}{cc}\hfill {\widehat{f}}_{x_S,ALE}\left(x_S\right)=& {\int }_{z_{0,1}}^{x_S}{E}_{X_C|X_S}\left[{\widehat{f}}^S(X_s,X_c)|X_S=z_S\right]d{z}_S-\mathrm{constant}\hfill \\ \hfill =& {\int }_{z_{0,1}}^{x_S}{\int }_{x_C}{\widehat{f}}^S(z_s,x_c)\mathbb{P}\left(x_C\right|z_S)d{x}_{C}d{z}_S-\mathrm{constant}\hfill \end{array}$$

(ii) Perturbation-based Explanations: Perturbation-based XAI technologies use two techniques named feature omission and feature occlusion. Perturbation-based methods examine the properties of machine learning models by perturbing the input features of a model, e.g., by occluding part of the input image with a mask or replacing a word in a sentence with its synonym and observing the changes in the output of the model [64]. The feature omission perturbation-based method explains a model’s prediction by comparing the output when all the features are known but one or more are omitted [65,66,67]. When features push a prediction toward a class, the contribution is positive; when they push a forecast against a class, they have no effect [65]. Through the feature occlusion perturbation-based method, subsets of the input are occluded, then forward propagated through the machine learning model to calculate the probability of the original input [65,68].

(iii) Model Summary: The main problem that hinders us from explaining a machine learning behavior in detail is the problem’s high dimensionality. Usually, many typical computer vision or NLP tasks consist of enormous feature sizes that make it difficult for us to analyze the effect of a specific feature (or a set of specific features) on the trained model. So it seems intuitive that reducing the number of variables to a manageable number is inevitable. Moreover, Feature Importance can help us rank features based on their weights on final prediction and filter least-important features. However, feature importance suffers from co-linearity between features. Breiman [69] introduced the permutation feature importance (PFI) concept on his famous Random Forest work for the first time. PFI measures the increase in the prediction error of the model after we permuted (shuffling) the feature’s values. Hence, it measures the model’s sensitivity when the feature’s relationship with the target is broken [70]. Figure 18 illustrates the PFI of each variable as well as the variance of PFI for trained RF model on the California Housing dataset using the SKLearn Inspection package [61].

Since the publication of PFI by Breiman [69], other researchers proposed various types of feature importance [71]. For example, Altmann et al. [72] proposed a heuristic method called permutation importance for normalizing feature importance based on repeated permutations test of the response vector in order to estimate a non-informative distribution of measured importance for each variable. P-values computed with permutation importance were the criteria used to show the significance of variables. The authors applied their method in Random Forest for correcting the bias of the Gini Index and showed that prediction accuracy was improved. However, it has a high computational cost, and needs 10 to 100 iterations.

Fisher et al. [73] also proposed a model-agnostic version to PFI that works as follows: Input: Trained model f, feature matrix X, target vector y, error measure $L(y,f)$.

Estimate the original model error $e_{orig}=L(y,f\left(X\right))$ (e.g., mean squared error).

For each feature $j=1,\dots ,p$ do:

• Generate feature matrix $permute{d}_X$ by permuting feature j in the data X. This breaks the association between feature j and actual outcome y.
• Estimate error $e_{perm}=L(Y,f\left(Xperm\right))$ based on the predictions of the permuted data.
• Calculate permutation feature importance $F{I}_j=e_{perm}/e_{orig}$. Alternatively, the difference can be used: $F{I}_j=e_{perm}-e_{orig}$

Sort features by descending FI.

Global SHAP for model summary: In more detail, we describe SHapely Additive exPlanations (SHAP) and original Kernel SHAP in the respective section. Figure 19 demonstrates the SHAP summary plots on our previously trained vanilla Random Forest Regression model on whole data in the California Housing dataset and Global Shapley values distribution for each feature and feature importance bar chart can be seen, respectively. from left to right. As is seen in the Bee-swarm plot, “median income” is the most important feature on average, and people with less median income are less likely to own a higher median house value.

Figure 20 demonstrates another example of using the summary plot of Shapely values [74]. Nur et al. developed a bidirectional RNN model on heterogeneous student data to predict CGPA ahead of time. The mean absolute value of each feature’s SHAP values can be used to make a conventional bar plot. Lundberg et al. extended the SHAP DeepExplainer [75] implementation to generate Shapely values for sequential or time-series data. The shapely values displayed here represent SHAP values normalized and summed up on the x-axis axis for this student cohort’s first four semesters. The figure demonstrates how the importance of different features changes over time while determining the students’ CGPAs. For example, the importance of the credits passed feature fades away from the first semester to the fourth semester, while “maximum required credits” becomes a more determining factor during the fourth semester to identify student success. This kind of XAI helps both data scientists and non-data scientists decision-makers discover unprecedented findings and plan actionable insights.

(iv) Global Surrogate Models approximate predictions from black box models with interpretable machine learning models [25]. A surrogate model can provide insight into the black box model by exchanging or approximating its results. There are two types of global surrogate models. Surrogate models that use approximation by Fourier Transform match the predictions of underlying models as closely as possible by randomly transforming the Fourier transform while preserving the original spectrum amplitude [65,76,77,78,79]. The method of training a surrogate model is model-agnostic. There is no need to understand how the black box model works. While approximating by intrinsic models, only access the data and the prediction function as necessary [77,80,81,82].

(v) Prototypes and Criticism Explanations: Prototypes are selections of representative data points, and criticisms are instances not well represented by the prototypes [25]. It needs criticism to better understand the complex data distributions instead of only using the prototype in example-based explanations. Maximum mean discrepancy critic (MMD-critic) can be used for efficiently learning prototype and criticism, and MMD-critic can also be analyzed using the nearest prototype classifier. The following function can be used for MMD-critic [83]:

$$L\left(C\right)=\sum _{iϵC}|\frac{1}{n}\sum _{iϵ\left|n\right|}k(x_i,x_l)-\frac{1}{m}\sum _{jϵS}k(x_j,x_l)|$$

(6)

For the test point $\widehat{x}$, the nearest neighbor can be reduced as:

$$\widehat{y}=y_{i*},wherei\ast =arg\underset{iϵS}{min}\left|\right|f_w\left(x^{\prime }\right)-y^{\prime }{\left|\right|}_{HK}^2=arg\underset{iϵS}{min}(\widehat{x},x_i)$$

(7)

For instance, the MMD-critic can be analyzed on the MNIST dataset. There are n = 7291 training and 2007 test greyscale images of 10 handwritten digits varying from 0 to 9 in the handwritten digits dataset. Considering global and local kernel, pixels point can be computed between all data points and points between different classes using relation of $exp(-\gamma ||x_i-x_j\left|\right|)1_{|y_i-y_j|}$, respectively. The results obtained suggested that MMD is effective in selecting the first few prototype and that global kernel outperforms local kernel.

(vi) Influential Instance-based Explanations: Influential instances are the training data points that were the most influential for the parameters of a prediction model or the predictions themselves [25]. Influential models can be crucial in providing valuable information in purposes, such as understanding the model behavior, debugging models, detecting dataset errors, and creating visually indistinguishable training set attacks. There are two types of influential instances, including Deletion Diagnostics and Influence functions. The Deletion Diagnostics is based on Cook’s distance on linear regression [84] and measures the sensitivity of a model by deleting the instance. This is a model agnostic approach, and it needs to delete an instance and retrain the model to measure the model’s sensitivity. The second type of influential instances explanation is called Instance Functions and it belongs to a model-specific class of explanation because this method needs the model’s loss function to be twice differentiable with respect to its parameters. The influential function measures the dependency between an instance and a model’s parameters or prediction. Koh and Liang [85] proposed the following theoretical background regarding measuring the influential instances. For input space of data X to an output space with labels Y, there can be training points $z_1,z_2,\dots ,z_n$ where $z_i=(x_i,y_i)ϵX\times Y$. Suppose that L(z,$\Theta$) is the loss for a point z and parameters $\Theta$, and $\frac{1}{n}{\sum }_{i=1}^{n}L(z_i,\Theta )$ is an empirical risk.

Then, the optimizer for empirical risk can be estimated as [85]:

$$\widehat{\Theta }:=arg\underset{\Theta ϵ\theta }{min}\frac{1}{n}\sum _{i=1}^{n}L{(z_i,\theta )}^1$$

(8)

where it can be assumed that empirical risk is twice differentiable and strictly convex in $\Theta$. For efficiently calculating influence, it has a computational challenge, such as the influence of up-weighting [85],

$$I_{up,loss}(z,z_{test})=-{▿}_{\Theta }L(z_{test},{\widehat{\Theta }}^{T}H\_{\widehat{\Theta }}^{-1}▿\_\Theta L(z,\widehat{\Theta })$$

(9)

which requires forming and inverting hessian of empirical risk, $H_{\widehat{\Theta }}=\frac{1}{n}{\sum }_{i=1}^n{▿}_{\Theta }^{2}L(z_i,\Theta )$.

This requires $O(n{p}^2+p^3)$ operations with n training points and $\Theta ϵR^P$, and it is computationally complex for system such as deep neural networks. Secondly, all training points $z_i,I_{up,loss}(z,z_{test})$ need to be calculated. Underlying different applications of debugging models and fixing datasets from creating training-set attacks, the influence function is a common tool with which we can understand model behavior by looking into its derivation from training data.

Global Post-hoc Model-specific Explanations: There are four types of XAI for global model-specific classes. We already covered influence functions in the previous section. There are a few methods developed by the author of the SHAP method [86] that can be categorized as global model-specific methods [87]. For example, Lundberg et al. [87] proposed a new approach to enhance the explainability of the tree-based family, i.e., Decision Trees, Random Forest, and Gradient Boost, called tree SHAP. This work can be seen as a transition from original local model-agnostic (Kernel SHAP) to model-specific explainability where it gets help from local explanations such as the amount of feature interaction effects in the prediction and combining these local explanations to represent a global structure of the model and maintaining the fidelity to the original model. Game theory-based optimal explanations are also one of the upcoming explanation methods. There are two more explainability methods related to traditional machine learning algorithms such as GAM and Tree-based Ensemble models: (i) feature weight in GAM, (ii) accuracy gain, feature coverage, and frequency of features used for a split in tree-based ensemble models.

i

Feature Weight in GAM: Researchers have created a number of strategies to explain neural network predictions using the saliency or distinctive distinguishing characteristics of an observation. By estimating each feature’s significance to a forecast, another set of approaches generates explanations. While some of these methods are model-neutral, others try to produce attributions by utilizing the architecture of neural networks. Existing methods for evaluating global predictive capacity alter the input space or rely on more interpretable surrogate models, such as decision trees. These methods generate a comprehensible set of rules but may miss the non-linear feature interactions that neural networks learn. To capture more or less sub-populations for global explanations, Generalized Additive Model (GAM) offers configurable granularity. In Hastie et al. [88], a novel soft computing model (artificial intelligence model) based on a boosted generalized additive model (BGAM) and a firefly algorithm (FFA), dubbed FFA-BGAM, is proposed for accurately simulating rock fragmentation (i.e., the size distribution of rocks). The BGAM model was therefore optimized using the FFA as a reliable optimization technique/meta-heuristic algorithm. The weights of the explainable linear model can be utilized to interpret a specific model prediction. The overall relevance of the features in the chosen global set is the measure by which SP-LIME maximizes coverage. The authors in Ref. [89] devised a method for identifying the feature dimension with the highest level of activation by using the gradient’s route integral for a neutral reference input. Based on real and synthetic datasets, the authors demonstrated how GAM illuminates global explanation patterns across subpopulations learned. Based on user studies, the authors also validated the global attributions produced by GAM match known feature importance as being insightful to humans through user studies.

ii

Tree-based ensemble models:

(a)

Accuracy Gain Landslide susceptibility mapping (LSM) is a major component of disaster risk management that involves planning and decision-making activities. A decision tree-based ensemble learning algorithm known as decision forest is one of the popular ML techniques based on a combination of several decision tree algorithms to construct an optimal prediction model. In Kutlug et al. [90], prediction performances of recently proposed decision tree-based ensemble-based algorithms namely canonical correlation forest (CCF) and rotation forest (RotFor) are tested on LSM. For the assessment of the performances, overall accuracy (OA), success rate curves (SRC), and area under the curve (AUC) are studied. Another popular application of Tree-based Ensemble Models is in the complexity of the decision-making process associated with lane changing. Mousa et al. [91] implement the XGB in predicting the onset of lane changing maneuvers using CV trajectory data. The performance of XGB is compared to three other tree-based algorithms, namely, decision trees, gradient boosting, and random forests. The results indicate that XGB is superior to the other algorithms with a high accuracy value of 99.7%. This outstanding accuracy is achieved when considering vehicle trajectory data two seconds prior to a potential lane change maneuver.

(b)

Feature Coverage A detailed characterization of land use and land cover at the ecotope level is necessary for environmental assessments. Chan et al. [92] investigate if it is feasible to categorize ecotopes using airborne hyperspectral imagery. The authors compare and contrast Adaboost and Random Forest, two tree-based ensemble classification algorithms, based on metrics such as classification accuracy, training time, and classification stability. Their results show that Adaboost and Random Forest outperform a neural network classifier and that there is just a 1% difference in their total accuracy, which is close to 70%. Random Forest is faster and more steady throughout training, however. It is believed that both ensemble classifiers perform well with hyperspectral data.

(c)

Frequency of Features Used for Split The study in Chen et al. [93] evaluates the effectiveness and prognostication of a number of tree-based ensemble approaches for mapping possible groundwater springs. In order to create a groundwater spring potential map, the paper offers a unique hybrid integration method based on the J48 Decision Trees (J48), AdaBoost (AB), Bagging (Bag), RandomSubSpace (RS), Dagging (Dag), and Rotation Forest (RF) algorithms. Based on the correlation attribute evaluation approach, the contribution of each groundwater spring-related variable was to find the best predictive value. The results showed that all models had strong predictive capabilities. The study emphasizes how effective and precise the ensemble technique is for measuring the potential of groundwater springs.

For flash flood susceptibility modeling, the combination of Tree-Ensemble models and Feature Selection Method (FSM) in Bui et al. [94] demonstrated superior learning and predictive abilities than the ensemble models that had not undergone an FSM. As a way to identify the best variables for use in evaluations of flood susceptibility models, the FSM employed a fuzzy rule-based algorithm known as FURIA as an attribute evaluator. GA was used as the search strategy. The innovative FURIA-GA method was merged with the ensemble techniques LogitBoost, Bagging, and AdaBoost. The use of different statistical metrics, provides different outcomes concerning the best prediction model, which mainly could be attributed to sites specific settings.

5.3. Deep Learning Models

The deep learning models’ internal representation for the neurons in the intermediate layers needs to be highly explainable since deep learning models are black-box. Data scientists use these XAI techniques for the purpose of fine-tuning and debugging during the training of the models. Domain experts without a data-science background need XAI techniques for the deep learning models to explore and understand the predictive power, results, and trustworthiness of a deep learning model. In this section, we explained three types of deep learning-based XAI techniques. A new model named Graph Convolutional Network (GNC) is used to project the predictions onto an interpretable domain unlike in DNN where this kind of projection is not possible. Although, GNC does not have the appropriate methods to explain the intermediate states in the layers. Hence, Schwarzen et al. [95] have come up with a methodology to use layer-wise relevance propagation (LRP) with GNC to form Convolutional Visualizations for Text Graph Classifiers. LRP helped to calculate the amount each neuron contributed to the activation. This is propagated backward layer wise (see Figure 21). It is also used to compute edge relevance factor throughout the layers. LRP has shown to be useful in the medical field as well. A term XAUG (eXpert AUGmented Variables) provides extra variables to augment the inputs. These are combined with LRP in Agarwal et al. [96]. This study shows that when XAUG is used with the low-level DNNs, then the accuracy of the classifier has increased by 30–40%. Combining LRP with XAUG variables helps to rank the features and forms a reduced feature set that helps to capture the network, which in turn helps to discover various underlying patterns.

5.3.1. Explanations of DNN Representation

This type of XAI methods use layer-wise relevance propagation (LRP), which propagates the prediction function backwards from the higher layers to the lower layers in the neural network [95,96,97,98]. The medical field is developing at a faster pace, where advanced Deep Neural Network (DNN) models are used to compare two very similar disorders or diseases [99]. One such application has been found in the method applied by Yan et al. [100], where a new framework DNN + LRP has been proposed to differentiate between schizophrenia patients (SZ) from healthy controls (HCs) using functional network connectivity (FNC). DNN with one input layer, multiple hidden layers, and one output layer is the standard framework used to distinguish images. The task mentioned above also had to make sense to a human as to why a particular decision was made by the model. In order to comprehend the meaning behind a prediction, LRP was used with the DNN.

Urban sound classification is a new field where DNN along with LRP was applied in Colussi et al. [101] in order to understand the results better. LRP explains the significant features that contributed to the successful or erroneous prediction. The output is decomposed into relevance scores, which are obtained when iterated from the output to the input layer. Due to LRP, certain relationships were identified between the sound frequencies and the prediction of the sound class. This result was helpful in assisting people suffering from hearing loss to be able to drive carefully. Lapuschkin wrote their thesis [102] on understanding the black boxes with the help of LRP. This shows that LRP helps to interpret the results of the non-linear classifiers. It is also compared with other interpreting tools to analyze which one works better on images and texts. This method helps to form a tool that compares pre-training models and datasets to decompose the prediction strategy and identify the flaws that might hamper the prediction.

5.3.2. Explanations of CNN Representation

XAI methods for Convolutional Neural Network (CNN) representations follow the activation maximization framework. The activation maximization method aims to maximize the activation of specific neurons in neural networks. The network weights and biases are iteratively tuned during the default training so that the neural network’s error is minimized across training examples [103]. It can be categorized into three types: Learning Level of Abstraction, Learning Semantic Concepts, and Learning Distributed Codes.

i

Learning Level of Abstraction: There is a very limited understanding of the way CNN represent images. To give a better understanding, network visualizations have proved to be effective. Mahendran et al. [104] focused on analyzing some landmark representations with the help of visualization techniques. In their findings, it was observed that several layers in CNN retain a photographically accurate image with slight variances in their degrees. Additionally, Qin et al. [105] provides comprehensive review of these representative CNN visualization methods that can be utilized for various computer vision tasks.

Healthcare is one of the critical domains where a lot of confidence and trust is required in the decisions made by any person or model. Thus, even though the black-box models usually give accurate results, we have very little idea about how the model makes a particular conclusion. This problem is handled in Maweu et al. [106] where a modular framework named CNN Explanation Framework is proposed. This framework allows users to understand the underlying structure of the CNN network using various statistics, visualizations, feature detection, etc. Streamlining the intraoperative cancer diagnosis is another field in the healthcare domain that has proved CNN to be effective after cancer diagnosis. Hollon et al. [107] designed and implemented a model combining the stimulated Raman histology, label-free imaging method, and deep CNN, that predicts the diagnosis in an automated fashion. Apart from the applications, CNN have been useful to solve common issue faced in biomedical imaging, which is, texture analysis. Ref. [108] made a CNN that is specific for texture analysis and is evaluated on general biomedical texture classification.

A new method for explaining the reasoning behind the predictions made by artificial intelligence (AI) systems has been proposed by a team of researchers. Zhang et al. [109] report that CNNs have achieved superior performance in various tasks. The authors aim to transform chaotic features of filters inside a CNN into semantically meaningful concepts, such as object parts. They use a decision tree to explain CNN predictions at the semantic level. Another work improving the interpretability of CNN is seen in Dong et al. [110]. The authors propose a method to improve the interpretability of deep neural networks.The authors propose a novel technique to improve the interpretability of deep neural networks by leveraging human descriptions.

ii

Learning Semantic Concepts: The term semantically meaningful means that the deep learning models such as CNN can be explained in human-interpretable form. One of the aspects of CNN can be explored in Ref. [111], where they tried to understand a CNN model with respect to the activation dimensionality reduction and visualization technique. Usually supervised learning uses a set of interpretable methods. In this paper, they have found that it is possible to even make the unsupervised methods semantically meaningful by experimenting on a dataset of activations, which resulted in three paths or semantically meaningful “tuning dimensions”. A new framework named Network Dissection has been proposed in Ref. [112] where latent representations of CNN are quantified by evaluating the position of the individual hidden units and the semantic concepts associated with it. The motive is to prove the hypothesis that interpretability of units is equivalent to a random linear combination of units. Existing DNNs can be explained by using a global as well as a local interpretability method. Global explanations are useful when we want to understand the reasoning behind the final output displayed by the model. The features associated with DNNs are usually difficult to interpret. An approach developed in Ref. [113] first associates the human-interpretable concepts with the vectors present in the feature space, which is done using a mathematical formula that formulates it into an optimization problem. The semantic vectors formed from the optimal solution helps to explain these models globally as well as locally. Usually, the post-hoc interpretability methods lack transparency in understanding the features learned by the model. In order to gain any individual’s trust, DNNs should be able to explain the concept using human-interpretation. Ref. [114] proposes a guided learning method where an additional layer in CNN is dedicated to understand the relations between the word phrases and the visuals. Sometimes, learning the semantics of the features takes a toll on the accuracy of the model. Thus, the method adopted in this paper optimizes the learning of the semantics related to the features as well as the accuracy.

iii

Learning Distributed Codes: Tasks related to image recognition, object detection, or any operation related to images are performed commendably well by CNN models. The advantage of CNN performing well with images is because of the features it learns with the help of intermediate layers, which are sometimes difficult to explain to a human. Qin et al. [105] talks about the several methods available to represent the learning of each layer in a visual form. A survey of several visualization methods has been done in this paper that compares different aspects of the model, in terms of algorithm, experiments, and results. Certain features learned by the model can be represented by the patterns. Convolutional filters are used to extract complex features that can be represented in a visual format for human interpretation. This helps in altering any adjustable parameter in order to get an optimal output. With the advancement of interpretability in CNN models, these have started to be deployed in various sectors such as security enhancement, network design etc. The black-box nature of the CNN keeps the understanding as well as organization of the model a mystery to the humans. To unravel these mysteries, Rafegas et al. [115] proposes an approach where the activity of individual neurons can be displayed using the Neuron Feature Visualization method. They make it interpretable by explaining the terms with respect to specific selective properties such as image color or image class. The framework proposed in this paper emphasizes finding color selective class selective neurons in the layers. They help to statistically determine how these selective features are varied along the layers and what contribution each of these feature have on the final outcome. Except in technological fields, deep learning models have paved the way for being used in analyzing clinical tasks from the data stored in Electronic Health Records (EHR). Shickel et al. [116] provides detailed information on the existing deep learning frameworks and techniques. They talk about how these methods have been applied on the EHR records to extract information, predict the outcome, phenotype, and deidentify. Currently, there is a lack of set universal benchmarks and certain limitations still persist. Hence, a lot of research in these areas are taking place.

5.3.3. Explanations of RNN Representation

Explanations of RNN layers can be classified into the following categories:

i

Learning Long-term Dependencies: Learning representations for sentences or multi-sentence paragraphs plays an important role in natural language processing. Many tasks that largely rely on sentence-representation learning have benefited greatly from recent developments in Recurrent Neural Networks (RNNs), especially Long Short-Term Memory (LSTM) and variations. Zhang et al. [117] presented a broad paradigm for text modeling that solely relies on convolutional and deconvolutional processes. Since the suggested method does not use sequential conditional generation, exposure bias and teacher-forcing training problems are avoided. Their method enables the model to completely encapsulate a paragraph into a latent representation vector, which can be decompressed to recreate the original input sequence. One of the LSTM models developed in Butepage et al. [118] was based on the concept of bottleneck encoding and decoding from earlier to later frames. They outperformed the recurrent methods for both short-term and long-term predictions, and the predictions generalize to new subjects and behaviors.The processing of variable-length input sequences and the provision of variable-length outputs, including the creation of full-length sentence descriptions that go beyond traditional one-versus-all prediction challenges, are both desirable in a video model. This can be demonstrated in Donahue et al. [119] where long-term RNNs can significantly outperform static or flat temporal models in visual tasks when there is a sufficient amount of training data to learn or hone the representation. Another application of RNNs can be seen in learning graph representations, which is done by capturing diverse graph features in vector space. Goyal et al. [120] suggest an embedding strategy that learns the structure of evolution in dynamic networks and can predict unseen links with improved precision. In order to understand how the network dynamics affect the prediction performance, the model, dyngraph2vec, uses a deep architecture made-up of dense and recurrent layers to understand the network’s temporal transitions. Recent developments in generative modeling has been explored in Zhao et al. [121] that successfully combine the variational auto-encoder (VAE) framework with discrete latent representations. Figure 22 demonstrates a SHAP force plot for the same bi-directional RNN model we discussed in Figure 20. Tracking the progress of target users through a temporal RNN model has become easy with SHAP. Nur et al. [74] demonstrates that the SHAP force plot helps to understand the effect of course level progression on CGPA for heterogeneous student data through eight semesters (x-axis). At the beginning of the enrollment, the higher magnitude of the low-level courses (indicated by blue region) shows a consistent pattern for being a successful student, with $\tilde{9}$5% confidence. Later semesters maintain the same confidence of the model for higher-level courses, demonstrated by the uniform width of the red region.

ii

Learning Hierarchy Dependencies: Action classification, motion prediction, and motion production require an expressive representation of human motion. The limited scope of generative models of 3D human motion makes it difficult to generalize to new movements or applications. Butepage et al. [118] builds a general representation from a huge corpus of motion capture data and generalizes it effectively to novel, unseen movements using a deep learning framework. The positional variation of the skeleton’s joints can be used to reflect the motion features of human motions. With carefully built hand-made features, traditional techniques often extract the spatial-temporal representation of the skeleton sequences. Du et al. [122] propose an end-to-end hierarchical RNN for skeleton-based action identification, which can identify actions based on the relative motion between the limbs and the trunk.

Another major area where hierarchical attention model has been developed is the classification of sentiments, which seeks to identify whether a user’s attitude is positive, neutral, or negative. For the purpose of classifying sentiment across many languages, Zhou et al. [123] suggest an attention-based a Long Short Term Memory (LSTM) network.

5.3.4. Local Post-Hoc Interpretability Methods

Local Post-Hoc explanations can be categorized into model-agnostic and model-specific explanations (see Figure 23).

5.3.5. Local Model-Agnostic Explanations

Local model-agnostic methods consist of four major methods: prediction-based, domain-specific, black box adversarial, and instance-based counterfactual. Prediction-based explanation includes techniques such as LIME that employ an intrinsic secondary model such as the linear model, fits the model locally, and inspects this surrogate model for explaining the black-box. Domain-specific expert systems use rules to make decisions and deductions used for rule-based methods [124]. The black box adversarial explanation includes techniques where the adversary has no assumptions regarding the model’s structure, parameters, or gradients and it only can use model prediction similar to an API system by passing the input and receiving the corresponding output [125]. Finally, instance-based or example-based counterfactual explanations can be best explained by Wachter et al. [126].

5.3.6. Local Interpretable Model-Agnostic Explanations (LIME)

Ribeiro et al. [127] proposed that LIME can approximate the black-box prediction locally by fitting (permuted a sample around the intended instance and black-box prediction of these permuted data points) new data points to an intrinsic explainable model such as linear regression or decision tree. We trained and tuned hyper-parameters of a Random Forest Regressor on the California housing dataset as black-box model, so LIME gives us some explanations about a random data pint and RF’s prediction of this instance (Figure 24). Prediction_local means the LIME’s prediction with the simplified surrogate model, and Right means the prediction by the Black Box model, which is a Random Forest Regressor here.

LIME samples increase the weights of local data points around the interest instance and learn an explanation for prediction of that instance (Figure 25).

The image explanation of LIME can be displayed directly on the image samples. After determining the superpixels (interconnected pixels with similar color intensity), LIME explains how much superpixels are contributing to the final model prediction. For this purpose, the LIME authors segmented the unstructured image data into various superpixels and transformed these superpixels into a tabular format. The features include a list of distinct superpixels and rows for the existence and lack of existence for each superpixel. To illustrate how LIME works for an image dataset including 1000 images of cats, dogs, and pandas, two black box models were trained: a CNN model with three convolutions, three max pooling, and a dense layer, and a pre-trained Inception-V3 model Google team [128]. As seen in Figure 26 the green parts indicate that the superpixels of the image increase the label’s probability, and the red parts indicate a decrease.

LIME’s explanation for textual data works differently from both tabular and image data due to its unstructured nature. Different from the tabular format, the permutation of textual data is not possible. So the the authors of LIME [127] proposed a new method to sample new texts from the original one by masking some of the words from the original one and calculating weight based on one minus the proportion of words that were removed. This method works with different vectorizations or embedding methods as long as there is a pipeline to map the text data to the output of the black-box model. To illustrate how LIME works for text data, a Random Forest model was tuned and trained on outputs of TFIDF vectorizer on BBC news dataset [129] to classify news into five classes including business, sport, politics, tech, and entertainment. As is seen in Figure 27, there is a random datapoint and corresponding explanation from LIME. In addition, the LIME explanation shows important vocabularies (features) such as “online”, “technology”, “web”, “digital”, “people”, “net”, “internet”, “using”, etc., for the black-box model to classify this news into tech class with 51% confidence.

Figure 26. Left: LIME explanations for the CNN model. Middle: Random datapoint and Right: LIME explanations for the Google’s Inception V3 model [130].

Figure 26. Left: LIME explanations for the CNN model. Middle: Random datapoint and Right: LIME explanations for the Google’s Inception V3 model [130].

An implementation of the LIME model can be found in Ref. [131], where the focus is on understanding the local behavior of the Species Distribution Model (SDMs) and the underlying relationships between environmental variables and species.

5.3.7. SHapley Additive Model Agnostic exPlanations (SHAP)

SHAP is an XAI tool introduced by Lundberg and Lee in 2017 [86]. This method helps to explain the predictions by using coalition game theory, where the prediction serves as the “payout” while each feature value acts as “players” in the game. This method aims to distribute payouts fairly to players [35]. Hence, Shapely values for each feature are calculated in the following manner:

$$[{\varphi }_i\left(v\right)=\sum _{S\subseteq Ni}\frac{\left|S\right|!\left(\right|N|-|S|-1)}{\left|N\right|!}(v(S\cup i)-v\left(S\right))]$$

(10)

Here, ${\varphi }_i\left(v\right)$ is the Shapley value calculated for the $i^{th}$ feature. N is the total number of features, while S acts as the subset of N. The function v calculates the payout or Shapely value of that particular feature. This method has proved to be a better feature selection method as compared to other traditional methods used for this purpose [132]. Various visualization graphs make the output of the model more readable as well as understandable. Recently, additional explanation methods have been added that can explain the reasoning behind why a sample is more likely to belong to a particular class, why a sample gives different observations when tested with discrete groups, and why a model performs poorly on a given sample [133]. In Figure 28, one can see the local Shapley values distribution and feature importance ranking of a random datapoint on a trained RF model for the California Housing dataset, illustrated from left to right. In this example, “median income”, “being located inland”, “longitude”, and “latitude” are the most import features for the black box model to determine the “median value of house”. In the left, a bee swarm plot was used with a subset of datapoints around the intended data point to illustrate how top important features impact the output of black box model. It should be noted that both LIME [127] and Kernel SHAP [86] depend on perturbing data in vicinity of the original data and approximating the prediction of the model by ignoring the quality of the black box prediction itself. Even though the perturbation-based sampling is simple and the prediction-based approach is straightforward, it can cause instability and unreliability in explanations [134,135]. In Figure 29, to illustrate how the SHAP works for text data, a Random Forest model was tuned and trained on outputs of TFIDF vectorizer on BBC news dataset [129] to classify news into five classes including business, sport, politics, tech, and entertainment. As is seen in Figure 29, there is a random datapoint from the dataset and the most important part of the text (red regions) that increases the output of the Random Forest model and helps the model to classify this example as a tech class. In the right figure, the SHAP summary plot shows an important group of vocabularies (features) for the black-box model to classify this news into tech class.

5.3.8. Anchor: Rule-Based Explanations

To explain the behaviors of complex models, anchors can be used that represent sufficient conditions for prediction. A model can be developed for anchors to enable users to predict the behavior of unseen instances with less effort and explanations than existing linear explanations. Suppose that A is a set of predicates acting on interpretable representation such that $A\left(x\right)$ returns 1 if all its feature predicates are true, for instance x, then A is an anchor if [124]

$$E_{D(z/A)}\left[1_{f\left(x\right)=f\left(z\right)}\right]\ge \tau ,A\left(x\right)=1.$$

(11)

If there is black box classifier f, instance x, distribution D, and desired precision $\tau$ and anchor A consists of a set of features on x achieving $prec\left(A\right)\ge \tau$ such that

$$prec\left(A\right)=E_{D(z/A)}\left[1_{f\left(x\right)=f\left(z\right)}\right]$$

(12)

For an arbitrary D and black-box model f, a probabilistic definition can be used for computation such that

$$P\left(prec\right(A)\ge \tau )\ge 1-\delta$$

(13)

The anchor with the largest value converges if multiple anchors meet this criterion. Additionally, coverage of an anchor in distribution D is given by

$$\underset{As.t.P\left(prec\right(A)\ge \tau )\ge 1-\delta }{max}cov\left(A\right)$$

(14)

Estimating precision and coverage bounds under D, perturbation distributions and a black box model can be used. Evaluation of anchor explanations can be obtained for complex models on different tasks, such as simulated users in which numerical datasets can be used by dividing them into a training, validation, and test set. For each dataset, anchor explanations can be obtained by computing coverage and precision. In Figure 30, an example of the Rule-based explanation by the Anchor package for the random datapoint on the Random Forest model trained on the California Housing dataset can be seen below.

In Figure 31, Tsai et al. present a novel adversarial XAI approach to transfer learning without knowing or modifying the pre-trained model. They proposed BAR, a novel approach for adversarial reprogramming of black-box ML models via zeroth order optimization and multi-label mapping techniques. By transferring access-limited ML models through black-box learning, the BAR method can outperform state-of-the-art fine-tuning methods by using input-output model responses instead of complete knowledge of the target ML model.

5.3.9. Counterfactual Instance-Based Explanations

A counterfactual explanation of a prediction describes the smallest change to the feature values that change the prediction to a predefined output [25]. Counterfactual explanation can be used in purposes, such as informing about a decision made of the subject matter, providing grounds for contesting adverse decisions, recommending changes in feature space for flipping output, and understanding the potential need for change in decision-making to obtain the desired result. Unconditional counterfactual explanations as a method of counterfactuals can be used for providing a meaningful explanation of automated individual decisions. For instance, counterfactual explanations can be explained with the statement, “Your annual income was $35,000, so you were denied loan. You would have been offered loan if your income has been $50,000”. It describes how the word has to be used for obtaining the desired result as an illustration of counterfactual illustrations [137]. In counterfactual explanation, it depends upon external facts for decision. For calculating counterfactuals, the following objective function can be used in many standard classifiers [126]:

$$arg\underset{w}{min}L(f_w\left(x_i\right),y_i)+\rho \left(w\right)$$

(15)

where $y_i$ is label of point $x_i$ and $\rho (.)$ is regularizer over weights. In order to obtain counterfactual $x^{\prime }$ that is close to $x_i$ such that $f_w\left(x_i^{\prime }\right)$ is equal to $y^{\prime }$, we can use the following relation:

$$arg\underset{x^{\prime }}{min}ma{x}_{\lambda }\lambda {(f_w\left(x^{\prime }\right)-y^{\prime })}^2+d(x_i,x^{\prime })$$

(16)

where $d(.)$ is distance function that measures the distance between $x_i$ and $x^{\prime }$, and $x^{\prime }$ can be iteratively solved to match with x to maximize $\lambda$. The distance can be measured by L1 norm or Manhattan distance normalized by inverse median absolute deviation over a set of points P.

$$MA{D}_k=media{n}_{jϵP}\left(\right|X_{j,k}-media{n}_{lϵP}\left(X_{l,k}\right)\left|\right)$$

(17)

and

$$dist(x_i,x^{\prime })=\sum _{kϵF}\frac{|x_{i,k}-x_k^{\prime }\left|\right)}{MA{D}_k^{\prime }}$$

(18)

where k is feature number that varies across dataset to make $x^{\prime }$ close to x.

Figure 32 is an example of Counterfactual Explanation with DiCE package for the random datapoint on trained RF on California Housing dataset. As is seen, the datapoint of interest has high median house value, and DiCE provides a list of datapoints with smallest changes and low median house values. The Counterfactual explanation can be employed as a recommender system to give the end users some insights about alternatives to different prediction. For example, if a loan application model rejected the eligibility of an applicant, counterfactual explanation would be utilized to recommend the alternatives to an applicant to get approved.

5.3.10. Adversarial Explanations

The prediction-based methods. including LIME and SHAP, imitate the black-box model prediction and report feature importance to explain how the decisions of the model have been made. On the other hand, the counterfactual explanations provide a closer form of explanation for the end users by generating diverse instances close to the intended datapoint and contradicting the predefined prediction, but these diverse examples do not provide a model summary [25]. Although both prediction-based and counterfactual explanation methods provide useful information at summary and example-wise levels, they have their limitations and presume that the black box model itself works well. Hence, the unification of both counterfactual explanations and feature importance helps the users to have better insights. One example of this unification can be seen in the work of Chapman et al. [138], named Feature Importance by Minimal Adversarial Perturbation (FIMAP). Another example called Explanation the Minimal Adversarial Perturbation (EMAP) [139] is the initial form of FIMAP, which is a neural networks-based approach that focuses on explaining the reason behind an instance being misclassified by the underlying black box model. An application of the Instance wise Feature selection (IFS) method in Liang et al. [140] predicts the output based on a few selected features. The output given by the model is also taken as an additional input to increase the learning as well as the accuracy of the explainer. Although the before-mentioned works are innovative, they are not significant works in adversarial machine learning. In the following, we overview two classes of model-agnostic and model-specific adversarial machine learning methods that help to understand and test the vulnerability of machine learning models. The black-box adversarial is model-agnostic due to having the least assumptions about the structure and architecture of black box model and only using training data, or just using the inference part of a black-box model like API (zero-knowledge attack) [125].

5.4. Black-Box Adversarial Explanations

5.4.1. Surrogate Attack

Papernot et al. [41] proposed a zero-knowledge adversarial method, in which the adversarial does not have access to the internal weights of the target model or even the training data. The adversarial only has limited knowledge regarding the domain of data and limited access to the inference part of the target model to get the labels for the synthetic data that they manipulated. In addition, they trained a local model to approximate the target model using synthetic data and the corresponding labels.

5.4.2. Data Poisoning Attack

In an adversarial attack, there is another type of attack called a man-in-the-middle attack, in which the attacker queries the API of an online machine-learning system with synthetic or poisoned input data samples [141,142]. A functionally equivalent adversarial classifier statistically close to the target is then trained from returned labels [141,142,143]. Black-box adversarial explainable attack techniques are implemented as counter-measures to adversarial samples to test the robustness of actual algorithms and limit the availability of the training data to the adversary [142,143]. For example, Cinà et al. developed a black-box adversarial model to validate the robustness against data poisoning attacks in clustering algorithms [143]. They use a constrained minimization algorithm that is general in structure and easy to customize by an attacker. In contrast, it assumes no knowledge about the internal structure of the victim clustering algorithm, and it only allows the attacker to query it. The authors demonstrate how their crafted algorithm works on different single and ensemble clustering algorithms against data poisoning attacks.

5.4.3. Local Model-Specific Explanations

However, the local model-specific explanations methods consist of various techniques corresponding to various types of black-box models available in the literature, we reviewed a few well-known methods in this study including class-specific error backpropagation, activation mapping, tracking weights of gradient descent, investigating deep representation, and saliency mask.

5.4.4. Class-Specific Error Backpropagation for Model-Specific Explanations

• Layer-wise Relevance Propagation
  Black box models often become a problem in critical areas such as aerospace, security, etc. XAI is a booming field where a lot of advancements related to understanding the reasoning behind the model’s predictions have been taking place. For visual explanation, a common technique named Layer-wise Relevance Propagation (LRP) is used. It helps by creating a heatmap depicting the contribution of each pixel value in the image. Selective LRP introduced by Jung et al. [144] is able to produce better heatmaps as it combines relevance-based and gradient-based methods. Lane change prediction is one application where LRP can be used. Wehner et al. [145] shows how LRP can be used on the normalized LSTM layer, which is used on live data and the explanations provided by a digital twin on a German highway. This information is passed onto an interface that communicates and explains the decision to the human user. A comparison of LRP to the LIME and SHAP explainability methods has been done in Ref. [146]. The experiments show that LRP is a better method as compared to the latter options. The experiments were conducted on a mixed numerical dataset of Credit Card Fraud Detection and Telecom Customer Churn Prediction.
• Excitation Backpropagation
  Most of the black box models use saliency maps to explain the decision as it does not use any internal values of the model. The concept of perturbing the input so as to note down the consequent changes in the output has been extended in Ref. [147]. Saliency maps are generated as a sequential search problem, which is leveraged upon Reinforcement Learning to detect the perturbation that led to the high-quality explanations. Progress can be seen in the methods explaining the decision of the model. The major drawback faced by these methods are the computational cost and the architectural constraints. Cooper et al. [148] developed a model-agnostic method named Hierarchical Perturbation that uses robust saliency maps and computes 20 times faster than other explanability methods.

5.4.5. Class Activation Mapping for Model-Specific Explanations

• Deep Convolutional Neural Networks (CNN) are one of the most commonly used models, yet their interpretation is still a challenge. Class Activation Map (CAM) represents the features learned by the model from the data. A more advanced CAM is introduced in Ref. [149], where the principal components of learned representations from the convolutional layers are visualized. They are efficient and can work with any CNN model without the need of the layers to be trained or modified. The application of these deep learning methods can be seen in Medical Image Analysis. Shi et al. [150] gives an overview of why these black-box methods have been restricted for clinical use. It talks about how these methods have been explained, which areas still pose a challenge to be explained, and which domain still requires further research. Portfolio management is one of the fields where an explainable reinforcement learning framework has been applied, e.g., in Ref. [150]. CAM is used to understand the network outputs. It maps the price movements and highlights the time intervals, which becomes useful to understand the trend and, hence invest in the target asset.
• GRAD-CAM is class specific and hence generates a separate visualization for each class present in the image. Decoding the CNNs is itself a tedious task, hence a new explainability tool named Neuroscope has been presented by Schorr et al. [151]. This tool not only offers state-of-the-art visualization techniques but also helps in semantic segmentation of the CNN layers by providing a visualization of all the layers used in CNN. There are many algorithms that have started to be used with CNN. A comparison of using Grad CAM and Integrated Gradient algorithms has been made in Ref. [152]. This revealed that when the heat-maps were compared of fair race and biased race models, fair race models were able to capture more salient features, and thus, it is more useful to work on fair datasets if available. With the increase of temporal datasets, Multivariate Time Series (MTS) classification has gained more application. Although these applications have been limited due to the deep nature of methods, which could not be explained completely with the existing methods, it relies on post hoc model-agnostic explainability methods. An Explainable Convolution network for MTS classification has been introduced by Fauvel et al. [153], which successfully extracts the precise variables and time stamps of the input data, which is critical to generate faithful explanations. Various approaches are introduced to explain the working of the deep learning models as without clear understanding behind the prediction of a model, trust cannot be infused. Pham et al. [154] comes up with a model-specific approach that unifies CAM and Attention Mechanism into Temporally Weighted Spatio-temporal Explainable Neural Network for Multivariate Time Series (TSEM). This has proved to outperform XCM as it combines the capabilities of both RNN and CNN. Here, RNN hidden units are used as weights for CNN feature maps temporal axis.

5.4.6. Tracking Weights of Gradient Descent for Model-Specific Explanations

Appearance modeling is a major application where a tracking algorithm is used. Usually, there are a lot of features involved to form a successful tracking algorithm. The challenge is to combine multiple features in a way that improves tracking accuracy in two ways. First, by increasing the representation accuracy, second, by enhancing the discriminability between the tracked target and the background. Lan et al. [155] have successfully formed an unified feature learning framework, which exploits both the respects and hence improves the visual tracker. With the advancement of visual object tracking, deep tracking has become an important issue to address. The initial solution was to deceive the deep trackers by applying adversarial attacks, which injected unnoticeable disruptions in the video frames. This could not provide a complete solution to the problem as it was video-specific and real life video tracking as well as re-initialization could still lead to deep tracking. A solution was proposed by Liu et al. [156], in which one perturbation can cause tracker malfunction in all videos, making it an offline universal adversarial attack. Tracking weights have been found to be useful in case of a low-level quadrotor, as proposed by Pi et al. [157], which uses neural networks with model-free reinforcement learning. The requirements of the policy gradient algorithm has been relaxed to improve the training efficiency. This quadrotor is trained to learn the improved algorithm, whose output is directly mapped to the four actuators in the simulator. This function is useful when the agent is in an unknown area.

5.4.7. Gradient-Based Adversarial Explanation

Deep Neural Networks are known to be susceptible to adversarial attacks that purposefully perturb inputs to force misclassifications [158,159]. Gradient-based attack methods or defenses against adversarial attacks have been the focus of most studies [160]. Classification performance has not significantly improved despite the widespread use of de-noiser models to reduce adversarial noise [159]. For example, Carbone et al. demonstrate the stability of saliency-based explanations of Neural Network predictions under adversarial attacks in a classification task [161]. The authors implement a gradient-based XAI method using Bayesian Neural Networks, which is considerably more stable under adversarial perturbations of the inputs and even under direct attacks on the explanations. Furthermore, they explain this result in terms of the geometry of the data manifold. Techniques using gradients-based adversarial explanations can be divided into two main categories.

• Minimum-distance: For gradient descent algorithms, minimum distance adversarial attacks accumulate velocity vectors across iterations in the gradient direction of the loss function [162,163,164,165]. Memorizing previous gradients helps navigate narrow valleys, small humps, and poor local minima or maxima. In stochastic gradient descent, the momentum method also stabilizes the updates. In a perturbation-based minimum distance adversarial attack method, the original input is slightly perturbed such that the perturbed input is classified differently than the original instance. Using these models in security-critical areas is impossible because such a small perturbation can cause them to falter. For example, Figure 33 illustrates how adversarial attacks involving small, imperceptible perturbations can compromise medical deep learning systems [163].

  Figure 33. Examples of adversarial attacks crafted by the Projected Gradient Descent (PGD) to fool DNNs trained on medical image Kaggle datasets$\acute{\mathrm{F}}$undoscopy [166] (first row, DR = diabetic retinopathy), Chest X-ray (Wang et al. [167]) (second row) and Dermoscopy [168] (third row). Left: normal images, Middle: adversarial perturbations, Right: adversarial images. The left bottom tag is the predicted class, and green/red indicates correct/wrong predictions [163].

  Figure 33. Examples of adversarial attacks crafted by the Projected Gradient Descent (PGD) to fool DNNs trained on medical image Kaggle datasets$\acute{\mathrm{F}}$undoscopy [166] (first row, DR = diabetic retinopathy), Chest X-ray (Wang et al. [167]) (second row) and Dermoscopy [168] (third row). Left: normal images, Middle: adversarial perturbations, Right: adversarial images. The left bottom tag is the predicted class, and green/red indicates correct/wrong predictions [163].
  By applying the gradient sign to an actual instance only once, the fast gradient sign minimum (FGSM) distance attack generates an adversarial example by assuming linearity around the data point [162,164,169]. However, a large distortion may make the linear assumption invalid in practice. Iterative FGSM moves the adversarial example greedily in the direction of the gradient sign with each iteration to avoid the problem of large distortion [170]. In one pixel adversarial attack, the authors show with explainable visualizations that by perturbing only one pixel with differential evolution, a black-box DNN can be compromised, with the only information available being the probability labels [169,171,172,173,174].
• Patch-optimization: Our discussion of adversarial attacks and XAI techniques has primarily focused on attacks against fixed inputs or single instances. Because adversarial perturbations are calculated only with a particular instance, they lose their effectiveness when applied to other instances or when slightly transformed. The adversarial patch approach optimizes the average adversarial objective function across all transformations [175,176,177,178]. It trains a single adversarial patch that fools a classifier with high probability by applying arbitrary transformations to random images. A single affine transformation can implement rotation, scaling, and translation, so the entire operator can be differentiated [179,180,181]. Wang et al. describe the proposed end-to-end physical camouflage adversarial attack in detail [179]. The research demonstrates how adversarial camouflage can be interpreted before and after an attack. This study attempts to explain why the adversarial camouflage detector fails. The authors can see that the model’s attention on the target category is dispersed after painting the camouflage on the decision evidence of the model. The authors provide some partial occlusion cases, demonstrating that adversarial camouflage works well for most partial occlusion scenarios.

5.4.8. Investigating Deep Representation for Model-Specific Explanations

Deep learning models are starting to be applied in various fields, even though the model-specific explanations for this are still under development. Du et al. [20] investigated the existing explanations behind a certain prediction made by the machine learning model. The methods listed in the paper, can be further improved by generating more user-friendly explanations as well as including evaluation metrics. Convolutional Neural Networks (CNN) are one of the extensively used deep learning models in the medical field. Stacke et al. [182] talks about the extent of a model-specific domain shift present in tumor classification as the internal representation learned by the trained CNN model, and compares it with the result of the test data. This method helps in revealing the sensitivity level of the model due to variations as well as informs about the new data that might cause a problem to be generalized by the model. Another application of deep learning models in the field of pathology can be seen in the study by Baur et al. [183]. This paper has developed an approach where the anomalies are properly detected instead of being treated as outliers, as in the previous approaches. Apart from detection, it also describes those anomalies by comparing the input images to the reconstruction of the 2D brain MR slices learned by the model.

5.4.9. Saliency Mask for Model-Specific Explanations

The purpose of Saliency Mask is to eliminate the portions of the image that highly deteriorate the classification performance and hence result in a vague explanation. Although the reasoning behind the model’s decision results in higher understanding, it equally results in harming the privacy. Image-based model inversion attacks have been explored by Zhao et al. [184]. It gives an idea about which explanations are at a higher privacy risk. They developed a method to reconstructing the private image data from the explanations provided by the models and surrogate models if the model is non-explainable. An application of saliency methods can be noticed in the work of Moayeri et al. [185]. They experimented on a dataset derived from the ImageNet dataset, in which segmentation masks and informative attributes were collected. Usually, models are more sensitive to background as compared to foreground. Saliency methods help to identify the features that affect the background sensitivity as well as helps in aligning saliency maps with the foreground. This helps in comparing feature saliency with ground truth localization of attributes. Simulations of realistic virtual humans use saliency methods to model visual attention. An approach used to aggregate a saliency score from user defined parameters of objects and characters in an agent’s view has been defined in the paper by Kremer et al. [186]. This score helps in providing a 2D saliency map, which is modeled with the help of attention field to incorporate 3D information as well as the character’s state of awareness. A diverse range of agents can be modeled using this approach.

6. Applications of XAI Technologies

Explainable AI provides us with tools for dealing with artificial intelligence challenges. Here are some examples of how XAI has been applied to different capabilities.

6.1. Human-Assisted AI for Decision-Making in Different Domains

There are many domains in which XAI has been applied to improve diagnostics and human decision-making tools, including healthcare, manufacturing, banking, education, insurance, autonomous driving, and so on [187,188,189]. Human communication, understanding, and learning depend on explaining decisions, and people naturally provide graphical and textual explanations [190]. Deep learning models must demonstrate their decisions fluently in visual and textual formats [190,191]. The underlying mathematical scaffolding in deep machine learning models can be difficult for mathematicians to fully grasp [190]. By explaining AI, users can gain insight into the “why” behind model predictions, which will help them understand better, trust the model, and recognize and correct any incorrect predictions [188,189,191]. Human-AI interactions have previously been studied based on interpretability, trustworthiness, and usability. It has become clear that AI is a growing and ever-evolving industry that has shown dominance in today’s society. However, people still need to question whether AI is beneficial and can outperform humans. Here is a list of examples of how XAI helps with decision-making.

• AI-assisted Human Decision-Making Tools: Based on the inclusion criteria described in Frutos et al. [192], there are 129 papers researching different ways to further develop and integrate specific AI algorithms into decision-making and machine learning. Lysaght et al. [193] cover the effectiveness of an AI-assisted Clinical Decision Support System (CDSS), which provides clinicians with specific diagnoses and predicts the patient’s treatment course. In addition, its implementation is encountered in the military industry. Rasch et al. [194] demonstrate that AI can produce effective decision-making by constructing effective battle plans. The Course of Action Display and Elaboration Tool (CADET) automatically generates detailed plans, which saves a tremendous amount of time spent on these same detailed plans if they were manually created. However, they are still some lingering doubts about its effectiveness compared to humans; Zhang et al. [195] Experiment 2 shows that AI had an accuracy of 75% while a human’s accuracy was 63%. This revelation contradicts the notion that if the model explained how it based its decisions, this would improve people’s trust in AI. On the other hand, Experiment 1 showed that human willingness to rely on AI predictions increases if the user is provided the AI’s confidence score. Similarly, Ref. [196] indicated that humans positively associate with Automated Decision Making (ADM) regarding general and domain-specific knowledge. In addition, they also found that ADM was similar or seen on par with human evaluations. However, the research of Ref. [197] indicated that creating a tool that focuses on the collaboration of humans and AI produces favorable outcomes. They presented an 0.5% error rate when combining inputs from AI and pathologists, demonstrating human-AI coordination’s superiority. In addition to human-AI collaboration, Karacapilidis et al. [198] proposed a Group Decision Support System (GDSS); this computer-mediated system strengthened human-human coordination by including an unbiased mediator, whose goal is to maintain the group’s objective.
• XAI for Diagnostics Tools: Machine learning (ML) cannot explain its behavior. XAI tools can aid us in understanding why ML makes certain decisions by providing explainable AI (XAI) tools. Researchers [189,199] have experimented on how real users would react with realistic explanations generated from a model built over a real dataset. The authors conclude that XAI should support abductive and hypothetico-deductive reasoning (H-D), so hypotheses can be narrowed down. Madhikermi et al. [200] incorporate Local Interpretable Model-Agnostic Explanation (LIME) to improve interpretability in their fault detector models, which aided in the justification of decisions made by the model, neural networks, and support vector machine (SVM). With LIME, the neural network achieved an accuracy score of 97% and an SVM score of 0.96. In addition, in the case of fault diagnosis, Brito et al. [201] used XAI with feature importance ranking. Not only were they able to understand the model, but it provided further understanding of how the methodology gives relevant information and root cause analysis. Similarly, XAI is implemented in the health industry [202,203,204,205]. El et al. [202] developed an explainable machine-learning model based on a random forest classifier to predict Alzheimer’s disease. The model achieved high performance in each layer using the Shapley Additive Explanation (SHAP) XAI tool. In addition, Ye et al. [203] developed a classifier that can distinguish the Covid-19 virus from CT scans, highlighting XAI enhancement to the classifier’s performance. Lastly, Jo et al. [204] built an explainable deep learning model (DLM) with the ability to detect atrial fibrillation (AF). Validated on the PTB-XL Chapman and PhysioNet ECG datasets, they concluded that the explainable DLM accurately detected AF in diverse formats. In the case of Convolutional Neural Networks (CNN), Chen et al. [206] implemented Gradient Class Activation Mapping (Grad-CAM) to assist CNN models. This provided explanations that aided the users in understanding that the high-frequency band was the most significant feature of their model. In addition, diagnostic tools are used in the chemical industry; specifically, McClary et al. [207] developed a tool to assist organic chemistry students in identifying alternative conceptions related to acid strengths.
• XAI for Autonomous Safety-critical Human Machine Interaction Systems: Neogi et al. [208] illustrate how Human-machine interaction (HMI) can be included to explain the behavior of learning-enabled increasingly autonomous agents to the pilot. This effort also investigated the types of HMI that need to be included for the human to better understand what is being learned by the learning-enabled IAS. Finally, in this effort, the outcomes of learning were formally verified. The approach was demonstrated by designing a learning-enabled, increasingly autonomous agent in a cognitive architecture, Soar. The agent includes symbolic decision logic with numeric decision preferences that are tuned by reinforcement learning to produce post-learning decision knowledge. The agent is then automatically translated into nuXmv, a model checker, and properties are verified over the agent. AI-guided principles are fundamental in designing the HMI and formally verifying the design so that the autonomous agent can be trusted. A formal method-based approach also guides the explanation process, as the behavior or learning outcomes must be explicitly represented to verify [209,210].

6.2. Model Comparison

Despite the limitless potential AI offers, it is difficult to determine how machine learning, especially deep learning algorithms, settle for a particular decision in the first place among so many hyper-parameters and model alternatives [203,211,212]. XAI helps to choose the best option for solving a problem by model comparison, besides the different performance metrics that data scientists use [213,214]. Many visualizations and XAI tools perform model comparison and seek to answer questions such as (1) Which model is the best while solving a particular problem with machine learning? (2) What are the similarities and differences between the two models? [215]. Arendt et al. [215] classify visualization and XAI-based model-comparison tasks into three main categories:

• Performance-oriented visualization and comparison;
• Concept-oriented performance and visualization analysis;
• Time-oriented performance and visualization analysis.

The performance-oriented comparison allows the users to compare different models based on performance metrics such as accuracy, F-score, etc., on a global or instance level [215,216]. The concept-oriented model comparisons examine the semantic difference between models and how they interact with data [215,217]. Time-oriented performance analysis, through XAI techniques, helps the users understand if a model is improving, deteriorating, or getting stuck through iterations or neural network layers during training over time [112,215,218,219]. Time-oriented performance analysis also helps fine-tune the model discussed in the next point.

6.3. Fine-Tuning and Training AI Models

Deep learning models are highly black-box in nature. They go through a wide range of hyper-parameter tuning until the data scientists are satisfied with the prediction results [220]. These tuning steps consist of pruning and quantization such as structured pruning, unstructured pruning, CNN filter pruning, weights pruning, mixed precision integer quantization, and weight sharing quantization, for optimizing memory as well as latency [220,221,222,223,224]. With transparent and intrinsic explanations, we can observe how a model learns over time, iterations [215] or layers, and figure out if the model is doing better, converging, or not learning at all [224]. These observations help us find the best-fitted model with refined tuning [225,226].

6.4. Bridging the Gap between Data Scientists and Non-Data Science Experts

There are several disciplines within data science, but the core objectives are to extract insights and knowledge from data using generalizable methods [227,228,229]. Organizations need help to recruit enough data scientists since employees with a background in software engineering or business analytics typically transfer to data scientist roles when data science teams are formed [227,228]. Moreover, prime decision-makers or domain experts in an organization, such as CFOs or CEOs, usually do not come from a background in data science or computer science; therefore, AI systems must communicate between non-data scientists and data science experts [230,231]. Domain experts and data scientists talk through Interactive data models and XAI [231]. Below we provide examples of model-agnostic XAI tools, illustrating how they bridge the gap between experts and novices in data science. As we proceed through the categorization section of this paper, we will explain each concept in more detail.

• Partial Dependencies Plot (PDP) [232], Individual Conditional Expectation (ICE) [62] and Accumulated Local Effects (ALE) [233] plots can help to visualize the effects of the features on the final prediction. These XAI visualization tools are intuitive and do not require prior data science knowledge. Basic knowledge of the data or domain is enough to understand the concepts;
• Permutation Feature Importance (PFI) [69], and SHapley values [75] help rank the most critical features in the black-box model;
• Local and global surrogate models [78] help us understand the black-box model’s local behavior without changing anything and displace the black-box model with a simpler and more interpretable one;
• A counterfactual [25] or anchor [124] XAI explains how sensitive the black-box model is to specific instances.

6.5. Trust, Accountability, and Fairness

As AI systems enter the marketplace, they will offer a wide range of benefits both from a convenience perspective and for safety-critical applications such as military and aviation operations [234]. By leveraging XAI, human users can comprehend, appropriately trust, and effectively manage the new generation of artificially intelligent partners [234,235]. By bringing transparency and accountability into AI systems, causal XAI assists people in understanding the decision-making process of AI algorithms, assessing the models, and ensuring that they are not blindly trusted [235,236]. Additionally, a good explanation may not be sufficient to reduce bias and heuristics in human decision-making [1,7,191,237]. AI models are prone to biases due to biases based on age, gender, race, location, and even personal perceptions of the model builders [238]. The performance of AI models can also drift or degrade because training data differs from production data [238,239]. It is crucial that a business continually monitors and manages models, so AI explainability can be promoted while measuring the business impact from such algorithms [191,238].

6.6. Debugging and Assurance

Information theory combined with XAI techniques can help us to see the flow of information inside the black-box models [240]. For example, attention mechanisms and weights allow us to see where and how the black box model focuses and makes decisions [213,241]. Machine learning models or AI technologies are being hugely used in safety-critical systems. As a result, machine-learning models and other safety-related AI technologies must follow standards for explanations in debugging and understanding, regardless of their black-box nature [2,241]. The assurance principles underlying these standards include validating that the system works as intended and verifying that the system meets the exact safety requirements [241].

In

Table 3. XAI use-cases and examples in the AI life-cycle.

	Data Collection and Pre-Processing	Model Building and Training	Evaluation and Decision-Making
State	Before training the model	While training and refining the model	Post-training, after the AI system and model is built- this process can be iterative depending on the feedback from the client(s).
Purpose	Reading into the data, and pre-processing for further analyses	Generating ML model(s) and AI system(s) to meet the clients’ requirement	Insight generation, decision-making, and planning intervention
Tasks Involved	Setting up a data collection protocol and pipeline, data cleaning, formatting, feature engineering (feature selection, calculating mostly correlated features and variance, etc.)	Development of machine learning models by tuning hyper-parameters to fit the data and problem, database setup, and AI GUI/system development for presenting results and communicating with clients	Generating reports based on evaluation metrics, visualization, and conducting user studies with clients and domain experts for feedback, etc.
Primary Stakeholders Involved	Primarily all stakeholders involved in the AI life cycle—clients, domain experts, software developers, and data scientists: It establishes a communication channel between the vendor(s) and the client(s) for defining the requirements.	Software developers and data scientists: This stage requires medium to advanced data science knowledge to develop machine learning models. If the objective is to build a system or tool, data scientists may collaborate with software developers or make it themselves if they have prior experience in software development.	Primarily all the stakeholders are involved through agile methodology-incremental feedback and fine-tuning models and systems until the final product is ready.
Use-case(s)	Bridging the Gap Between Data Scientists and Non-data Science Experts	Fine-tuning, model comparisons to find the best model, Debugging and Assurance	Human-assisted AI for decision-making in different domains, Fine-tuning and Training AI Models, Bridging the Gap Between Data Scientists and Non-data Science Experts, Trust, Accountability, and Fairness
Example XAI tools and Techniques	Different visualization plots such as pair correlation, Individual Conditional, Partial Dependence, Accumulated Local Effect plots, etc.	Some categories of Intrinsic, Ad-hoc and global and local post-hoc model-specific and model-agnostic explanations such as DL representation explanations, global and local attention models, joint prediction and explanations, perturbation-based explanations, kernel SHAP, layer-wise relevance propagation, adversarial over transformations, etc.	Some categories of global and local model-specific and model-agnostic explanations such as Global Shapely values, feature weights in GAM, Anchor and LIME; feature importance-based and saliency visualization-based explanations, etc.

, we demonstrate the state, purpose, tasks, and primary stakeholders involved in every state of a standard pipeline of AI systems—from data gathering or processing to decision-making and intervention. In the table, rather than citing state-of-the-art XAI algorithms and tools, we mentioned some example categories of XAI tools or techniques (see Figure 7) corresponding to that particular state of the AI pipeline.

7. Future Direction and Conclusions

All in all, black-box models have recently been growing fast, and unfortunately, researchers and practitioners focus on increasing accuracy and decreasing errors rather than the explainability of AI models. However, as our Bibliometric analysis (Appendix A) has shown, there is a noticeable trend toward explainability. Besides, interpretability techniques would not only be valuable tools to make decision-making black boxes transparent and unbiased but also give machine learning models the ability to explain or present their behaviors understandably to humans. Furthermore, interpretability tools can also help debug misprediction cases, improve and verify the machine learning models, and learn new insights by inspecting their decision boundaries. Explainability may be considered optional today, but with the growth of AI decision-making systems, explainability will undoubtedly be required to address transparency in all procedures of designing, validating, and implementing black-box models in the future. In this paper, we analyzed the effectiveness and evolution of XAI research with statistical analysis of the Scopus publication dataset. Furthermore, we created a repository of XAI applications categorized and influenced by influential XAI researchers. We discussed potential tools and ongoing research that can help explain black boxes in Natural Language Processing and Human-centered Artificial Intelligence in more detail. For machine learning approaches to be implemented for safety-critical applications, there still needs to be assurance and explainability methods that need to be developed. Because without a proper explanation of the operations or classification, if the autonomy executes an action, pilots or ground operators might make risky decisions, as exhibited by the Tesla driver that ended in an accident [242]. So, further work needs to be done in explainable AI to provide the correct explanation for the pilot or user to trust the AI.