Hardware Implementation of a Softmax-Like Function for Deep Learning ^†

Abstract

In this paper a simplified hardware implementation of a CNN softmax-like layer is proposed. Initially the softmax activation function is analyzed in terms of required numerical accuracy and certain optimizations are proposed. A proposed adaptable hardware architecture is evaluated in terms of the introduced error due to the proposed softmax-like function. The proposed architecture can be adopted to the accuracy required by the application by retaining or eliminating certain terms of the approximation thus allowing to explore accuracy for complexity trade-offs. Furthermore, the proposed circuits are synthesized in a 90 nm 1.0 V CMOS standard-cell library using Synopsys Design Compiler. Comparisons reveal that significant reduction is achieved in area × delay and power × delay products for certain cases, respectively, over prior art. Area and power savings are achieved with respect to performance and accuracy.

1. Introduction

Deep neural networks (DNN) have emerged as a means to tackle complex problems such as image classification and speech recognition. The success of DNNs is attributed to the big data availability, the easy access to enormous computational power and the introduction of novel algorithms that have substantially improved the effectiveness of the training and inference [1]. A DNN is defined as a neural network (NN) which contains more than one hidden layer. In the literature, a graph is used to represent a DNN, with a set of nodes in each layer, as shown in Figure 1. The nodes at each layer are connected to the nodes of the subsequent layer. Each node performs processing including the computation of an activation function [2]. The extremely large number of nodes at each layer impels the training procedure to require extensive computational resources.

A class of DNNs are the convolutional neural networks (CNNs) [2]. CNNs offer high accuracy in computer-vision problems such as face recognition and video processing [3] and have been adopted in many modern applications. A typical CNN consists of several layers, each one of which can be convolutional, pooling, or normalization with the last one to be a non-linear activation function. A common choice for normalization layers is usually the softmax function as shown in Figure 1. To cope with increased computational load, several FPGA accelerators have been proposed and have demonstrated that convolutional layers exhibit the largest hardware complexity in a CNN [4,5,6,7,8,9,10,11,12,13,14,15]. In addition to CNNs, hardware accelerators for RNNs and LSTMs have also been investigated [16,17,18]. In order to implement a CNN in hardware, the softmax layer should also be implemented with low complexity. Furthermore, the hidden layers of a DNN can use the softmax function when the model is designed to choose one among several different options for some internal variable [2]. In particular, neural turing machines (NTM) [19] and differentiable neural computer (DNC) [20] use softmax layers within the neural network. Moreover, softmax is incorporated in attention mechanisms, an application of which is machine translation [21]. Furthermore, both hardware [22,23,24,25,26,27,28] and memory-optimized software [29,30] implementations of the softmax function have been proposed. This paper, extending previous work published in MOCAST2019 [31], proposes a simplified architecture for a softmax-like function, the hardware implementation of which is based on a proposed approximation which exploits the statistical structure of the vectors processed by the softmax layers in various CNNs. Compared to the previous work [31], this paper uses a large set of known CNNs, and performs extensive and fair experiments to study the impact of the applied optimizations in terms of the achieved accuracy. Moreover the architecture in [31] is further elaborated and generalized by taking into account the requirements of the targeted application. Finally, the proposed architecture is compared with various softmax hardware implementations. In order for the softmax-like function to be implemented efficiently in hardware, the approximation requirements are relaxed.

The remainder of the paper is organized as follows. Section 2 revisits the softmax activation function. Section 3 describes the proposed algorithm and Section 4 offers a quantitative analysis of the proposed architecture. Section 5 discusses the hardware complexity of the proposed scheme based on synthesis results. Finally, conclusions are summarized in Section 6.

2. Softmax Layer Review

CNNs consist of a number of stages each of which contains several layers. The final layer is usually fully-connected using ReLU as an activation function and drives a softmax layer before the final output of the CNN. The classification performed by the CNNs is accomplished at the final layer of the network. In particular, for a CNN which consists of ${\displaystyle i+1}$ layers, the softmax function is used to transform the real values generated by the ${\displaystyle i}$th CNN layer to possibilities, according to

$$f_j\left(z\right)=\frac{e^{z_j}}{{\displaystyle \sum _{k=1}^n}{e}^{z_k}},$$

(1)

where ${\displaystyle z}$ is an arbitrary vector with real values ${\displaystyle z_j}$, ${\displaystyle j=1,\dots ,n}$, generated at the ${\displaystyle i}$th layer of the CNN and ${\displaystyle n}$ is the size of the vector. The (${\displaystyle i+1}$)st layer is called the softmax layer. By applying the logarithm function to both sides of (1), it follows that

$$\begin{array}{ccc}\hfill log\left(f_j\left(z\right)\right)& =& log\left(\frac{e^{z_j}}{{\displaystyle \sum _{k=1}^n}{e}^{z_k}}\right)\hfill \end{array}$$

(2)

$$\begin{array}{ccc}& =& log\left(e^{z_j}\right)-log\left(\sum _{k=1}^n{e}^{z_k}\right)\hfill \end{array}$$

(3)

$$\begin{array}{ccc}& =& z_j-log\left(\sum _{k=1}^n{e}^{z_k}\right).\hfill \end{array}$$

(4)

In (4), the term ${\displaystyle log\left(\sum _{k=1}^n{e}^{z_k}\right)}$ is computed as

$$\begin{array}{ccc}\hfill log\left(\sum _{k=1}^n{e}^{z_k}\right)& =& log\left(\sum _{k=1}^n\frac{e^m}{e^m}{e}^{z_k}\right)\hfill \end{array}$$

(5)

$$\begin{array}{ccc}& =& log\left(e^m\sum _{k=1}^n\frac{1}{e^m}{e}^{z_k}\right)\hfill \end{array}$$

(6)

$$\begin{array}{ccc}& =& log{e}^m+log\left(\sum _{k=1}^n{e}^{-m}{e}^{z_k}\right)\hfill \end{array}$$

(7)

$$\begin{array}{ccc}& =& m+log\left(\sum _{k=1}^n{e}^{z_k-m}\right),\hfill \end{array}$$

(8)

where ${\displaystyle m=\underset{k}{max}\left(z_k\right)}$. From (4) and (8) it follows that

$$\begin{array}{ccc}\hfill log\left(f_j(z)\right)& =& z_j-\left(m+log(\sum _{k=1}^n{e}^{z_k-m})\right)\hfill \end{array}$$

(9)

$$\begin{array}{ccc}& =& z_j-\left(m+log(\sum _{k=1}^n{e}^{z_k-m}-1+1)\right)\hfill \end{array}$$

(10)

$$\begin{array}{ccc}& =& z_j-\left(m+log(Q+1)\right),\hfill \end{array}$$

(11)

where

$$Q=\sum _{k=1}^n{e}^{z_k-m}-1=\sum _{\begin{array}{c}k=1\\ z_k\ne max\end{array}}^n{e}^{z_k-m}+1-1=\sum _{\begin{array}{c}k=1\\ z_k\ne max\end{array}}^n{e}^{z_k-m}.$$

(12)

Due to the definition of ${\displaystyle m}$, it holds that

$$\begin{array}{ccc}\hfill z_j& \le & m\Rightarrow z_j-m\le 0\Rightarrow e^{z_j-m}\le 1\Rightarrow \hfill \end{array}$$

(13)

$$\begin{array}{ccc}\hfill \sum _{k=1}^n{e}^{z_k-m}& \le & n\Rightarrow \sum _{\begin{array}{c}k=1\\ z_k\ne max\end{array}}^n{e}^{z_k-m}+1\le n\Rightarrow \hfill \end{array}$$

(14)

$$\begin{array}{ccc}\hfill Q+1& \le & n\Rightarrow \frac{Q}{n-1}\le 1\Rightarrow Q^{{}^{\prime }}\le 1,\hfill \end{array}$$

(15)

where ${\displaystyle Q^{{}^{\prime }}=\frac{Q}{n-1}}$. Expressing ${\displaystyle Q}$ in terms of ${\displaystyle Q^{{}^{\prime }}}$, (11) becomes

$$log\left(f_j\left(z\right)\right)=z_j-m-log\left((n-1)Q^{{}^{\prime }}+1\right).$$

(16)

The next section presents the proposed simplifications for (16) and the derivative architecture for the softmax-like hardware implementation.

3. Proposed Softmax Architecture

Equation (15) involves the distance of the maximum component from the remainder of the components of a vector. As ${\displaystyle Q^{{}^{\prime }}\approx 0}$, the differences in ${\displaystyle z_i}$s increase and ${\displaystyle z_j\ll m}$. On the contrary, as ${\displaystyle Q^{{}^{\prime }}\to 1}$ the differences in ${\displaystyle z_i}$s are eliminated. Based on this observation, a simplifying approximation can be obtained, as follows. The third term in the right hand side of (16), ${\displaystyle log\left((n-1)Q^{{}^{\prime }}+1\right)}$, can be roughly approximated by 0. Hence, (16) is approximated by

$$log\left(\widehat{f_j}\left(z\right)\right)\simeq z_j-m.$$

(17)

Furthermore, this simplification substantially reduces hardware complexity as described below. From (17), it follows that

$$\widehat{f_j}\left(z\right)=e^{z_j-\underset{k}{max}\left(z_k\right)}.$$

(18)

Theorem 1.

Let ${\displaystyle \underset{z}{argmax}\left(f_j\left(z\right)\right)=q}$ and ${\displaystyle \underset{z}{argmax}\left(\widehat{f_j}\left(z\right)\right)=r}$ be the decisions obtained by (1) and (18), respectively. Then ${\displaystyle q=r}$.

Proof. 

Due to the softmax definition, it holds that

$$\begin{array}{ccc}\hfill max{f}_j\left(z\right)& =& max\left(\frac{e^{z_q}}{{\displaystyle \sum _{k=1}^n}{e}^{z_k}}\right)\Rightarrow \hfill \end{array}$$

(19)

$$\begin{array}{ccc}\hfill z_q& =& \underset{k}{max}\left(z_k\right).\hfill \end{array}$$

(20)

For the case of the proposed function (18), it holds that

$$\begin{array}{ccc}\hfill max\widehat{f_j}\left(z\right)& =& max\left(e^{z_r-\underset{k}{max}\left(z_k\right)}\right)\Rightarrow \hfill \end{array}$$

(21)

$$\begin{array}{ccc}\hfill z_r& =& \underset{k}{max}\left(z_k\right).\hfill \end{array}$$

(22)

From (20) and (22), it is derived that ${\displaystyle z_q=z_r}$. Hence, ${\displaystyle \underset{z}{argmax}\left(f_q\left(z\right)\right)=\underset{z}{argmax}\left(f_r\left(z\right)\right)\Rightarrow q=r}$.  □

Corollary 1.

It holds that ${\displaystyle \frac{\widehat{f_j}\left(z\right)}{f_j\left(z\right)}=\sum _{k=1}^n{e}^{z_k-m}.}$

Proof. 

Proof is trivial and is omitted. □

Theorem 1 states that the proposed softmax-like function and the actual softmax function always derive the same decisions. The proposed softmax-like approximation is based on the idea that the softmax function is used during training to target an output ${\displaystyle y}$ by using maximum-log likelihood [2]. Hence, if the correct answer has already the maximum input value to the softmax function then ${\displaystyle log\left((n-1)Q^{{}^{\prime }}+1\right)\simeq 0}$ will roughly alter the output decision due to the exponent function used in term ${\displaystyle Q^{{}^{\prime }}}$. In general, ${\displaystyle \sum _j\widehat{f_j}\left(z\right)>1}$, since the sequence ${\displaystyle \widehat{f_j}\left(z\right)}$ cannot be denoted as a probability density function. For models where the normalization function is required to be a pdf, a modified approach can be followed, as detailed below.

According to the second approach, from (11) and (12) it follows:

$$\begin{array}{ccc}\hfill log\left(\widehat{f_j}\left(z\right)\right)& =& z_j-m-log(\sum _{\begin{array}{c}k=1\\ z_k\ne max\end{array}}^n{e}^{z_k-m}+1)\hfill \end{array}$$

(23)

$$\begin{array}{ccc}& =& z_j-m-log(\sum _{\begin{array}{c}k=1\\ m_k\ne max\end{array}}^p{e}^{m_k-m}+\sum _{\begin{array}{c}k=p+1\\ m_k\ne max\end{array}}^n{e}^{m_k-m}+1)\hfill \end{array}$$

(24)

$$\begin{array}{ccc}& =& z_j-m-log(\sum _{\begin{array}{c}k=1\end{array}}^p{e}^{m_k-m}+\sum _{\begin{array}{c}k=p+1\end{array}}^n{e}^{m_k-m})\hfill \end{array}$$

(25)

$$\begin{array}{ccc}& =& z_j-m-log(Q_1+Q_2),\hfill \end{array}$$

(26)

with ${\displaystyle Q_1=\sum _{\begin{array}{c}k=1\end{array}}^p{e}^{m_k-m}}$, where ${\displaystyle M_1=\{m_1,\dots ,m_p\}=\{m_k:k=1,\dots ,p\}}$ are chosen to be the top ${\displaystyle p}$ maximum values of ${\displaystyle z}$. For the quantity ${\displaystyle Q_2}$, it holds ${\displaystyle Q_2=\sum _{\begin{array}{c}k=p+1\end{array}}^n{e}^{m_k-m}}$, with ${\displaystyle M_2=\{m_{p+1},\dots ,m_n\}=\{m_k:k=p+1,\dots ,n\}}$ being the remainder values of the vector ${\displaystyle z}$, i.e., ${\displaystyle M_1\cup M_2=\{z_1,\dots ,z_n\}}$.

A second approximation is performed as

$$\begin{array}{ccc}\hfill log\left(Q_1\right)=log(\sum _{\begin{array}{c}k=1\end{array}}^p{e}^{m_k-m}-1+1)& \approx & \sum _{\begin{array}{c}k=1\end{array}}^p{e}^{m_k-m}-1.\hfill \end{array}$$

(27)

$$\begin{array}{ccc}\hfill Q_2& \approx & 0.\hfill \end{array}$$

(28)

From (26)–(28), it derives that

$$\begin{array}{ccc}\hfill log\left(\widehat{f_j}(z,p)\right)& \approx z_j& -m-\sum _{\begin{array}{c}k=1\end{array}}^p{e}^{m_k-m}+1\hfill \end{array}$$

(29)

$$\widehat{f_j}(z,p)=e^{z_j-m-{\displaystyle \sum _{k=1}^p}{e}^{m_k-m}+1}$$

(30)

Equation (30) uses parameter ${\displaystyle p}$ which defines the number of additional terms used. By properly selecting ${\displaystyle p}$, then it holds that ${\displaystyle \sum _j\widehat{f_j}(z,p)\simeq 1}$ and (30) approximates pdf better than (18). This is derived from the fact that in a real life CNN, the ${\displaystyle p}$ maximum values are those that contribute to the computation of the softmax since all the remainder values are close to zero.

Lemma 1.

It holds ${\displaystyle \widehat{f_j}(z,p)=\widehat{f_j}\left(z\right)}$ when ${\displaystyle p=1.}$

Proof. 

By definition, it holds that when ${\displaystyle p=1}$ then ${\displaystyle m_1=m}$, since the ${\displaystyle p=1}$ maximum value ${\displaystyle m_1}$ is identified as the maximum ${\displaystyle m}$. Hence, by substituting ${\displaystyle p=1}$ in (30), it derives that

$$\begin{array}{ccc}\hfill \widehat{f_j}(z,1)& =& e^{z_j-m-{\displaystyle \sum _{k=1}^1}{e}^{m_k-m}+1}\Rightarrow \hfill \\ \hfill \widehat{f_j}(z,1)& =& e^{z_j-m-e^{m-m}+1}\Rightarrow \hfill \\ \hfill \widehat{f_j}(z,1)& =& e^{z_j-m}\Rightarrow \hfill \\ \hfill \widehat{f_j}(z,1)& \stackrel{\left(18\right)}{=}& \widehat{f_j}\left(z\right)\hfill \end{array}$$

□

From a hardware perspective, (18) and (30) can be performed by the same circuit which implements the exponential function. The contributions of the paper are as follows. Firstly, the quantity ${\displaystyle log\left((n-1)Q^{{}^{\prime }}+1\right)}$ is eliminated from (16), implying that the target application requires decision making. Secondly, further mathematical manipulations are proposed to be applied to (30), in order to approximate the outputs as pdf i.e., probabilities that sum to one. Thirdly, the circuit for the evaluation of ${\displaystyle e^x}$ is simplified, since

$$\begin{array}{ccc}\hfill z_j& \le & m\Rightarrow \hfill \end{array}$$

(31)

$$\begin{array}{ccc}\hfill z_j-m& \le & 0\Rightarrow \hfill \end{array}$$

(32)

$$\begin{array}{ccc}\hfill e^{z_j-m}& \le & 1.\hfill \end{array}$$

(33)

and

$$\begin{array}{ccc}\hfill z_j& \le & m\Rightarrow \hfill \end{array}$$

(34)

$$\begin{array}{ccc}\hfill z_j-m-\sum _{k=1}^p{e}^{m_k-m}+1& <& 0\Rightarrow \hfill \end{array}$$

(35)

$$\begin{array}{ccc}\hfill e^{z_j-m-{\displaystyle \sum _{k=1}^p}{e}^{m_k-m}+1}& <& 1.\hfill \end{array}$$

(36)

Figure 2 depicts the various building blocks of the proposed architecture. More specifically, the proposed architecture is comprised of the block which computes the maximum ${\displaystyle m}$, i.e., ${\displaystyle \underset{k}{m=max\left(z_k\right)}}$. The particular computation is performed by a tree which generates the maximum by comparing the elements by two, as shown in Figure 3. The depicted tree structure generates ${\displaystyle m=\underset{k}{max\left(z_k\right)}}$, ${\displaystyle k=0,1,\dots ,7}$. Notation ${\displaystyle z_{ij}}$ denotes the maximum of ${\displaystyle z_i}$ and ${\displaystyle z_j}$, while ${\displaystyle z_{ijkl}}$ denotes the maximum of ${\displaystyle z_{ij}}$ and ${\displaystyle z_{kl}}$. The same architecture is used to compute the top ${\displaystyle p}$ maximum values of ${\displaystyle z_i}$s. For example, ${\displaystyle Z_{01}}$, ${\displaystyle Z_{21}}$, ${\displaystyle Z_{45}}$ and ${\displaystyle Z_{67}}$ are the top four maximum values and ${\displaystyle \underset{k}{m=max\left(z_k\right)}}$ is the maximum.

Subsequently, ${\displaystyle m}$ is subtracted from all the component values ${\displaystyle z_k}$ as dictated by (17). The subtraction is performed through adders, denoted as ${\displaystyle ⨁}$ in Figure 2, using two’s complement representation for the input negative values ${\displaystyle -m}$. The obtained differences, also represented in two’s complement, are used as inputs to a LUT, which performs the proposed simplified ${\displaystyle e^x}$ operation of (18), to compute the final vector ${\displaystyle \widehat{f_j}\left(z\right)}$ as shown in Figure 2a. Additional ${\displaystyle p}$ terms are added and subsequently each output ${\displaystyle \widehat{f_j}\left(z\right)}$ through (30) generates the final value for the softmax-like layer output as shown in Figure 2b. For the hardware implementation of the ${\displaystyle e^x}$ function, an LUT is adopted the input of which is ${\displaystyle x=z_j-m}$. The LUT size increases on the larger range of ${\displaystyle e^x}$. Our proposed hardware implementation is simpler than other exponential implementations which propose CORDIC transformations [32], use floating-point representation [33], or LUTs [34]. In (33), the ${\displaystyle e^x}$ values are restricted to the range ${\displaystyle \left(0,1\right]}$ and the derived LUT size significantly diminishes and leads to simplified hardware implementation. Furthermore, no conversion from the logarithmic to the linear domain is required, since ${\displaystyle f_j\left(z\right)}$ represents the final classification layer.

The next section quantitatively investigates the validity and usefulness of employing ${\displaystyle f_j\left(z\right)}$, in terms of the approximation error.

4. Quantitative Analysis of Introduced Error

This section quantitatively verifies the applicability of the approximation introduced in Section 2, for certain applications, by means of a series of Examples.

In order to quantify the error introduced by the proposed architecture, the mean square error (MSE) is evaluated as

$$\mathrm{MSE}=\frac{1}{n}\sum {\left(f_j\left(z\right)-\widehat{f_j}\left(z\right)\right)}^2,$$

(37)

where ${\displaystyle \widehat{f_j}\left(z\right)}$ and ${\displaystyle f_j\left(z\right)}$ are the expected and the actually evaluated softmax output, respectively.

As an illustrative example, denoted as Example 1, Figure 4 depicts the histograms of component values in test vectors used as inputs to the proposed architecture, selected to have specific properties detailed below. The corresponding parameters are evaluated by using the proposed architecture for the case of a 10-bit fixed-point representation, where 5 bits are used for the integer and 5 bits are allocated to the fractional part. More specifically, the vector in Figure 4a contains ${\displaystyle n=30}$ values, for which it holds that ${\displaystyle z_j=5,j=1,\dots ,11}$, ${\displaystyle z_j=4.99,j=12,\dots ,22}$ and ${\displaystyle z_j=5,j=23,\dots ,30}$. For this case the softmax values obtained by (1) are

$$f_j\left(z\right)=\left\{\begin{array}{cc}0.0343,\hfill & j=1,\dots ,11\hfill \\ 0.0314,\hfill & j=12,\dots ,22\hfill \\ 0.0347,\hfill & j=23,\dots ,30.\hfill \end{array}\right.$$

(38)

From a CNN perspective, the softmax layer output generates similar values, where possibilities are all around 3%, and hence classification or decision cannot be made with high confidence. By using (18), the modified softmax values are

$$\widehat{f_j}\left(z\right)=\left\{\begin{array}{cc}0.9844,\hfill & j=1,\dots ,11\hfill \\ 0.8906,\hfill & j=12,\dots ,22\hfill \\ 1,\hfill & j=23,\dots ,30.\hfill \end{array}\right.$$

(39)

The statistical structure of the vector is characterized by the quantity ${\displaystyle Q^{{}^{\prime }}=0.9946}$ of (15). The estimated ${\displaystyle \mathrm{MSE}=0.8502}$ dictates that the particular vector is not suitable as an alternative to softmax input in terms of CNN performance, i.e., the obtained classification is performed with low confidence. Hence although the proposed approximation in (18) demonstrates large differences when compared to (1), neither is applicable in CNN terms.

Consider the following Example 2. The component values for vector ${\displaystyle z}$ in Example 2 are

$$z_j=\left\{\begin{array}{cc}3,\hfill & j=1\hfill \\ 6,\hfill & j=2\hfill \\ 4,\hfill & j=3\hfill \\ 2,\hfill & j=4\hfill \\ <1,\hfill & j=5,\cdots ,30,\hfill \end{array}\right.$$

(40)

the histogram of which is shown in Figure 4b. In this case, the statistical structure of the vector demonstrates ${\displaystyle Q^{{}^{\prime }}=0.0449}$ and ${\displaystyle \mathrm{MSE}=0.0018}$. The feature of vector ${\displaystyle z}$ in Example 2 is that it contains three large different component values close to each other, namely ${\displaystyle z_1=3,z_2=6,z_3=4,z_4=2}$ and all other components are smaller than 1. The softmax output in (1) for the values in the particular ${\displaystyle z}$ are

$$f_j\left(z\right)=\left\{\begin{array}{cc}0.0382,\hfill & j=1\hfill \\ 0.7675,\hfill & j=2\hfill \\ 0.1039,\hfill & j=3\hfill \\ 0.0141,\hfill & j=4\hfill \\ <0.005,\hfill & j=5,\dots ,30.\hfill \end{array}\right.$$

(41)

By using the proposed approximation (18), the obtained modified softmax values are

$$\widehat{f_j}\left(z\right)=\left\{\begin{array}{cc}0.0469,\hfill & j=1\hfill \\ 1,\hfill & j=2\hfill \\ 0.1250,\hfill & j=3\hfill \\ 0.0156,\hfill & j=4\hfill \\ 0,\hfill & j=5,\dots ,30.\hfill \end{array}\right.$$

(42)

Equations (41) and (42) show that the proposed architecture chooses component ${\displaystyle z_2}$ with value 1 while the actual probability is 0.7675. This means that the introduced error of ${\displaystyle \mathrm{MSE}=0.0018}$ can be negligible depending on the application, dictated by ${\displaystyle Q^{{}^{\prime }}=0.0449\ll 1}$.

In the following, tests using vectors obtained from real CNN applications are considered. More specifically, in an example shown in Figure 5, the vectors are obtained from image and digit classification applications. In particular, Figure 5a,b depict the values used as input to the final softmax layer, generated during a single inference for a VGG-16 imagenet image classification network for 1000 classes and a custom net for MNIST digit classification for 10 classes. Quantity ${\displaystyle Q^{{}^{\prime }}}$ can be used to determine whether the proposed architecture is appropriate for application on vector ${\displaystyle z}$ before evaluating the MSE. It is noted that ${\displaystyle \mathrm{MSE}}$ for the example of Figure 5a and ${\displaystyle \mathrm{MSE}}$ for the example of Figure 5b are of the orders of ${\displaystyle {10}^{-13}}$ and ${\displaystyle {10}^{-5}}$, respectively which renders them as negligible.

Subsequently, the proposed method is applied on the ResNet-50 [35], VGG-16, VGG-19 [36], InceptionV3 [37] and MobileNetV2 [38] CNNs, for 1000 classes with 10000 inferences of a custom image data set. In particular, for the case of ResNet-50, Figure 6a,b depict the histograms of the MSE and ${\displaystyle Q^{{}^{\prime }}}$ values, respectively. More specifically, Figure 6a demonstrates that the MSE values are of magnitude ${\displaystyle {10}^{-3}}$ with 8828 of the values be in the interval${\displaystyle [2.18\times {10}^{-28},8.16\times {10}^{-4}]}$. Furthermore, Figure 6b shows that the ${\displaystyle Q^{{}^{\prime }}}$ values are of magnitude ${\displaystyle {10}^{-2}}$ with 9096 of them be in the interval ${\displaystyle [0,0.00270]}$. Furthermore, Table 1a–e depict actual softmax and the proposed method softmax-like values obtained by executing inference on the CNN models for six custom images. The values are sorted from left to right with the maximum value on the left side. Furthermore, the inferences ${\displaystyle a\dots f}$ and ${\displaystyle a^{{}^{\prime }}\dots f^{{}^{\prime }}}$ denote the same custom image as input to the model. More specifically, Table 1a demonstrates values from six inferences for the ResNet-50 model. It is shown that for the case of inference ${\displaystyle a}$ the maximum obtained values are ${\displaystyle f_1\left(z\right)=0.75371140}$ and ${\displaystyle \widehat{f_1}\left(z\right)=1}$, for ${\displaystyle f_j\left(z\right)}$ and ${\displaystyle \widehat{f_j}\left(z\right)}$, respectively. Other values are ${\displaystyle f_2\left(z\right)=0.027212996}$, ${\displaystyle f_3\left(z\right)=0.018001331}$, ${\displaystyle f_4\left(z\right)=0.014599146}$, ${\displaystyle f_5\left(z\right)=0.014546203}$ and ${\displaystyle \widehat{f_2}\left(z\right)=0.03515625}$, ${\displaystyle \widehat{f_3}\left(z\right)=0.0234375}$, ${\displaystyle \widehat{f_4}\left(z\right)=0.0185546875}$, ${\displaystyle \widehat{f_1}\left(z\right)=0.0185546875}$, respectively. Hence, as shown by colorally 1, the maximum takes the value ‘1’ and the remainder of the values follow the values obtained by the actual softmax function. Similar analysis can be obtained for all the inferences ${\displaystyle a\dots f}$ and ${\displaystyle a^{{}^{\prime }}\dots f^{{}^{\prime }}}$ for each one of the CNNs. Furthermore, the same class is outputted from the CNN in both cases for each inference.

For the case of VGG-16, Figure 7a depicts the histogram of the MSE values. It is shown that the values are of magnitude ${\displaystyle {10}^{-3}}$ and 8616 are in the interval ${\displaystyle [1.12\times {10}^{-34},7.26\times {10}^{-4}]}$. Figure 7b demonstrates histogram for the ${\displaystyle Q^{{}^{\prime }}}$ values with magnitude of ${\displaystyle {10}^{-2}}$ and more than 8978 values are in the interval ${\displaystyle [0,0.00247]}$. Table 1b demonstrates that in the case of the ${\displaystyle e}$ and ${\displaystyle e^{{}^{\prime }}}$ inference the top values are 0.51182884 and 1, respectively. The second top values are 0.18920843 and 0.369140625, respectively. In this case, the decision is made with a confidence of 0.51182884 for the actual softmax value and 1 for the softmax-like value. Furthermore, the second top value is 0.369140625 which is not negligible when compared to 1 and hence denotes that the selected class is of low confidence. The same conclusion derives for the case of the actual softmax value. Furthermore, for the case of ${\displaystyle a-f}$ and ${\displaystyle a^{*}-f^{*}}$ inferences, the values obtained by Figure 8b are close to the actual pdf softmax outputs. In particular, for the ${\displaystyle d}$ and ${\displaystyle d^{*}}$ cases, the top 5 values are 0.747070313, 0.10941570, 0.065981410, 0.018329350, 0.012467328 and 0.747070313, 0.110351563, 0.06640625, 0.017578125, 0.01171875, respectively. It is shown that values are similar. Hence, depending the application, an alternative architecture, as shown in Figure 2b can be used to generate pdf softmax values as outputs.

Moreover, Figure 8a,b depict graphically the values for the actual softmax and the proposed softmax-like output for inferences A and B, respectively, for the case of the VGG-16 CNN for the output classification. Furthermore, Figure 8c–f depict values for the architectures in Figure 2a,b, respectively. It is shown that in the case of Figure 2a, the values demonstrate a similar structure. In case of Figure 2b, values are similar to the actual softmax outputs.

Similar analysis can be performed for the case of VGG-19. In particular, Figure 9a demonstrates that MSE is of magnitude ${\displaystyle {10}^{-3}}$ and 8351 of which are in the interval ${\displaystyle [2.49\times {10}^{-34},6.91\times {10}^{-4}]}$. In Figure 9b 8380 values are in the interval ${\displaystyle [0,0.00192]}$. For the case of InceptionV3, histograms in Figure 10a,b demonstrate MSE and ${\displaystyle Q^{{}^{\prime }}}$ values the 9194 and 9463 of which are in the intervals ${\displaystyle [2.62\times {10}^{-25},7.08\times {10}^{-4}]}$ and ${\displaystyle [0,0.003]}$, respectively. For the MobileNetV2 network, Figure 11a,b demonstrate MSE and ${\displaystyle Q^{{}^{\prime }}}$ values the 8990 and 9103 of which are in the intervals ${\displaystyle [2.48\times {10}^{-25},1.05\times {10}^{-3}]}$ and ${\displaystyle [1.55\times {10}^{-7},0.004]}$, respectively. Furthermore, Table 1c–e derive similar conclusions as in the case of VGG-16.

In general, in all cases identical output decisions are obtained for the actual softmax and the softmax-like output layer for each one of the CNNs.

Considering the impact of the data wordlegth representation, let ${\displaystyle (l,k)}$ denote the fixed-point representation of a number with ${\displaystyle l}$ integral and ${\displaystyle k}$ fractional bits. Figure 12a,b depict histograms for the MSE values obtained for the case of 1000 inferences by the VGG-16 CNN. It is shown that the case ${\displaystyle w=(6,2)}$ demonstrates the smaller MSE values. The reason for this is that the maximum value of the inputs in the softmax layer is 56 for all the 1000 inferences, and hence the value of 6 for the integral part is sufficient.

Summarizing, it is shown that the proposed architecture suits well for the final stage of a CNN network as an alternative to implementing the softmax layer stage, since the MSE is negligible. Next, the proposed architecture is implemented in hardware and compared with published counterparts.

5. Hardware Implementation Results

This section describes implementation results obtained by synthesizing the proposed architecture outlined in Figure 2. Among several authors reporting results on CNN accelerators, [22,23,24] have recently published works focusing on hardware implementation of the softmax function. In particular, in [23], a study based on stochastic computation is presented. Geng et al. provide a framework for the design and optimization of softmax implementation in hardware [26]. They also discuss operand bit-width minimization, taking into account application accuracy constraints. Du et al. propose a hardware architecture that derives the softmax function without a divider [25]. The appproach relies on an equivalent softmax expression which requires natural logarithms and exponentials. They provide detailed evaluation of the impact of the particular implementation on several benchmarks. Li et al. describe a 16-bit fixed-point hardware implementation of the softmax function [27]. They use a combination of look-up tables and multi-segment linear approximations for the approximation of exponentials and a radix-4 Booth–Wallace-based 6-stage pipeline multiplier and modified shift-compare divider.

In [24], the architecture demonstrates LUT-based computations that add complexity and exhibits 444,858 ${\displaystyle \mathsf{\mu }}$m${\displaystyle {}^2}$ area complexity by using 65-nm standard-cell library. For the same library the architecture in [25] reports 640,000 ${\displaystyle \mathsf{\mu }}$m${\displaystyle {}^2}$ area complexity with 0.8 ${\displaystyle \mathsf{\mu }}$w power consumption at a 500 MHz clock frequency. The architecture in [28] reports 104,526 ${\displaystyle \mathsf{\mu }}$m${\displaystyle {}^2}$ area complexity with 4.24 ${\displaystyle \mathsf{\mu }}$w power consumption at a 1 GHz clock frequency. The proposed architecture in [26] demonstrates power consumption and area complexity of 1.8 ${\displaystyle \mathsf{\mu }}$w and 3000 ${\displaystyle \mathsf{\mu }}$m${\displaystyle {}^2}$, respectively at a 500 MHz clock frequency with UMC 65 nm standard cell library. In [27], it is reported 3.3 GHz and 34,348 ${\displaystyle \mathsf{\mu }}$m${\displaystyle {}^2}$ frequency and area complexity at 45 nm technology node. Yuan [22] presented an architecture for implementing the softmax layer. Nevertheless there is no discussion of the implementation of the LUTs and there are no synthesis results. Our proposed softmax-like function differs from the actual softmax function due to the approximation of the quantity ${\displaystyle log\left((n-1)Q^{{}^{\prime }}+1\right)}$, as discussed in Section 3. In particular, (18) approximates the softmax output as a decision making application and not as a pdf function. The proposed softmax-like function in (30) approximates outputs as pdf function, depending on the number of ${\displaystyle p}$ terms used. As ${\displaystyle p\to n}$, (30)${\displaystyle \to }$ actual softmax function. The hardware complexity reduction derives from the fact that a limited number, ${\displaystyle p}$, of ${\displaystyle z_i}$s contribute to the computation of the softmax function. Summarizing, we compare both architectures depicted in Figure 2a,b with [22] to quantify the impact of ${\displaystyle p}$ on the hardware complexity. Section 4 shows that the softmax-like function suits well in a CNN. For a fair comparison we have implemented and synthesized both architectures, our proposed and [22], by using a 90 nm 1.0 V CMOS standard-cell library with Synopsys Design Compiler [39].

Figure 13 depicts the architecture obtained from synthesis where the various building blocks, namely maximum evaluation, subtractor and the simplified exponential LUTs that perform in parallel, are shown. Furthermore registers have been added at the circuits inputs and outputs for applying the delay constraints. Detailed results are depicted in Table 2a–c for the proposed softmax-like of Figure 2a, the [22] and the proposed softmax-like of Figure 2b layer with size 10, respectively. Furthermore, results are plotted graphically in Figure 14a,b where area vs. delay and power vs. delay are depicted, respectively. Results demonstrate that substantially area savings are achieved with no delay penalty. More specifically, for a 4 ns delay constraint the area complexity is 25,597 ${\displaystyle \mathsf{\mu }}$m${\displaystyle {}^2}$ and 43,576 ${\displaystyle \mathsf{\mu }}$m${\displaystyle {}^2}$ in case of architectures in Figure 2b and [22], respectively. For the case where the pdf output is not significant, the area complexity reduction can be 17,293 ${\displaystyle \mathsf{\mu }}$m${\displaystyle {}^2}$ for the architecture on Figure 2a. Summarizing, depending on the application and the design constraints there is a trade-off between the additional ${\displaystyle p}$ terms used for the evaluation of the softmax output. As we increase the value of the parameter ${\displaystyle p}$, then the actual softmax value is better approximated while hardware complexity increases. When ${\displaystyle p=1}$, then the hardware complexity is minimized while softmax output approximation diverges.

6. Conclusions

This paper proposes hardware architectures for implementing the softmax layer in a CNN with substantially reduced reduction in ${\displaystyle \mathrm{area}\times \mathrm{delay}}$ and ${\displaystyle \mathrm{power}\times \mathrm{delay}}$ product, respectively, for certain cases. A family of architectures that can approximate the softmax function have been introduced and evaluated, each member of which is obtained through a design parameter ${\displaystyle p}$, which controls the number of terms employed for the approximation. It is found that a very simple approximation using ${\displaystyle p=1}$, suffices to deliver accurate results in certain cases, even though the derived approximation is not a pdf. Furthermore, it has been demonstrated that for image and digit classification applications, the proposed architecture suits ideally as it achieves MSEs of the order of ${\displaystyle {10}^{-13}}$ and ${\displaystyle {10}^{-5}}$, respectively, which are considered low.