High-Accuracy Gaussian Function Generator for Neural Networks

Abstract

A new improved accuracy CMOS Gaussian function generator will be presented. The original sixth-order approximation function that represents the basis for designing the proposed Gaussian circuit allows a large increase in the circuit accuracy and also of the input variable maximal range. The original proposed computational structure has a large dynamic output range of 27 dB, for a variation smaller than 1 dB as compared with the ideal Gaussian function. The circuit is simulated for 0.18 μm CMOS technology and has a low supply voltage (V_DD = 0.7 V). Its power consumption is smaller than 0.22 μW, for V_DD = 0.7 V, while the chip area is about 7 μm². The new proposed architecture is re-configurable, the convenient modification of the coefficients allowing to obtain many mathematical functions using the same computational structure.

1. Introduction

Signal processing computational structures are widely used in the latest designs, the existing VLSI structures dedicated to implement a multitude of circuit functions. The capability to generate these functions is usually split in order to obtain linear structures, such as amplifiers or multiplier/dividers and to accurate generation of the nonlinear computational structures: squaring/square-rooting circuits, Euclidean or exponential structures.

The Gaussian function [1,2,3,4,5,6,7,8,9,10,11,12,13,14] finds a multitude of applications in wavelet transform, classification and neuro-fuzzy applications, on-chip non-supervised learning or pattern recognition. Important applications of the accurate Gaussian function generation are referring to the VLSI neural networks or algorithms and to back-propagation neural networks. The possibility of developing accurate Gaussian activation functions for neurons in digital networks is correlated with the associated errors to the Gaussian function approximation. Additionally, the fine-tuning of the Gaussian function, achieved using the original method proposed in this paper, contributes to the performance improvement of the neural networks that use the new proposed circuit.

Although synaptic weights are usually reasonable imprecise, a decrease in the errors associated with the Gaussian function computation can increase the system’s overall performance that is based on the Gaussian function generator. Because the control and reduction of the second class of errors are strongly limited by the process used for implementing the computational structure, the best method for improving the circuit performance is to develop a superior-order high-accuracy approximation function to generate the required Gaussian function. In conclusion, a tradeoff between the circuit’s overall accuracy and its power consumption must be performed in order to choose the optimal approximation function. The new approximation function, based on a sixth-order approximation and on a modified superior-order Taylor series, minimizes the circuit complexity and power consumption, maximizing, at the same time, the approximation accuracy.

There are a multitude of Gaussian function applications, in which the overall performance strongly depends on accuracy of the Gaussian function computation. From this point of view, the requirements for an accurate generation of the Gaussian function, associated with possibility of the independent tuning for each parameter of the Gaussian function, become very important for the neural network applications. The previously presented in literature Gaussian circuits have a relatively important degradation of the overall performances as a consequence of the temperature and technological-caused errors. In [1], the Gaussian function is obtained using variable resistors in order to increase circuit bandwidth, the classical MOS transistors being replaced by FGMOS active devices. The utilization of the current sources was proposed in [2] for generating the Gaussian function. The mixed-signal circuit reported in [3] presents a programmability that can be achieved by modifying the reference voltages and transistors’ sizes from its composing differential pairs. Both subthreshold-operated and saturation-biased MOS active devices were used in [4] in order to generate the Gaussian function, this affects the circuit’s frequency response. Switched-current technology represented [5] the basis for developing fully programmable Gaussian function generators. In [6], a compact analog synapse cell, capable to accurately approximate the Gaussian function only in the ideal case is presented. In [7], a Gaussian generator was presented; the Gaussian function being implemented using a fourth-order approximation function. The sixth-order additional term, used for developing the approximation function proposed in this paper, is very important in the context of neural computation, because the possibility to develop accurate Gaussian activation functions for neurons in digital networks is correlated with the errors associated with the approximation of the Gaussian function. In comparison with the fourth-order approximation function proposed and developed in [7], the original sixth-order approximation function presented in this paper allows an important increase in the output dynamic range (from about $5\mathrm{dB}$, for the fourth-order approximation function, shown in [7], to about $27\mathrm{dB}$, for the new sixth-order approximation function, developed in this paper, at the same maximal error of $1\mathrm{dB}$). Consequently, the maximal range of the input variable is strongly increased using the sixth-order approximation in comparison with the fourth-order one (from $0.7$ to $1.75$). Compared with the power consumption of the circuit presented in [7], the new proposed circuit has a power consumption greater with approximately 20%. This disadvantage is entirely compensated for by the much greater accuracy of the new structure comparing with the previous one. The relatively small increasing of the power consumption for the original Gaussian circuit with respect with the computational structure [7] is obtained by using a proper biasing of MOS transistors and also an original architecture of the current-mode squaring circuit the new Gaussian structure is based on.

2. Theoretical Analysis

The proposed Gaussian function generator is designed for requiring small power consumption, also for performing a real-time operation, from this point of view its analog implementation, associated with current-mode operation representing the fundamental basis for obtaining these advantages.

The new implementation of Gaussian function generator proposed in this paper is based on an original approximation function, capable of accurately sixth-order match the Gaussian function. In order to implement this mathematical function in CMOS technology, the current-mode squaring circuits will be used. This approach will allow an important improvement of the operation frequency for the final computational structure, as well as a reduction in the technological-caused errors and the temperature’s impact on the circuit operation.

A.

The new approximation functions

The most convenient choice for developing an accurate approximation function is to use only current-mode squaring circuits, due to its relatively resonable complexities.

The approximation order for the developed approximation function is imposed by the maximal error that can be accepted to generate the Gaussian function, also by the required generator’s output dynamic range. Evidently, a tradeoff between the accuracy of Gaussian circuit implementation, correlated with its maximal output dynamic range and its overall accuracy must be performed. Taking into account the previous considerations, the optimal choice is based on a sixth-order approximation function. This approach will allow an extremely Gaussian circuit large dynamic output range, associated with a very good accuracy using reasonable complexity of the CMOS implementation.

The basic original proposed sixth-order approximation function can be written as follows:

$$g_1\left(x\right)=1-\frac{x^2}{2{\sigma }^2}+\frac{x^4}{8{\sigma }^4}-\frac{x^6}{48{\sigma }^6}$$

(1)

This function represents the sixth-order limited Taylor series expansion of the Gaussian function, $f\left(x\right)$, that can be generally expressed as follows:

$$f\left(x\right)=A\exp \left(-\frac{x^2}{2{\sigma }^2}\right)$$

(2)

$A$ and $\sigma$ being constants that define the amplitude and the variance for the Gaussian function, respectively.

The approximation error resulted from the utilization of $g_1\left(x\right)$ function is mainly represented by the eighth-order term from the Taylor series expansion of $f\left(x\right)$ Gaussian function. It is proportional to the value of $x$ input variable, that models the amplitude of the input signal. In order to increase the output dynamic range for the Gaussian function generator, the variation domain of $x$ variable, required in order to obtain a wide output variation of the approximation function must be reduced (this aproach minimizes the approximation error for the same output dynamic range). The original proposed method for fulfilling this desiderate uses a variable changing, $x\to x/2$. Using the following mathematical identity:

$$\exp \left(-\frac{x^2}{2{\sigma }^2}\right)={\left\{\exp \left[-\frac{{\left(\frac{x}{2}\right)}^2}{2{\sigma }^2}\right]\right\}}^4$$

(3)

The original approximation function will have the following expression:

$$g_2\left(x\right)={\left(1-\frac{x^2}{8{\sigma }^2}+\frac{x^4}{128{\sigma }^4}-\frac{x^6}{3072{\sigma }^6}\right)}^4$$

(4)

which is obtained from the Expression (1) of $g_1\left(x\right)$, using the previous presented variable changing.

B.

Implementations of the Gaussian function generator

The Gaussian function generator block diagram is presented in Figure 1. As a result of the variable changing that was used for designing this circuit, it is expected that its output dynamic range to be strongly extended. The functional characteristic of the current-mode squaring circuits from Figure 1 is $I_C=I_B^2/8I_A$, as it will be demonstrated. Using this relation for all five squaring circuits from Figure 1, the expression of the $I_{OUT1}$ output current can be written as follows:

$$I_{OUT1}=\frac{I_{IN}^2}{8{\sigma }^2{I}_O}$$

(5)

while the $I_{OUT2}$ current can be expressed as follows:

$$I_{OUT2}=\frac{I_{OUT1}^2}{2I_O}=\frac{I_{IN}^4}{128{\sigma }^4{I}_O^3}$$

(6)

The expression of the $I_{OUT3}$ current is:

$$I_{OUT3}=\frac{2I_{OUT2}^3}{3I_{OUT1}}=\frac{I_{IN}^6}{3072{\sigma }^6{I}_O^5}$$

(7)

The $I$ current has a linear dependence on the prevuious expressed currents:

$$I=I_O-I_{OUT1}+I_{OUT2}-I_{OUT3}$$

(8)

So:

$$I=I_O\left[1-\frac{1}{8{\sigma }^2}{\left(\frac{I_{IN}}{I_O}\right)}^2+\frac{1}{128{\sigma }^4}{\left(\frac{I_{IN}}{I_O}\right)}^4-\frac{1}{3072{\sigma }^6}{\left(\frac{I_{IN}}{I_O}\right)}^6\right]$$

(9)

Because:

$$I_{OUT}=\frac{I_{OUT4}^2}{I_O}=\frac{I^4}{I_O^3}$$

(10)

It results:

$$I_{OUT}=\frac{I_{OUT4}^2}{I_O}=I_O{\left[1-\frac{1}{8{\sigma }^2}{\left(\frac{I_{IN}}{I_O}\right)}^2+\frac{1}{128{\sigma }^4}{\left(\frac{I_{IN}}{I_O}\right)}^4-\frac{1}{3072{\sigma }^6}{\left(\frac{I_{IN}}{I_O}\right)}^6\right]}^4$$

(11)

So, because $x$ variable is defined as $x\stackrel{def}{=}{I}_{IN}/I_O$, it is possible to write that:

$$I_{OUT}=I_O{\left(1-\frac{x^2}{8{\sigma }^2}+\frac{x^4}{128{\sigma }^4}-\frac{x^6}{3072{\sigma }^6}\right)}^4$$

(12)

In comparison with (4), can conclude that the $I_{OUT}$ current is proportional to the second approximation function, $g_2\left(x\right)$, $I_{OUT}=I_O{g}_2\left(x\right)$. So, the relation (12) will approximate the required Gaussian function expressed by (2). The advantages of these new proposed achievements of Gaussian function generators are related to the independence of their operation on temperature variations and on technological-caused errors. Additionally, they allow the changing of the Gaussian function variance, $\sigma$, using a single current mirror $1:\sigma$.

The new proposed current-mode squaring circuits (named SQ in Figure 1) are presented in Figure 2. The $I_C$ is the circuit’s output current, $I_B$ represents the input current and $I_A$ is the reference current. The MOS transistors from Figure 2 have an aspect ratio $\left(W/L\right)=0.18\mathsf{\mu }\mathrm{m}/0.5\mathsf{\mu }\mathrm{m}$. The Gaussian function generator chip area, using the squaring circuit presented in Figure 2, is about ${7\mathsf{\mu }\mathrm{m}}^2$.

It can be written that $I_B=I_3-I_2$ and $I_C=I_2+I_3-2I_O$. The relation between gate-source voltage of the MOS transistors from Figure 2 is:

$$V_{GS1}+V_{GS4}=V_{GS2}+V_{GS3}$$

(13)

For a biasing in saturation of the MOS transistors, it results:

$$2\sqrt{I_A}=\sqrt{I_2}+\sqrt{I_3}$$

(14)

Considering the expressions between circuit currents, it is possible to write that:

$$I_2=I_A+\frac{I_C-I_B}{2}$$

(15)

and:

$$I_2=I_A+\frac{I_C-I_B}{2}$$

(16)

From the previous three expressions, it results:

$$I_C=\frac{I_B^2}{8I_A}$$

(17)

The CMOS implementation of the original Gaussian function generator is presented in Figure 3. The CM1 and CM2 blocks are multiple current mirrors, having the currents ratios similar with the current mirrors from the block diagram presented in Figure 1.

3. Simulation Results

The circuit proposed in Figure 2 was simulated using LT Spice. The $I_C\left(I_A,I_B\right)$ current variation, expressed by (17), is shown in Figure 4 ($I_A=50\mathrm{nA}$ and a range of $I_B$ from $-50\mathrm{nA}$ to $50\mathrm{nA}$). The simulated error for Function (17) using the structure proposed in Figure 2 is about $0.35\%$. The process parameters are from $0.18\mathsf{\mu }\mathrm{m}$ CMOS technology and a supply voltage $V_{DD}=0.7\mathrm{V}$. The circuit consumption is approximately $0.3\mathsf{\mu }\mathrm{A}$ for the entire achievement of the original circuit.

Comparing $g_2\left(x\right)$ function and $f\left(x\right)=\exp \left(-x^2\right)$ function for $\sigma =1/\sqrt{2}$, results the information presented in

Table 1. Comparison between $g_2\left(x\right)$ and $f\left(x\right)$ functions.

x	g2 (x) [dB]	f (x) [dB]	ε [dB]
0	0	0	0
0.2	−0.35	−0.35	0
0.4	−1.39	−1.39	0
0.6	−3.13	−3.13	0
0.8	−5.56	−5.56	0
1	−8.69	−8.69	0
1.2	−12.54	−12.52	0.02
1.4	−17.15	−17.04	0.11
1.6	−22.64	−22.25	0.39
1.7	−25.82	−25.12	0.70
1.75	−27.55	−26.61	1.06

.

Table 1 shows a deviation of the approximation function from the ideal Gaussian function not greater than $1\mathrm{dB}$ for a circuit’s large output dynamic range, $27\mathrm{dB}$.

A graphical comparison between the $g_2\left(x\right)$ and $f\left(x\right)$ functions is presented in Figure 5.

The new proposed Gaussian function generator is designed for implementing in $0.18\mathsf{\mu }\mathrm{m}$ CMOS technology. The circuit has a minimal supply voltage $V_{DD}=0.7 \mathrm{V}$ and a power consumption of $0.22\mathsf{\mu }\mathrm{W}$, for $V_{DD}=0.7 \mathrm{V}$. The chip’s area is estimated to be about ${7\mathsf{\mu }\mathrm{m}}^2$.

In

Table 2. Comparison between the proposed Gaussian circuit and previous reported works.

Reference/Parameter	Technology [μ]	Supply Voltage [V]	Error [%]
[2]	0.25	3.3
[3]			3
[4]			2
[5]			1
[6]	0.35	1.5
[7]	0.18	1	0.1
This work	0.18	0.7	0.1

a comparison between the new Gaussian circuit and previous similar works is presented, from the point of view of the obtained approximation error when generating the Gaussian function.

As a result of the information presented in Table 2, the estimated accuracy of a neural network employing the proposed Gaussian function generator circuit will be better than or equal to the accuracies of neural networks employing other already reported in literature similar Gaussian function generator circuits. Additionally, the new proposed computational structure will have the very important advantage, namely, to allow an extremely low-voltage operation (a minimal supply voltage of 0.7 V).

As compared with the approximation function developed in [7], the original function proposed in this paper allows an important increase in the output dynamic range, from about $5\mathrm{dB}$ to about $27\mathrm{dB}$, for the same maximal error of $1\mathrm{dB}$.

4. Conclusions

In the context of the existence of a multitude of applications for the accurate Gaussian function generation in the VLSI neural networks or algorithms and back-propagation neural networks areas, a new accurate CMOS Gaussian function generator was proposed. The new sixth-order approximation function allows an important increase in the circuit accuracy also of the maximal range for the input variable. The possibility of fine-tuning the Gaussian function, based on the original method proposed in this paper, strongly increases the performance of the neural networks that use the new proposed circuit.

The original proposed method used in this paper for obtaining the Gaussian function was based on the utilization of a new approximation function that strongly increases the circuit’s accuracy. In order to further increase the Gaussian function accuracy, a new method based on the utilization of a convenient variable changing was used. Because of imposing a current-mode operation of the proposed circuit, many advantages could be obtained. Firstly, the circuit’s frequency response was significantly improved as compared with similar designs. Secondly, the circuit’s independence on the technological parameters additionally increases the accuracy of the Gaussian function generation. Thirdly, the independence of the circuit output current on the technological parameters allows an operation of the original proposed structure on temperature drifts.

The original sixth-order approximation function, that represents the basis for designing the proposed Gaussian circuit, allows a large improvement from the accuracy and range of input voltage perspectives. The new proposed computational structure has an extended dynamic output range of $27\mathrm{dB}$, for a deviation from the ideal Gaussian function smaller than $1\mathrm{dB}$. The new proposed architecture has the important property of re-configurability (the choosing of the approximation function coefficients allows to obtain many mathematical functions, starting from the same computational structure).