On the Use of the Cumulative Distribution Function for Large-Scale Tolerance Analyses Applied to Electric Machine Design

Abstract

In the field of electrical machine design, excellent performance for multiple objectives, like efficiency or torque density, can be reached by using contemporary optimization techniques. Unfortunately, highly optimized designs are prone to be rather sensitive regarding uncertainties in the design parameters. This paper introduces an approach to rate the sensitivity of designs with a large number of tolerance-affected parameters using cumulative distribution functions (CDFs) based on finite element analysis results. The accuracy of the CDFs is estimated using the Dvoretzky–Kiefer–Wolfowitz inequality, as well as the bootstrapping method. The advantage of the presented technique is that computational time can be kept low, even for complex problems. As a demanding test case, the effect of imperfect permanent magnets on the cogging torque of a Vernier machine with 192 tolerance-affected parameters is investigated. Results reveal that for this problem, a reliable statement about the robustness can already be made with 1000 finite element calculations.

1. Introduction

With the rise of modern optimization techniques in combination with continuously increasing computational power, it is becoming more and more likely to find outstanding solutions to certain design tasks, even if such optima are hidden in the narrowest design-space gaps [1,2]. In the field of electrical machine design, usual optimization targets are performance measures, like efficiency, cost, or torque ripple [3]. A common problem of highly optimized designs is that a small deviation of the design parameters from the ideal values may lead to a significant deterioration of the performance—the design is thus sensitive to parameter tolerances. To obtain designs which are not prone to a significant performance degradation in the presence of inevitable tolerances, some measure describing this resistance has to be incorporated into the (multi-objective) optimization process. A design which has a certain resistance against tolerances is usually called a robust design, and an optimization which takes into account the effect of tolerances is called a robust optimization. In this paper, a method is presented on how to evaluate the robustness of a design that has a large number of tolerance-affected parameters. The application of the method within the scope of an optimization will be dealt with in future work.

In the field of electrical machine design, intensive research is being carried out on the topic of robust design. A huge number of publications deal with the effect of tolerances, especially considering the cogging torque change caused by permanent magnet uncertainties, such as in [4,5,6,7]. The different influences of size, positioning, or magnetization strength and direction are often investigated by applying two or three discrete tolerance levels and searching for worst-case combinations. To reduce the inherent computational burden associated with finite element analyses, symmetry conditions with semi-analytical methods [5] or surrogate models [7,8] are often utilized. Besides searching for worst-case configurations based on discrete tolerance levels, stochastic approaches, like the six sigma quality method [9,10,11], are also investigated to predict the behavior of systems subject to tolerances. Using stochastic measures like the standard deviation $\sigma$ might be preferable compared to the worst-case method [12]. For example, when designing an airplane propulsion, the worst-case measure might definitely be the proper method, since each single failure can have tremendous consequences; whereas for a standard industrial electric machine, this leads to an over-designed, and thus over-priced system which is not competitive.

An overview of different robustness criteria applied in the field of electrical machine design was recently presented in [13], where a quantile measure based on the evaluation of cumulative distribution functions (CDFs) was also introduced for a problem with four tolerance-affected design parameters. The capability of CDFs for large-scale tolerance analysis of problems showing a huge number of parameters and the feasibility of predicting their confidence bounds is presented in this paper by focusing on electric machine applications. The chosen test problem will be presented in Section 2, followed by an introduction to the cumulative distribution function and a theoretical method for confidence estimation in Section 3. In Section 4, the calculation of the sample data is described and the evolution of the CDF with increasing number of samples is illustrated. For a data-based estimation of the CDFs‘ confidence, the bootstrap method [14,15,16] was utilized (Section 5). Finally, we discuss benefits and limitations of the presented method and give a short outlook and conclusion.

2. Test Case Definition

To show the capabilities of using CDFs as a robustness measure for large-scale tolerance analysis, an outer rotor Vernier motor is used as a striking example. Such motors modulate the flux created by a low pole count winding system to an air gap field with major higher harmonic components. As a consequence, an appropriate high number of magnets has to be arranged on the outer rotor to interact with the corresponding air gap field. Such motors can be designed to have outstanding performance in low speed and high torque applications, but also suffer from a significantly nonlinear relationship between the dimensions and the magnetic flux distribution [17]. Thus, reliable surrogate models with sufficient accuracy are hard to obtain. The Vernier motor used for this investigation is shown in Figure 1, and some basic parameters are given in

Table 1. Parameters of the Vernier motor shown in Figure 1.

Parameter	Value	Description
p	24	number of pole pairs of the outer rotor
n	27	number of micro slots of the stator
$d_o$	$80\mathrm{mm}$	outer diameter of the rotor
$l_{fe}$	$20\mathrm{mm}$	stack length of the motor
$l_{air}$	$0.5\mathrm{mm}$	air gap length
$b_m$	$3.74\mathrm{mm}$	nominal width of magnets
$h_m$	$1.73\mathrm{mm}$	nominal height of magnets

.

It might be mentioned that this Vernier motor has not a claim to be optimal in any sense—that is, it was not optimized for any special purpose, but should only serve as an illustrative and demanding real-world test case. The goal of this investigation was to predict the impact of permanent magnet uncertainties on the cogging torque. The uncertainties concern the

• magnet width ($\Delta b_m$),
• magnet height ($\Delta h_m$),
• radial position ($\Delta x_r$), and
• circumferential position ($\Delta x_{\phi }$).

The tolerance-affected parameters are also shown in Figure 1. The width and height vary due to the manufacturing process of the permanent magnets, and their tolerances have maximum values of ${\delta }_{b_m}=\pm 0.05\mathrm{mm}$ and ${\delta }_{h_m}=\pm 0.05\mathrm{mm}$, respectively. Radial and circumferential positions can vary due to the assembling process of the permanent magnets. If, for example, the magnets are glued, the adhesive gap thickness can vary, leading to a radial position which deviates from the expected value. For this investigation, a radial maximum displacement tolerance of ${\delta }_{x_r}=-0.01\mathrm{mm}$ is assumed. Note that for the radial displacement, only a negative tolerance limit is given, what corresponds to a movement towards the center. The fourth parameter is the circumferential position of the magnets. They are buried into slots with a width of $b_s$, as illustrated in the highlighted area of Figure 1. The maximum circumferential displacement of a certain magnet i is a function of its actual tolerance-affected width, that is, ${\delta }_{x_{\phi }}^i=\pm \frac{b_s-b_{m\Delta }^i}{2}$, where $b_{m\Delta }^i=b_m+\Delta b_m^i$.

In the test machine, the slots have a width of $b_s=b_m+{\delta }_{b_m}$. Thus, a magnet i prone to the maximum tolerance level, that is, $b_{m\Delta }^i=b_m+{\delta }_{b_m}$, cannot move any more ($\Delta x_{\phi }^i=0$). A magnet with a minimum width of $b_{m\Delta }^i=b_m-{\delta }_{b_m}$ can move up to $\Delta x_{\phi }^i=\pm {\delta }_{b_m}$. All parameters subject to tolerances are assumed to have uniform distribution and are summarized in

Table 2. Tolerance definition.

Parameter	Symbol $\Delta \mathit{X}$	Range of $\Delta \mathit{X}$	Tolerance Limit ${\mathit{\delta }}_{\mathit{X}}$	Distribution
magnet width	$\Delta b_m$	$[-{\delta }_{b_m},+{\delta }_{b_m}]$	$0.05\mathrm{mm}$	uniform
magnet height	$\Delta h_m$	$[-{\delta }_{h_m},+{\delta }_{h_m}]$	$0.05\mathrm{mm}$	uniform
radial position	$\Delta x_r$	$[-{\delta }_{x_r},0]$	$0.01\mathrm{mm}$	uniform
circumferential position	$\Delta x_{\phi }$	$[-{\delta }_{x_{\phi }},+{\delta }_{x_{\phi }}]$	$\frac{b_s-b_{m\Delta }}{2}$	uniform *

* on a normalized range (see Section 4).

.

It might be worth emphasizing that due to the stochastic nature of the tolerances, all 48 magnets are likely to have individual values for a given design. Therefore, their specific values will be modeled independently. Assuming a discretization of three steps for each of the four parameters, this leads to $3^4=81$ possible combinations for a single magnet. Taking into account that every magnet features its independent tolerance values, a total of ${\left(3^4\right)}^{48}=3^{192}>4\times {10}^{91}$ different variations are possible. If the simulation of one design would need one microsecond, the evaluation of all variants would need more than ${10}^{78}$ years. Even if a proper analytical (surrogate) model would be available to calculate cogging torque, evaluating all combinations would never be an option. Hence, alternative approaches need to be developed.

3. Introduction to Cumulative Distribution Functions

From a set of n tolerance-affected sample values $X_1,\dots ,X_n$, that is, n samples of a certain target evaluated under the influence of tolerances, the empirical cumulative distribution function (ECDF) ${\widehat{F}}_x\left(x\right)$ is defined as [18]

$${\widehat{F}}_x\left(x\right)=\frac{1}{n}\sum _{i=1}^{n}I(X_i,x),I(X_i,x)=\left\{\begin{array}{c}1\mathrm{if}{X}_i\le x\hfill \\ 0\mathrm{if}{X}_i>x.\hfill \end{array}\right.$$

(1)

In words, ${\widehat{F}}_x\left(x\right)$ is the weighted count of samples $X_i$ for which $X_i$ is below or equal to the threshold, x. A set of exemplary random samples $X_i$, and the respective ECDF ${\widehat{F}}_x\left(x\right)$ are plotted in Figure 2.

From the cumulative distribution function, it is easy to determine how many of the samples have a value below or equal to a certain limit, or what the parameter limit $q_x$ is, where a certain percentage p of the samples has a smaller or equal value. This threshold $q_x^p$ is called the p-quantile. In Figure 2, the 90% quantile is shown, meaning that 90% of the samples have a value smaller or equal to $q_x^{0.9}=2.73$.

Given the n samples, a continuous probability density function $f\left(x\right)$ can also be fitted, using kernel density estimation (KDE). KDE is a popular approach to estimate the probability density function of random datasets, and due to its non-parametric nature, it is also capable of fitting complicated distributions [18,19,20]. In Figure 2, this KDE estimate $f\left(x\right)$ is shown as a blue line.

From $f\left(x\right)$, the CDF $F_x\left(x\right)$ can be determined as

$$F_x\left(x\right)={\int }_{-\infty }^{x}f\left(\xi \right)\mathrm{d}\xi .$$

(2)

The 90%-quantile in Figure 2 is shown based on this continuous CDF. As an absolute measure for the robustness of a design, any quantile value can be used. Another indicator is the steepness of $F_x$—that is, the steeper the CDF is, the more robust the system is.

It is clear that the ECDF ${\widehat{F}}_x$, and consequently, also the fitted CDF $F_x$, can only be an estimate of the real CDF with $n\to \infty$. Thus, the remaining but essential question is: How good is this estimate for a given number of n? From the Dvoretzky–Kiefer–Wolfowitz inequality, a $(1-\alpha )$-confidence band for $F_x$ can be constructed as follows [18]:

$$\mathcal{P}\left(L\left(x\right)\le F\left(x\right)\le U\left(x\right)\mathrm{for}\mathrm{all}x\right)\ge 1-\alpha ,$$

(3)

where

$$\begin{array}{cc}\hfill L\left(x\right)& =max\{{\widehat{F}}_n\left(x\right)-{ϵ}_n,0\}\hfill \\ \hfill U\left(x\right)& =min\{{\widehat{F}}_n\left(x\right)+{ϵ}_n,1\}\hfill \end{array}$$

(4)

and

$${ϵ}_n=\sqrt{\frac{1}{2n}log\left(\frac{2}{\alpha }\right)}.$$

(5)

Equation (3) implies that the probability $\mathcal{P}$ of the real CDF $F_x$ lying in between some lower $L\left(x\right)$ and upper $U\left(x\right)$ boundary is greater than or equal to $1-\alpha$, where the definitions of the lower and upper bound are given by (4) and (5). For example, if we want 95% confidence ($\alpha =0.05$) of the estimation ${\widehat{F}}_x$ shown in Figure 2 with $n=34$ samples, from (5) an uncertainty of ${ϵ}_n=0.233$, which is equivalent to $\pm 23.3\%$, can be calculated. Thus, the real CDF has a probability of 95% laying in the confidence slot of $\widehat{F_x}\pm 23.3\%$. This slot, bounded by the light-grey dashed lines $L\left(x\right)$ and $U\left(x\right)$, is also displayed in Figure 2. The method based on (3) to (5) will be referred to as the DKW method throughout the paper.

Let us assume that a 95% confidence interval with a slot of $\pm 1\%$ is desired. An approximation of the required sample size can be calculated by rearranging (5), leading to n = 18,445 samples for this example. Note that no information about the actual problem is necessary using the DKW method. However, from (5), it can also be observed that the confidence slot and thus ${\widehat{F}}_x$ converges with $\frac{1}{\sqrt{n}}$.

If we are interested in the confidence range in parameter space x rather than in the probability space $p=F_x\left(x\right)$, the (E)CDF can be inverted to get $F_x^{-1}\left(p\right)$. Since $F_x^{-1}\left(p\right)$ is basically the definition of the p-quantile, the inverse is also called a quantile function.

4. Creation of Sample Data and Evolution of the CDF

To create the test data for this investigation, a parametric finite element model was set up using the SyMSpace framework [21]. For the finite element calculations, FEMM software was used [22], to which SyMSpace features an interface. Since the circumferential positioning $\Delta x_{\phi }^i$ of each magnet i is dependent on its actual width, which is also subject to tolerances, for this parameter the uniform distribution is mapped to a normalized range of $\Delta {\xi }_{x_{\phi }}^i=[-1,1]$, where

$$\Delta {\xi }_{x_{\phi }}^i=\frac{\Delta x_{\phi }^i}{{\delta }_{x_{\phi }}}.$$

(6)

Considering (6) and Figure 1, if $\Delta {\xi }_{x_{\phi }}^i=1$, the affected magnet is moved in clockwise direction, such that it is touching the edge of the slot. For $\Delta {\xi }_{x_{\phi }}^i=-1$, the magnet is against the slot edge in counterclockwise direction. In Figure 3 the normalized tolerance values of all magnets for an exemplary sample are visualized, where

$$\Delta {\xi }_{x_r}^i=\frac{\Delta x_r^i}{{\delta }_{x_r}},\Delta {\xi }_{b_m}^i=\frac{\Delta b_m^i}{{\delta }_{b_m}},\Delta {\xi }_{h_m}^i=\frac{\Delta h_m^i}{{\delta }_{h_m}}.$$

(7)

To speed up calculations done by the finite element program, usually sectors with some periodic boundary conditions are defined and the rotor is rotated by one electrical period. Since these investigations consider tolerances leading to asymmetric designs, that is, every magnet is exposed to different tolerance values, the full cross-section has to be modeled in the finite element program and the rotor has to be rotated by the angular width of one main stator tooth ($360°/9=40°$ in this case) to allow for analysis of all possible harmonics of the cogging torque. In our case, such a calculation takes approximately 15 min. From the obtained cogging torque $T_c\left(\phi \right)$ as a function of the rotor angle $\phi$, the peak-to-peak value $T_{c,pp}$ is evaluated, which also serves as a target variable for the following investigations.

After evaluating $n=20$ randomly defined motor cross-sections (a motor subject to a random set of tolerances will also be referred to as a sample throughout the paper), the CDF $F_T^{20}$ is fitted from the $T_{c,pp}$ values, as displayed in Figure 4. The right superscript denotes the number of n samples upon which the CDF is based, that is, $F_T^n$.

The continuous CDF $F_T^{20}$, which is fitted by kernel density estimation using a Gauss kernel, as well as its deviation to the empiric CDF ${\widehat{F}}_T^{20}$ with respect to probability ${\epsilon }_p$ (lower axis), and its deviation to ${\widehat{F}}_T^{20}$ with respect to the quantile value (cogging torque) ${\epsilon }_q$ (right axis) are also shown in Figure 4. From the appended axes, it can be observed that the deviation from the fitted to the empiric CDF is below 10% and $5\mathrm{mNm}$.

To have a single measure for the deviation between two (E)CDFs, $F_{x,1}\left(x\right)$ and $F_{x,2}\left(x\right)$, the absolute area between them, $A_d\left(F_{x,1},F_{x,2}\right)$, can be evaluated:

$$A_d\left(F_{x,1},F_{x,2}\right)=\underset{min\left(x\right)}{\overset{max\left(x\right)}{\int }}\left|F_{x,1}\left(x\right)-F_{x,2}\left(x\right)\right|\mathrm{d}x.$$

(8)

Since the (E)CDF is a normalized quantity, $A_d$ has the unit of the parameter which the distribution describes ($\mathrm{Nm}$ in this case). $A_d$, generally defined as the deviation between two arbitrary CDF functions, will serve as a parameter predicting the convergence of the approach presented in the following.

Now, from Figure 4 and also from Figure 2, it can be seen that the ECDF can be reasonably fitted by kernel density estimation, but it is not clear if a CDF, based on a low number of samples, reflects the statistical behavior to be expected when manufacturing this motor in a mass production process. Thus, the idea is to observe the evolution of the CDF when more and more samples are calculated. Starting from the CDF based on 20 samples, a selection of CDFs based on an increased number of samples is shown in Figure 5.

In total, 2000 samples were evaluated. The CDF fitted from this full dataset will serve as a reference CDF $F_T^{2k}$. In Figure 5, and also in all the following investigations, only the KDE-fitted CDFs are considered.

Now, from Figure 5, it can be observed that $F_T$ converges very fast. The appended axes below and on the right side show the deviation of $F_T^n$ to $F_T^{2k}$. From this Figure, it seems that $F_T^{500}$ already converged quite close to a “final” CDF. Thus, it looks like a statistical statement on the robustness of cogging torque against magnet tolerances can be derived within 500 simulations for a problem with an incredible number of possible combinations.

An explanation for this desirable behavior is the insensitivity of the KDE-fitted CDF to fluctuations of the underlying data. On the one hand, the CDF is based on a set of sample values, where each single sample contributes with a value of $1/n$ to the functional, and on the other hand, the fitting process has a smoothing effect.

However, thinking on the iterative procedure of creating new samples and updating $F_T$, the question of how to define convergence remains, since in Figure 5 the convergence was identified by referring to a CDF based on much more samples ($F_T^{2k}$) than actually available. We cannot rely on a comparison to this “best available” CDF, but have to compare to the CDFs from previous update steps and track the evolution of the CDFs.

To evaluate this behavior, every 10 samples a CDF was calculated, that is, $F_T^{10},F_T^{20},F_T^{30},\dots F_T^{2k}$, and the deviation area between successive CDFs, $A_d(F_T^n,F_T^{n-10})$, was evaluated. This course, and also the deviation with respect to $F_T^{2k}$, $A_d(F_T^n,F_T^{2k})$, is shown in Figure 6.

Looking at the deviation to $F_T^{2k}$, which gives an absolute measure of the convergence, it turns out that a significant drop appears within the first 100 samples, then there is some fluctuation and from about 500 samples, $A_d(F_T^n,F_T^{2k})$ is stable and shows a steady convergence towards zero. The relative deviation, $A_d(F_T^n,F_T^{n-10})$, converges even faster to very low values, as can be seen from the lower axis of Figure 6. Thus, from $A_d(F_T^n,F_T^{n-10})$, one could interpret a faster convergence of $F_T$, but since it is a relative measure, it could have an integrating character. This means that even if $A_d(F_T^n,F_T^{n-10})$ is low, $F_T$ could slowly but steadily move in one direction, and no information is available about the “real” limit, $F_T^{\infty }$.

Another point which was not discussed so far is also shown in Figure 6—the effect of random sampling. Since the samples are defined randomly, or more precisely, the tolerance values of each parameter for every magnet are selected randomly considering its distribution, it is obvious that quite different CDFs can emerge, especially at the beginning. To verify this, the evolution of $A_d(F_T^n,F_T^{2k})$ and $A_d(F_T^n,F_T^{n-10})$ was evaluated with the available samples in reversed order—that is, the sample number 2000 was used as the first one, sample number 1999 was used as the second one, and so on. The reversed order was designated with a negative sign of the sample index n in the corresponding deviation function, as drawn in Figure 6 with dotted lines. When comparing the deviations based on the standard ordered samples with the ones based on the reversed ordered samples, it is worth emphasizing that, up to sample index 1000, they relied on completely different samples. Having this in mind, the comparison shows interesting insights—the first one comes by observing $A_d(F_T^{-n},F_T^{-2k})$, which has an (expected) deviation to $A_d(F_T^n,F_T^{2k})$ in the lower samples range, but shows good agreement with $A_d(F_T^n,F_T^{2k})$ already from $n=400$ onwards. The second one is that the relative evolution shows pretty much the same behavior, regardless of whether the samples are in standard or in reversed order.

To summarize this section:

• The method seems promising, especially since $A_d(F_T^n,F_T^{2k})$ and $A_d(F_T^{-n},F_T^{-2k})$ converge to the same value even for different sets of samples (cf. range $400<n<1000$ in Figure 6).
• From the relative deviation area $A_d(F_T^n,F_T^{n-10})$, no reference is available about the remaining deviation to $F_T^{\infty }$.
• The relative deviation seems to be quite robust concerning the random sampling, since its evolution has a high correlation for different sets of samples ($n<1000$ in Figure 6).

5. Evaluation of Data-Based CDF Confidence Values

For an iterative approach, $A_d(F_T^n,F_T^x)$ with $x\gg n$ cannot be evaluated, since no “best” reference $F_T^x$ ($F_T^{2k}$ in the previous section) is available. From the relative evolution of $F_T$, or more precisely, of $A_d(F_T^n,F_T^{n-10})$, no statement can further be drawn about the absolute deviation, due to the (possible) integrating character.

From the presented theory of Section 3, especially (5), one could determine which sample count n is required for guaranteeing a specified confidence interval. However, the underlying problem to be investigated is not taken into account, as (5) was developed for deriving confidence intervals for any empiric cumulative distribution function without loss of generality. Such general (not problem-specific) derivations usually implicate very conservative measures, such as confidence interval bounds. Besides, they often imply assumptions, like unbiased data sets, which can hardly be ensured for practical applications. While it can be more easily guaranteed for the tolerance-affected input variables, ensuring an unbiased performance measure is practically impossible, as the input–output relation is not known in general.

To overcome the lack of problem-specific information, the bootstrap approach is used to estimating the confidence interval [14,15,16,20]. This common method in statistical inference utilizes the sample data itself and not only the number of samples for confidence boundary estimation.

The confidence value is created using the following approach:

• Create $n_b$ batches (bootstrap samples), where every batch contains n samples which are drawn from the original dataset with replacement. The original dataset also features n samples.
• Create a CDF for every bootstrapped batch of samples $F_T^b$ and calculate the deviation to the CDF based on the original dataset, $A_d(F_T^b,F_T^n)$.
• Investigate the distribution of the deviations by creating a CDF based on the deviation values $F_{A_d}$.
• Evaluate a confidence value based on $F_{A_d}$.

Let us go through the steps using an example: Assume a dataset with $n=100$ samples. From these 100 samples, $n_b=200$ batches of samples are created, where each batch also contains $n=100$ samples. The samples of one batch are randomly selected out of the full dataset with replacement. From this, it follows that a specific sample from the original dataset can occur in a batch multiple times. The CDF based on all available samples, $F_T^{100}$, is plotted in Figure 7a together with the $n_b=200$ different CDFs calculated from the 200 batches, $F_T^b$.

Since every batch consists of a different set of samples, the corresponding CDFs look different. They form a tube around $F_T^{100}$, which defines some kind of confidence for the case where 100 samples are available. In the next step, we want to have a look at the deviation of every batch CDF to the original data CDF, $A_d(F_T^b,F_T^{100})$. The probability density function $f_{A_d}$ of these deviations, as well as the corresponding cumulative distribution function $F_{A_d}$ are shown in Figure 7b.

From $F_{A_d}$, we can now easily read which share of the batches leads to a deviation area below a certain limit, or the other way round: what is the maximum deviation area that a certain amount of batches does not exceed? In our example, 90% of the batches lead to a deviation area of $A_d\le 0.0082\mathrm{Nm}$, as also shown in Figure 7b. This parameter will be used to rate the confidence of the batches and is designated by ${\mathcal{C}}_T^{0.9}$. Although ${\mathcal{C}}_T^{0.9}$ gives the 90%-quantile of $A_d(F_T^b,F_T^{100})$, the index T is used. On the one hand, this was chosen because it should serve as a confidence value for pretending the distribution of the cogging torque $T_{c,pp}$. On the other hand, this choice can be argued by the following interpretation of ${\mathcal{C}}_T$: If any CDF $F_x\left(x\right)$ is horizontally shifted by any value ${\delta }_x$, the area $A_d$ between the original and the shifted CDF just corresponds to the shift value, since $F_x$ is always normalized between zero and one—thus, $A_d={\delta }_x$. So, if in our example $F_T^{100}$ is shifted by $\pm {\mathcal{C}}_T^{0.9}$ on the parameter axis, as depicted in Figure 7a, it can be interpreted as a worst-case slot, where 90% of all batch CDFs are within.

An aspect to consider when using this confidence interval is related to the worst-case scenario—especially for large-scale problems, it is very unlikely that the worst-case performance can be identified through the relatively small share of initially evaluated samples. Additionally, the CDF turns very flat when approaching a quantile level close to 1, following significant errors for the corresponding quantile values (here, cogging torque levels), even if the CDF exhibits only small uncertainties on the quantile level (that is, probability value). To be on the safe side, it is recommended to use this method only up to a quantile level of approximately 0.95.

In Figure 7a, the 90% limit slots calculated with the DKW method are additionally shown. Not to be confused with the ${\mathcal{C}}_T^{0.9}$ limit, the $L-U$ slot defines the slot, wherein the real CDF ($F_T^{\infty }$) is with a probability of 90%. Similar to the evolution of $F_T$, now the question arises about how many batches are necessary until $F_{A_d}$ converges. Since it is again about convergence of a cumulative distribution function, Equations (3)–(5) can be applied, which basically means the same convergence behavior for $F_{A_d}$ as for $F_T$. For 5000 batches, ${ϵ}_n=1.9\%$ can be evaluated from (5). To show convergence, the deviation of $F_{A_d}^{n_b}$ to $F_{A_d}^{5k}$, $A_d(F_{A_d}^{n_b},F_{A_d}^{5k})$ was evaluated as a function of the number of batches $n_b$ and presented in Figure 8.

It can be seen that convergence is fast, and that the prediction continuously gets better with an increasing number of samples. It is further important to emphasize that the creation of $F_{A_d}$ is very fast, because no finite element calculations are needed. One additional aspect concerning convergence is that taking the shape of the whole CDF into account is probably too strict as a criterion. Actually, we are interested in the convergence of particular quantiles ${\mathcal{C}}_T^p$, rather than the convergence of the whole distribution function. In Figure 9, ${\mathcal{C}}_T^{0.9}$ is shown over the number of batches and for different values of total samples n.

It can be observed that ${\mathcal{C}}_T^{0.9}$ is fluctuating within the first 500 batches and has almost reached convergence after 1000 batches. What can further be seen even better than in Figure 8 is that ${\mathcal{C}}_T^{0.9}$ drops significantly with an increasing number of total samples.

The evolution of different confidence values over the number of available samples is shown in Figure 10, together with the absolute and relative deviation area (cf. Figure 6).

What can be observed from Figure 10 is that ${\mathcal{C}}_T$ seems to be a good measure for the absolute deviation, although it is only based on data available at every sample step.

In Figure 11 the evolution of different quantile values of the cogging torque $T_{c,pp}^p$ are plotted together with the worst-case 90% confidence band calculated with the batch method presented within this section.

Additionally, the confidence bounds based on the quantile functions of L and U of the DKW method are illustrated by dashed lines. Especially for high quantile levels, p, the DKW method leads to much more conservative predictions of the confidence slot.

6. Discussion

What we have shown so far is a method to statistically predict the effect of manufacturing tolerances apparent in electric machine mass production, where a large number of tolerance-affected parameters are taken into account. If the performance variation of such an investigation is not satisfying, it might be interesting to investigate which of the tolerance-affected parameters has the highest impact. Unfortunately, this information cannot be deduced from the resulting CDF, $F_T$. Luckily, the method gives reasonable results with low computational effort, measured against the complexity of the problem. For example, the 2000 samples of the test case were simulated within approximately 8 h on an available HTCondor [23] cluster with around 150 workstation cores (where, of course, not all of them were used simultaneously). So, to identify which tolerance parameter has a high influence, and also which one can maybe neglected, the presented analysis can be performed for every parameter in a one-factor-at-a-time manner. For the test case, four different runs (every parameter of Table 2) would be necessary.

It should also be mentioned that for problems with a low number of tolerance-affected parameters, this method might not be the first choice. Sampling of tolerance-affected space using design of experiments, creating surrogate models upon these samples, and evaluating the influence of tolerances using the surrogate models will be more effective. The blurry and problem-dependent decision boundary for using state-of-the-art methods or the one presented here will be a topic of further investigation.

So far, most of the presented work on tolerance analysis has been about symmetric changes, such as the same change of the height of all utilized permanent magnets. This is done because the more realistic individual changes require much more computational effort and the modeling and evaluation complexity are significantly increased. The approach presented here allows for analyzing realistic scenarios that are much more likely to be present in the mass manufacturing of electric machines, while still maintaining reasonable computational cost.

By contrast, available counterparts from the literature featuring a considerable number of individual parameter changes are, for example, either based on (i) analytical models [24] or (ii) on theoretical investigations about worst-case scenarios [12]. Such approaches constitute solutions with much less required computational effort. However, they are generally not flexible, considering their application on many different machine types and corresponding designs. Besides, they are often limited to, for example, a particular performance measure, such as the worst-case one [12].

With the approach derived here, those limitations do not apply. In order to further improve the approach presented here, the authors are considering the development of a hybrid version, that is, an analysis facilitated through making use of the cumulative distribution function, which is both based on finite element-based analysis of selected samples, and consequent surrogate modeling and evaluation of further configurations. This would allow for further minimization of the computational burden, while utmost flexibility for analyzing diverse scenarios is sustained. Consequently, such further improved approaches are more likely to be applied for online robustness evaluation of designs to be evaluated within electric machine design optimization problems.

7. Conclusions and Outlook

This investigation has shown that a cumulative distribution function is suitable to statistically predict the effect of parameter uncertainties on some target functions with comparably low computational effort. More specifically, it is especially useful if there is a large number of uncertain parameters which would be hard to analyze by applying other available methods from the literature. The considered test case was about investigating the effect of manufacturing and assembling tolerances of Vernier machine permanent magnets on cogging torque. A data-based method has been presented for solving such problems. This further allows for predicting the confidence of the cumulative distribution function, which was applied to describe the statistical behavior of the cogging torque. The confidence evaluation utilizing the bootstrap method was compared with a mathematical confidence definition based on the Dvoretzky–Kiefer–Wolfowitz inequality. As was observed, the Dvoretzky–Kiefer–Wolfowitz definition follows much more conservative estimations. Future work will include the following activities:

• Testing the presented approach on different problems to show its reliability and applicability.
• Incorporating the method into a robust optimization framework.
• Trying to gain separability of the tolerance parameters’ influence on robustness—that is, to find a method to evaluate from CDF data which parameters have a crucial impact on the target, and further derive which ones have little effect and thus can be neglected.