Privacy-Preserving Deep Learning Framework Based on Restricted Boltzmann Machines and Instance Reduction Algorithms

Abstract

The combination of collaborative deep learning and Cyber-Physical Systems (CPSs) has the potential to improve decision-making, adaptability, and efficiency in dynamic and distributed environments. However, it brings privacy, communication, and resource restrictions concerns that must be properly addressed for successful implementation in real-world CPS systems. Various privacy-preserving techniques have been proposed, but they often add complexity and decrease accuracy and utility. In this paper, we propose a privacy-preserving deep learning framework that combines Instance Reduction Techniques (IR) and the Restricted Boltzmann Machine (RBM) to preserve privacy while overcoming the limitations of other frameworks. The RBM encodes training data to retain relevant features, and IR selects the relevant encoded instances to send to the server for training. Privacy is preserved because only a small subset of the training data is sent to the server. Moreover, it is sent after encoding it using RBM. Experiments show that our framework preserves privacy with little loss of accuracy and a substantial reduction in training time. For example, using our framework, a CNN model for the MNIST dataset achieves 96% accuracy compared to 99% in a standard collaborative framework (with no privacy measures taken), with training time reduced from 133.259 s to 99.391 s. Our MLP model for MNIST achieves 97% accuracy compared to 98% in the standard collaborative framework, with training time reduced from 118.146 s to 87.873 s. Compared to other studies, our method is a simple approach that protects privacy, maintains the utility of deep learning models, and reduces training time and communication costs.

1. Introduction

Deep learning models trained on massive amounts of data may contain private and sensitive information that is subject to organizational or governmental regulations [1], resulting in serious privacy concerns related to the potential misuse or leakage of training data. Various attacks may compromise the privacy of deep learning frameworks, such as reconstructing [2] or extracting data points [3], stealing parameters [4], or constructing nearly identical models to the targeted models [5].

Collaborative deep-learning scenarios where multiple parties jointly train models further increase these concerns [6]. The privacy problems associated with collaborative deep learning become particularly pertinent in the setting of Cyber-Physical Systems (CPSs), which include the integration between computational and physical processes. CPS applications frequently deal with sensitive data from physical locations, and the collaborative nature of model training poses difficulties with safe data sharing and communication. Medical devices, for example, which are essential components of many CPS applications in healthcare, are particularly vulnerable to privacy threats. These devices deal with extremely sensitive data, such as personal health information and medical records [7]. In response to privacy concerns, privacy-preserving deep learning concepts have emerged [1]. These practices aim to preserve the privacy of training data, the trained model and its parameters, and the input and output in the prediction phase [1,8,9]. They use techniques such as Differential Privacy [1], Federated Learning [10], Homomorphic Encryption [11], Secure Multiparty Computation [12], and Trusted Execution Environments [13]. However, these solutions pose significant challenges, such as the trade-off between privacy and utility, the cost of encryption, communication and computation overheads, and the inability to handle complicated deep learning models and enormous datasets.

In this study, we propose a privacy-preserving collaborative deep learning framework that preserves the privacy of the client’s training data, the model and its parameters, and the client’s data point in the inference phase, maintains the utility of deep learning models, and reduces the training time and communication overhead. The framework combines Instance Reduction Techniques [14] and the Restricted Boltzmann Machine [15]. RBM learns the relevant features and generates encodings of input data that preserve the relevant features. Then, an Instance Reduction Technique is used to select the relevant encoded training instances to send to the central server for model training. Using our framework, clients only need to send the selected encoded instances to the server for model training rather than sending all their unencoded data. This approach ensures the complete preservation of privacy for the unsent data while also providing a certain degree of privacy protection for the subset of data that is sent. Our framework is novel, effective, and practical and can be applied to various deep-learning tasks in a privacy-preserving manner. Furthermore, it is suited to the special challenges of Cyber-Physical Systems (CPSs). It prioritizes training data privacy, fulfills real-time decision-making requirements, and allows for deep learning models’ utility in CPS applications. The framework’s goals of reducing training time and communication overhead effectively address the challenges posed by the time-sensitive and interconnected nature of CPS environments.

Our framework satisfies all privacy-preserving requirements specified and guarantees the privacy qualitative metrics [9] by guaranteeing the privacy of the training datasets by sending a small subset of the user-encoded data rather than the entire raw data. Furthermore, the same technique can be employed to protect the privacy of the input in the prediction phase. Additionally, the trained model and its parameters are protected from the training parties because no model sharing takes place.

We evaluated the proposed learning framework on three image classification benchmark datasets: MNIST [16], Fashion-MNIST [17], and EMNIST-LETTERS [18]. We compared our framework’s performance to a conventional collaborative deep learning framework that does not protect privacy. We used two neural network architectures: multilayer perceptron and convolutional neural networks [15]. Our experimental results show that our framework preserves privacy with some loss in accuracy. Also, it decreases the training time significantly compared to a conventional collaborative framework that does not protect privacy.

The main contribution of this work is the proposal of a novel paradigm in which Restricted Boltzmann Machines are used to extract salient features from training data, resulting in stochastic representations or encodings. These encodings act as a proxy for real data during deep learning model training, achieving equivalent accuracy. Additionally, our technique incorporates Instance Reduction, allowing a training entity to selectively share a portion of its encoded training data.

Our framework’s importance goes beyond privacy concerns. Extraneous noise and irrelevant instances are effectively filtered out by IR, leading to significant improvements in computer efficiency, communication time, memory use, and overall classification accuracy. In contrast to the existing literature, our approach stands out for its inherent simplicity, providing privacy preservation while preserving deep learning model efficiency, all while reducing computing and communication costs. Our study is notable for its synergistic combination of RBMs and Instance Reduction Techniques, which presents a unique framework that has not been previously studied in the literature, as far as we know. Our use of Restricted Boltzmann Machines (RBMs) to ensure privacy and security represents a novel approach. To the best of our knowledge, there are no previous studies that have utilized RBMs for security or privacy protection purposes [8].

The article is structured as follows: Section 2 introduces the key techniques used in our framework—Restricted Boltzmann Machines and Instance Reduction Techniques. In Section 3, we review related works on privacy preservation in deep learning. The heart of our contribution is in Section 4, where we present our innovative framework. Section 5 and Section 6 cover the evaluation and discussion of our results, respectively. Finally, Section 7 concludes our study and suggests directions for future research.

2. Background

2.1. Restricted Boltzmann Machines (RBMs)

We use Restricted Boltzmann Machine (RBM) models [19] to learn the relevant features before we use an IR technique to select the subset of relevant instances to send to the server. RBMs are generative models [19,20] and undirected probabilistic graphical models that are applied to a wide range of applications, including feature extraction, collaborative filtering, and image reconstruction [21]. RBMs are composed of two layers [15] as shown in Figure 1. The visible layer represents the observable variables like training data, while the hidden layer represents the latent variables of the extracted features from the visible data [22].

These two layers are linked to each other via weighted connections [19]. Furthermore, there are no links between the units of each layer [15]. The training dataset is presented to the network’s visible layer, while the network’s hidden layer learns how to represent these data in a low-dimensional probabilistic way [23]. For every pairing of visible and hidden units, the network assigns the following Boltzmann probability distribution $P(v,h)$ [19]:

$$p(v,h)=\frac{1}{z}\exp \left(-E(v,h)\right)$$

(1)

where v represents visible layer inputs, v ≡ {v₁, ..., v_n}; and h represents the hidden layer outputs, h ≡ {h₁, ..., h_m}. Z is the partition function obtained by summing all possible pairs of v and h. E is the energy of the joint configuration of $(v,h)$ given as:

$$E(v,h)=-{\displaystyle \sum _{i\in visible}{a}_i}{v}_i-{\displaystyle \sum _{j\in hidden}{b}_j}{h}_j-{\displaystyle \sum _{i,j}{v}_i}{h}_j{w}_{ij}$$

(2)

where a_i and b_j are biases, and w_ij is the weight between visible unit i and hidden unit j. RBMs use Gibbs Sampling to generate and sample units in the visible layer and hidden layer [22]. Gibbs Sampling is an algorithm that involves many steps, and in each step, a variable’s value is replaced with a value taken from its distribution that is conditioned on the values of the remaining variables [24]. RBMs are trained using the Contrastive Divergence Algorithm, which iteratively applies Gibbs sampling until convergence [25].

Restricted Boltzmann Machines can be employed to create new samples by learning the probability distribution of the visible layer $P\left(v\right).$ Also, RBMs can be used to extract features and reduce dimensionality by learning the probability distribution of the hidden layer $P\left(h\right)$ [23]. The probability distribution of these layers is given as [23]:

$$p\left(v\right)=\frac{1}{z}{\sum }_h\exp (-E(v,h\left)\right)$$

(3)

$$p\left(h\right)=\frac{1}{z}{\sum }_v\exp (-E(v,h\left)\right)$$

(4)

In this study, our focus lies on exploring the latent space representation of the input data, specifically the hidden layer’s representation. We term these representations as “encodings” due to their ability to capture and encode the crucial features present within the input instances. These encodings serve as a condensed representation that captures the salient information from the input instances, allowing us to use them for training a global model instead of using the original instances.

2.2. Instance Reduction Techniques (IR)

Instance Reduction Techniques (IR), sometimes referred to as Instance Selection Algorithms (IS), reduce the size of the training dataset by selecting which instances to keep for use in the generalization process [14]. Instance Reduction Techniques are most commonly utilized in the context of Instance-Based Learning (IBL) [26], although they are used to improve classification accuracy by removing noisy instances, speed up neural networks [27,28], and reduce excessive memory storage [29].

Instance Reduction Techniques aim to obtain a subset S from a training set, T, with the objective of excluding irrelevant instances that may be noisy or redundant. The resulting subset S should ideally preserve the overall characteristics of the original training set, ensuring that the accuracy of the learning algorithm on S (denoted as Accuracy(S)) is close to the accuracy on the full training set T (denoted as Accuracy(T)) [30].

The purpose of these techniques is to reduce the size of the training set while minimizing the impact on performance. Interestingly, it has been observed that instance reduction techniques can even enhance the classification accuracy of neural networks and reduce training time by eliminating outliers or border instances, which in turn helps to smooth the decision boundaries prior to the model training stage [27,28]. By removing these potentially misleading or noisy instances, the instance reduction techniques contribute to improving the overall quality of the training set, leading to more robust and accurate neural network models.

Various techniques for reducing the number of instances have been extensively studied and documented in the literature [14,30,31,32,33]. In this study, we select the Edited Nearest Neighbor (ENN) algorithm as our choice for instance reduction. Despite its moderate reduction rate [14], ENN is known for its efficiency, making it a suitable option for our purposes. Efficiency plays a crucial role in our study due to the large-scale nature of the datasets we use in our experiments, which are characterized by a high number of attributes. Additionally, ENN is readily available in popular open-source frameworks, ensuring its accessibility for implementation. It is important to note that this work should be seen as proof of concept, aiming to demonstrate the practicality of the proposed framework. While we have opted for the ENN algorithm, it is worth mentioning that any instance reduction algorithm can be utilized in the proposed framework, providing flexibility in the choice of techniques based on the characteristics of the datasets.

The ENN algorithm was developed by Wilson in 1972 [34]. Starting with S = T, this technique eliminates every instance in S if its class is different from the majority of its k nearest neighbors’ classes (k is often 3) [14]. ENN eliminates noisy instances as well as border instances that are close to instances from other classes [28]. Therefore, it usually increases classification accuracy while having a lower reduction rate because it keeps the internal instances. The Edited Nearest Neighbor algorithm is presented in Algorithm 1.

Algorithm 1: Edited Nearest Neighbor (T) [34]
	Data: $\mathit{T}=[{Instance}_1,\dots \dots {Instance}_m]$
	Result: Reduced dataset S
1	begin
2	$S=T$
3	for ${Instance}_i\in S$
4	find the k-nearest neighbors to ${Instance}_i$
5	find the class with the majority of points among the k-nearest neighbors.
6	(Break ties randomly)
7	if the class of ${Instance}_i$ is different from the majority class, then
8	$S=S-{Instance}_i$
9	return $S$

3. Related Work

Much work has addressed privacy-preserving in deep learning. Shokri et al. [1] define privacy objectives as protecting the privacy of the model, the privacy of the model’s output, and essentially the data used to learn the model. Additionally, they distinguish between direct and indirect data leakage. Direct leakage may occur during the training phase when training sets are revealed to other parties, as well as during the prediction phase when input data or the model output are revealed to another party. In contrast, sharing model parameters and updates across parties may result in indirect leakage.

In their work, Zhang et al. [6] explore the topic of privacy in collaborative deep learning settings. Collaborative deep learning involves two main phases: the training phase, where algorithm parameters are optimized using labeled data, and the prediction phase, which utilizes user input to make predictions. The authors classify collaborative deep learning frameworks into two modes: direct collaborative deep learning and indirect collaborative deep learning. They further investigate the privacy-preserving technologies commonly employed in collaborative deep learning and examine their respective advantages and drawbacks. According to their analysis, current approaches in the field aim to strike a balance between data privacy and utility by combining secure multi-party computing, homomorphic encryption, and differential privacy techniques. These approaches strive to provide a reasonable trade-off between preserving the privacy of data and maintaining the usefulness of the collaborative deep learning process.

In [35], Nasr et al. employ white-box inference attacks to conduct privacy analysis on deep learning models. They define privacy leakage as the information that an adversary can extract from a model regarding its training data, which cannot be obtained from other models trained on different data from the same distribution. The authors utilize inference attacks to measure the extent of privacy leakage and provide empirical evidence showing the high effectiveness of white-box membership inference attacks against well-generalized models.

In their review paper [8], Tariq et al. delve into the current advancements in deep learning with a specific focus on security and privacy. They highlight the privacy-preserving requirements that a framework should fulfill, including the non-disclosure of training party data and predictions, as well as the client’s lack of awareness of the server and global model configurations. The authors conduct a comprehensive analysis of threats and attacks related to deep learning from both the security and privacy domains. They also provide a thorough examination of existing privacy-preserving techniques and frameworks in the literature.

Furthermore, Tariq et al. emphasize the need for further research and emphasis on the development of efficient private training methods in deep learning. They point out that existing solutions proposed in the literature often come with a higher computational overhead, highlighting the importance of exploring more efficient approaches in order to achieve privacy preservation in deep learning effectively.

In their studies [9,36], Tanuwidjaja et al. provide a comprehensive overview of privacy-preserving deep learning (PPDL) in the context of Machine Learning as a Service (MLaaS). The authors also delve into traditional privacy-preserving techniques like differential privacy and cryptography and discuss their applicability to deep learning architectures. They highlight that some adjustments need to be made to combine these traditional approaches with deep learning, such as the approximation of activation functions, inclusion of batch normalization layers, and utilization of large stride convolutional layers. Additionally, Tanuwidjaja et al. critically examine the challenges and limitations faced by privacy-preserving deep learning frameworks. They identify two main issues: communication overhead and computation overhead. These challenges remain open problems in the field of Deep Learning and present significant hurdles in achieving optimal performance and privacy. It is worth noting that in this work, we address these issues, particularly the communication and computation overheads. By tackling these challenges, we contribute to advancing the field of privacy-preserving deep learning and improving the overall performance and privacy guarantees of the framework.

The following subsection reviews the related work on privacy-preserving deep learning frameworks.

3.1. Privacy-Preserving Deep Learning (PPDL)

Privacy-Preserving Deep Learning (PPDL) studies involve the integration of deep learning frameworks with various privacy-preserving techniques. The following sections provide a brief overview of the mentioned techniques and their associated challenges.

3.1.1. Works Based on Differential Privacy

Differential Privacy (DP) aims to provide strong privacy guarantees by adding noise to the data or model parameters. It ensures that the presence or absence of individual data points does not significantly affect the model’s output. It has become a common approach for preserving privacy in machine learning [37]. Shokri et al. [1] is the first work considered privacy-preserving in collaborative deep learning where clients (training parties) learn a global model without sharing their data. In [1], each party obtains parameters from the server, runs the stochastic gradient descent (SGD) algorithm on its local dataset to update the local parameters, and then selectively uploads some of the gradients to the server. Sharing part of the gradients reveals some information about the training data; hence, they use differential privacy by adding noise to the parameter’s updates. Ref. [38] proposes a differentially private stochastic gradient descent (SGD) algorithm that calculates the gradients from a random sample and clips them. After that, it computes the average, adds noise, and descents in the opposite direction of the average noisy gradients. However, in this work, the accuracy is affected by using large privacy budgets. Different approaches suggest adding noise to the objective function [39] and the data [40] rather than gradients. Other works such as [41] suggest using local Differential Privacy to address scenarios where a centralized entity for data aggregation is not feasible or trustworthy, providing a decentralized alternative to protect individual data privacy without relying on a central curator. However, they are complicated approaches that demand much computation from the training party and result in lower accuracy even with low privacy budgets.

To implement differential privacy mechanisms in deep learning, compromises are necessary to preserve utility. The choice of differential privacy variants and privacy budgets also affects the privacy and utility of machine learning models in practical applications. Jayaraman and Evans [42] examined these matters in their work and discovered a significant difference between the upper bounds of privacy loss guaranteed by differential privacy mechanisms and the actual privacy losses measured through inference attacks.

Overall, the application of differential privacy in deep learning requires careful consideration of the compromises needed to preserve utility while ensuring privacy. The choice of privacy mechanisms, privacy budget values, and evaluation of privacy losses through inference attacks are important aspects to address in privacy-preserving machine learning research.

3.1.2. Works Based on Federated Learning

Federated Learning is a decentralized computation approach, proposed by Brendan McMahan and others [10]. In this method, each data owner trains a model locally on their own dataset and then sends the model’s updates, or gradients, to a central server. The server then combines, or averages, these gradients. This approach, however, does not inherently guarantee privacy, so it is often combined with other techniques like differential privacy, secure multi-party computation, and group signature as noted by [43,44,45].

Despite numerous improvements, Federated Learning still faces several problems and challenges, as identified by Kairouz et al. [46]. They argue that many of these problems are interdisciplinary, involving not only machine learning but also areas like distributed optimization, compressed sensing, privacy and differential privacy, security and cryptography, and fairness. Therefore, for a Federated Learning framework to be superior, it needs to address these issues and understand how they intersect.

3.1.3. Works Based on Homomorphic Encryption

In homomorphic encryption (HE) [47], parties encrypt their data, and the server conducts operations on the ciphertexts and delivers the encrypted prediction to the parties. CryptoNets [11] was the first work to show how to apply homomorphic encryption to neural networks. They approximate the non-linear activation functions and transform a neural network to polynomials, and hence any polynomial function can be computed over encrypted data using homomorphic encryption [11]. CryptoDL is a similar work that approximates low-degree polynomials for activation functions and evaluates the convolutional neural networks over the encrypted data [48]. Furthermore, homomorphic encryption began to be adopted as a solution to protect the privacy of the learning process in combination with other technologies [49,50]. However, many of these works are limited to the predictive phase since the decryption and encryption processes add communication and computational overhead. To overcome the communication and computational issues, ref. [51] proposed a GPU acceleration solution to accelerate the full homomorphic evaluation of the CNN model on encrypted data. Due to the higher number of operations and the longer runtime, it does not tolerate the training stage on encrypted data.

Despite all efforts, the fundamental issue with using HE with neural networks is that they add a computational overhead to the training phase. Also, in practice, the poor approximation of the activation function might cause neural networks to perform poorly. It also generates larger encrypted messages, which consume more memory [52].

3.1.4. Works Based on Secure Multiparty Computation

Multi-party computing (MPC) protocols use cryptographic techniques including homomorphic encryption, secret sharing, and oblivious transmission to allow participants to jointly compute a function over their private inputs [53]. Traditional secure computation protocols are not easily scalable for Machine Learning (ML) applications. Therefore, many improvements have been proposed to make them suitable for training and inferencing in ML models [12,54,55]. However, the main drawback of MPC-based frameworks is the communication cost and computation overhead. Therefore, recent works [56,57,58] recommended utilizing graphics processing units (GPUs) to accelerate cryptographic operations and MPC implementation.

However, a recent work by Tramer et al. [59] doubts the privacy guarantees in multiparty computation protocols. Tramer et al. demonstrated that adversaries could increase information leakage by modifying their input even if they honestly follow the protocols. Therefore, the MPC-based frameworks fail to recognize all adversary threats in collaborative learning scenarios [59]. Moreover, combining MPC with other cryptographic primitives like differential privacy has issues such as limitations of both techniques will be applied to the hybrid framework, and the suitability of these mixed frameworks is a concern in their applications since they involve more complex assumptions and hence need more security investigation [60].

In contrast, our work eliminates these costs and overhead issues by sending a reduced subset of the encodings without too many privacy concerns. Furthermore, the RBM’s generation of the encodings has less computation overhead compared to cryptographic approaches like MPC and HE. Also, Applying the Instance Reduction Technique reduces training time.

3.1.5. Works Based on Trusted Execution Environments

Trusted execution environments (TEEs) provide secure hardware environments to isolate code and data from other software, even from the operating system and hypervisor [13]. The first attempt to employ the secure enclave is Intel’s Software Guard Extensions (SGX), which are a set of instructions and memory access mechanisms designed to secure running a trusted code on an untrusted platform [9]. The trusted code must be public to be inspected by the users [61]. Therefore, using simply SGX to protect privacy is insufficient from a security and privacy perspective, as the machine-learning service provider’s code may not be fully trustworthy [9].

Hunt et al. [61] proposed a combination of SGX and a sandbox to prevent data exfiltration, providing assurance to users that their data will not be compromised by untrusted code, even if they cannot inspect it [9]. They also introduced Chiron, a machine-learning solution within the SGX enclave, which allows data owners to train ML models without revealing their training data [61]. However, Chiron’s reliance on SGX means it is restricted to CPU and cannot utilize GPUs or other machine-learning processors. Tramèr et al. [62] expanded the use of TEEs to GPUs for machine-learning inference. They presented Slalom, a framework for evaluating deep neural networks that offloads all linear layers from a TEE, like SGX, to a GPU. While this approach enhances throughput and energy efficiency by assigning linear layers to GPU, using Slalom to train deep neural networks still presents complex, unresolved challenges [62].

In general, the fundamental problem with TEEs is their limited memory and CPU resources, which make the training of complex deep learning models a significant challenge [46]. Furthermore, there is a risk of data leakage due to side-channel attacks [9].

3.1.6. Works Based on Data Anonymization

More recent approaches use generative modeling and deep learning techniques like GAN, DGAN, AUTOENCODER, RBM, and others to anonymize data by generating synthetic data [23,63]. As a result, these synthetic data serve as an accurate representation of the distribution of the original data but are less sensitive, reducing privacy concerns [23]. The intention behind these generative models-based approaches is to enable the sharing and publishing of private data amongst various entities [63], rather than to ensure privacy in collaborative learning scenarios. To the best of our knowledge, there have been no studies that apply synthetic data generation using generative models to preserve privacy in collaborative learning scenarios or to maintain privacy in deep learning models more generally.

3.1.7. Works Based on Instance-Encoding Technique

A more recent approach to preserving privacy in deep learning is to encode the training instances before feeding them to a deep learning model. Huang et al. propose InstaHide [64], which is an instance encoding scheme for private learning. They assert that this scheme is efficient for distributed learning in federated learning, despite a reduction in accuracy. InstaHide [64] comprises two steps: the first involves mixing the images by taking the private image’s linear combination with randomly selected images from the private set (Inside dataset) or a public dataset (Cross dataset). The second step uses a random pattern of sign flips on the mixed image. Carlini et al. demonstrated in their work [65] that InstaHide fails to maintain privacy, as it does not comply with recognized privacy notions and is susceptible to privacy attacks. Moreover, they developed an attack strategy that successfully recovered source images with high similarity, thereby undermining the privacy protection offered by InstaHide.

Generally, InstaHide [64] is the work most closely related to ours, as it introduces the concept of encoding training instances before inputting them into a machine learning model, and it also proposes preserving privacy within a collaborative framework. However, as the authors themselves acknowledge, using the Mixup technique to blend instances with other instances does not guarantee privacy [64]. Additionally, random sign-flipping does not offer inherent privacy protection [65]. In contrast, our work efficiently preserves both the privacy and utility of data within collaborative deep learning frameworks.

In summary, the reviewed literature highlights significant challenges including the trade-off between privacy and utility, threat model assumptions, encryption, and communication and computation overheads, as well as the inability to handle complex deep learning models. These challenges underscore the need for a framework that safeguards privacy while addressing these issues. Our work steps in to address this gap by protecting the privacy of the training data, the model, and the prediction. Furthermore, compared to cryptographic methods like Multi-Party Computation (MPC) and Homomorphic Encryption (HE), our use of Restricted Boltzmann Machines to generate encodings introduces a significantly lower computational overhead.

4. Our Proposed Framework

In this section, we present a privacy-preserving learning framework based on Restricted Boltzmann Machines and Instance Reduction Algorithms. The goal of this framework is to preserve privacy while maintaining model accuracy and reducing the computation and communication overhead in collaborative learning settings. Our work differs from other privacy-preserving frameworks in the reviewed literature by addressing the privacy–utility trade-off, encryption, communication, and computation overheads, and the inability to handle complex deep learning models.

We assume that the deep learning framework is collaborative deep learning, where two or more training parties share their local data to jointly learn a deep learning model [6]. Each training party (user or client) applies an RBM model to their training data to encode it in a way that retains the relevant features. Then, a reduction technique is used to select a reduced subset of the encoded data to send to the server. This reduced and encoded set is sent to a central server without too many concerns about data privacy or computation and communication overhead, as the majority of the data are not sent depending on the retention rate of the reduction technique used. The central server combines the reduced datasets into one training set and then builds and trains an MLP or a CNN global model using the combined dataset. Figure 2 shows our privacy-preserving deep learning framework. The central server does not need to share any hyperparameters of the model with other parties. Similarly, in the prediction phase, the training party applies the RBM model to a data point to encode data and then sends it to a central server for prediction. The server then runs the deep learning models and returns the prediction output’s label, not the prediction score.

The server might be owned by a service provider operating in sectors such as healthcare or finance, or a Machine Learning as a Service (MLaaS) platform [9], which provides predictive services to clients. We also make the common assumption that the server and participants are honest but might be curious since they follow the same training protocol but might be curious about other participants’ training datasets. If they are not honest, then they may deviate from the protocol rules or even falsify data.

4.1. Phase 1: Encoding Instances Using Restricted Boltzmann Machine

In the first step, we use RBM to learn the hidden layer’s representation (latent representation) of the input data. We refer to these representations of the input data as “encodings” because they encode the essential features contained in the input instances, as shown in Figure 3.

The reason we do not utilize other generative models instead of RBM to generate the encodings is that, in contrast to conventional autoencoders that provide deterministic representations of the data in the hidden layer, Restricted Boltzmann Machines produce stochastic representations of the data since they learn the joint probability distribution of the visible and hidden units [25]. This stochasticity works in favor of privacy since it produces harmless noise during alternating Boltzmann sampling, making it difficult to trace the generated encodings back to the actual data points [23]. Therefore, the hidden layer learns the essential properties without being overly constrained by the samples from the training data. Furthermore, as far as we know, no work has used the mentioned method of generating the encodings of the inputs to preserve privacy.

4.2. Phase 2: Selecting a Subset of the Encoded Training Data Using Instance Reduction Techniques

Next, each client applies an instance reduction technique on the encodings produced by the RBM to obtain a reduced subset, R, from the encodings set E where R ⊂ E, such that R does not contain irrelevant (noisy or redundant) instances. The goal of using IR is to select a subset of the encoded data points to send to the central server, as shown in Figure 4.

Furthermore, removing irrelevant instances tends to improve memory usage and reduce training time and communication overhead. Employing instance reduction strategies, mitigates the impact of outliers, reducing the training time and chances of overfitting the training data [27] and the susceptibility to privacy attacks that may exploit such outliers [66].

Despite the existence of numerous instance reduction techniques proposed in the literature, with several investigated by [27] for their efficacy in creating reduced datasets for multilayer neural network training, our work exclusively employs the Edited Nearest Neighbor method to produce reduced datasets. This study should primarily be viewed as proof of concept; the focus is not on employing the best instance reduction method Instead, we aim to verify whether a central server can build a practical model using reduced subsets of encoded training data from multiple sources, without having to decode this training data. Another question we address is the server’s ability to make accurate predictions using RBM-encoded data points rather than the data points themselves.

4.3. Phase 3: Sending the Subsets of Encoded Training Data to the Central Server to Train Deep Learning Models

As illustrated in Figure 5, the reduced sets of encodings are transmitted to a central server, which amalgamates them into a single training set for use in training global deep-learning models. During the prediction phase, clients can dispatch encodings of the data points to the server. The server then operates the deep learning models and returns the label of the predicted output. Notably, there is no need for the server to decode either the training data or the query data.

4.4. Privacy of the Proposed Framework

Our framework is designed for collaborative deep learning scenarios, specifically for image classification models that involve a service provider and various client institutions, which could be in fields such as healthcare or finance. In these collaborations, we adopt a commonly assumed behavior model in data privacy research—participants are considered to be “honest-but-curious” [9]. This means participants adhere to the proposed training protocol but may have an interest in gaining additional knowledge about others’ training datasets. The reason this assumption is common in data privacy is that it reflects a realistic scenario where entities follow protocol rules but might attempt to infer more information if possible. It allows us to develop mechanisms that protect sensitive information, even when entities are curious to learn beyond what they are permitted, hence ensuring robust privacy protection.

Using our framework, we seek to guarantee the privacy of the training datasets by training a global model on subsets of users’ data after encoding them using RBM. Thus, clients do not reveal their complete raw training data to the server. Furthermore, the trained model and its parameters are protected from other clients. Similarly, in the prediction phase, the client encodes the data to be classified using the RBM model before sending it to the central server. The central server runs the deep learning models and returns the prediction output’s label, not the prediction vector. Therefore, our model satisfies the privacy-preserving framework requirements [8] where a framework preserves the privacy of the client’s training data, the model and its parameters, and the client’s data point in the inference phase.

Furthermore, common privacy violation attacks cannot be used against our framework based on the definition of our framework. Membership Inference Attacks are built based on two assumptions: the first is that overfitting in machine learning models is the cause of the high confidence score for member instances more than it would be for non-member instances [42]. The second is that the gradient distribution is more distinct for member and non-member instances [35]. In our framework, we prevent such attacks by addressing these assumptions.

We only share the output label in the inference phase; we do not share the prediction vector. Further, we prevent direct access to the model and its parameters. Therefore, it is more challenging to launch a membership attack on our framework to determine if a specific data point is included in the training set or not. The target model training set is made up of a subset of the encoded versions of the data points rather than the entire data. Also, sharing the output label and preventing direct access to the model and its parameters prevent model-stealing attacks. The model extraction attacks [5] require the obtaining of the confidence score while hyperparameter-stealing attacks [4] require obtaining access to the training data and model parameters. Similarly, attribute inference attacks are prevented since they are used with collaborative systems that enable training parties to access the models or their parameters, as in federated learning.

Furthermore, data extraction attacks are inapplicable to our frameworks because the training dataset is made up of encodings rather than real data points. If the adversary in a black box setting tries to influence the model to leak information, it might do so with the encodings rather than the actual data. Also, by using Instance Reduction Techniques, we eliminate the effect of the outliers and the near-border instances, therefore reducing overfitting [27] and, thus, susceptibility to these privacy attacks [66].

Moreover, is it possible for a malicious server or a training party to retrieve the original instance from the encoding? The answer to this is probably not, because each instance has distinct weights that are not shared with the server or any other training party. An adversary needs the same weights and biases to learn the visible units’ probability and be able to generate reconstructions of the original images.

5. Evaluation

5.1. Datasets

In our experiments, we utilize three benchmark datasets for image classification, namely, MNIST [16], Fashion-MNIST [17], and EMNIST-Letters [18]. We selected these grayscale image datasets because our work employs the Learnergy implementation of RBM [67], which is designed to handle grayscale images.

The MNIST dataset [16] comprises 60,000 training instances and 10,000 testing instances. Each instance represents a handwritten digit formatted as a 28 × 28 grayscale image. The learning objective is to classify these images into one of ten classes, corresponding to integer values ranging from 0 to 9.

Similarly, Fashion-MNIST [17] consists of 60,000 grayscale images, with the training and testing split, image size, and format identical to those of MNIST. The goal is to classify each fashion product image (e.g., dresses, coats, and shirts) into one of ten classes from 0 to 9, each representing a different fashion product.

The EMNIST-Letters dataset [18] is a collection of 28 × 28 grayscale images of handwritten letters and contains 145,600 training examples and 20,800 testing examples. The learning objective here is to classify an image into one of 26 classes, corresponding to the letters from A to Z.

Both MNIST [16] and Fashion-MNIST [17] datasets have separate test sets of 10,000 instances each, whereas the EMNIST-Letters [18] dataset has a separate test set of 20,800 instances. We use these test datasets to evaluate the performance of the global models trained by the central server.

5.2. Computing Frameworks

To develop and test our framework, we use Colab notebooks [68], which come with a single NVIDIA Tesla K80 GPU and 12 GB of memory to accelerate computations. Also, we use Learnergy [67], which is a Python package to build an energy-based machine learning to build RBMs and imbalanced-learn tools [69] to implement instance selection algorithms. Furthermore, we use Pytorch [70] to train the deep neural network models.

5.3. Evaluation Metrics

In our experiments, we utilize a variety of evaluation metrics to monitor the model’s performance throughout the training and testing phases. Our primary metrics include accuracy, F1-score, and loss. We also record the overall times for training, validation, and testing. In terms of privacy evaluation, we adopt qualitative metrics as proposed by [9].

5.4. Experimental Setup

In our privacy-preserving framework, we have six clients (data owner parties) and one server (computing party). We randomly assigned each of the six clients a portion of the total training data. The same subset is maintained by each client throughout the different experiments, while the server has the corresponding test data. We used three datasets in our empirical work: MNIST [16], Fashion-MNIST [17], and EMNIST-Letters [18].

In the first phase of our framework, each client applies a Restricted Boltzmann Machine model to their data to encode data. We first build RBM models using the Learnergy library [67].

Table 1. RBM architecture used for the datasets.

Datasets	Model Architecture	Model Parameters
MNIST	Visible_Layer_Size = 784 Hidden_Layer_Size = 576 Activation_Function = sigmoid function	epochs = 500 Gibbs’ sampling steps = 1 learning_rate = 0.05 batch_size = 10
Fashion-MNIST	Visible_Layer_Size = 784 Hidden_Layer_Size = 576 Activation_Function = sigmoid function	epochs = 500 Gibbs’ sampling steps = 1 learning_rate = 0.009 batch_size = 10
EMNIST-Letters	Visible_Layer_Size = 784 Hidden_Layer_Size = 576 Activation_Function = sigmoid function	epochs = 500 Gibbs’ sampling steps = 1 learning_rate = 0.05 batch_size = 128

shows the model architecture used for each data set. Then, each client uses and fits the built RBM model on its data to generate encodings, which are the learned hidden layer representations of the input data. In each RBM model, we set the size of the visible layer equal to input image 784 and the size of the hidden layer to 576 to generate 24 × 24 encodings. We also employed the same RBM architectures on the server to encode the test data. Of course, the training parties must use the same RBM architecture.

The next phase of our framework that follows the use of RBM is using an Instance Reduction Technique. Each client uses an Instance Reduction Technique to select a reduced set of data encodings to send to the server. We use the Edited Nearest Neighbor algorithm as implemented in the imbalanced-learn library [69]. The neighbor parameter is set to 3.

Finally, the reduced sets of encodings are used by a central server to train global models. After combining the reduced sets of encodings from all the clients into a single global training dataset on the server, we randomly divided the data into training and validation sets. The validation set represents 20% of the training set. We use the validation set to tune the model. We train two types of classifiers for the global model (trained by the server): multilayer perceptron (MLPs) and convolutional neural networks (CNNs).

Table 2. CNN Architecture for MNIST and Fashion-MNIST datasets.

Model Architecture	Conv1→relu→pool→Conv2→relu→pool→Conv3→relu→pool→dropout→fc1→relu→dropout→fc2 Conv1: 16 filters, size 3 × 3, padding = 1 Conv2: 32 filters, size 3 × 3, padding = 1 Conv3: 64 filters, size 3 × 3, padding = 1 pool: size = 2 × 2, stride = 2 dropout:(0.25) fc1: linear layer (576 -> 500) fc2: linear layer (500, 10)
Model Parameters	criterion = CrossEntropyLoss optimizer = SGD lr = 0.01 epochs = 50 batch_size = 10 valid_size = 0.2

and

Table 3. MLP Architecture for MNIST and Fashion-MNIST datasets.

Model Architecture	FC1→relu→dropout→FC2→relu→dropout→FC3 fc1: linear layer (784 -> 512) fc2: linear layer (512 -> 512) fc3: linear layer (512 -> 10) dropout: (0.2)
Model Parameters	criterion = CrossEntropyLoss optimizer = SGD lr = 0.01 epochs = 50 batch_size = 10 valid_size = 0.2

show the CNN and MLP model architectures and training parameters for the MNIST and Fashion-MNIST, while

Table 4. CNN Architecture for the EMNIST-Letters datasets.

Model Architecture	Conv1→relu→pool→Conv2→relu→pool→Conv3→relu→pool→dropout→fc1→relu→dropout→fc2 Conv1: 16 filters, size 3 × 3, padding = 1 Conv2: 32 filters, size 3 × 3, padding = 1 Conv3: 64 filters, size 3 × 3, padding = 1 pool: size = 2 × 2, stride = 2 dropout: (0.25) fc1: linear layer (576 -> 500) fc2: linear layer (500, 27)
Model Parameters	criterion = CrossEntropyLoss optimizer = SGD lr = 0.01 epochs = 50 batch_size = 128 valid_size = 0.2

and

Table 5. MLP Architecture for the EMNIST-Letters datasets.

Model Architecture	FC1→relu→dropout→FC2→relu→dropout→FC3 fc1: linear layer (784 -> 512) fc2: linear layer (512 -> 512) fc3: linear layer (512 -> 27) dropout: (0.2)
Model Parameters	criterion = CrossEntropyLoss optimizer = SGD lr = 0.01 epochs = 50 batch_size = 128 valid_size = 0.2

show the model architectures for the EMNIST-Letters dataset.

6. Results and Discussion

We investigate and compare two scenarios: our proposed privacy-preserving collaborative framework, which we refer to as the ‘privacy-preserving collaborative scenario’, and a conventional collaborative learning scenario that does not protect privacy. We evaluate these two frameworks based on accuracy, loss, F1-score, and overall training time. Furthermore, we statistically evaluated if our results were significant using the Wilcoxon signed-rank test [71].

In a collaborative scenario that does not protect privacy, each client sends all of their available training data to the server. These data often contain sensitive details, thereby violating privacy. The server then combines the raw data from the clients and uses it to train a global model.

Table 6. The sizes of the datasets used in the conventional collaborative models experiments.

	Training Data Size	Test Data Size
MNIST	60,000	10,000
Fashion-MNIST	60,000	10,000
EMNIST-Letters	145,600	20,800

describes the sizes of the datasets used in the experiments.

As shown in

Table 7. The performance of CNN and MLP on MNIST, Fashion-MNIST, and EMNIST-Letters datasets in the conventional collaborative models experiments.

Dataset	CNN								MLP
	Training		Validating		Test				Training		Validating		Test
	Accuracy	Time	Accuracy	Time	Accuracy	Loss	F-Score	Time	Accuracy	Time	Accuracy	Time	Accuracy	Loss	F-Score	Time
MNIST	98.59%	133.25 s	99.06%	7.68 s	99%	0.023	99.27%	8.07 s	98.56%	118.14 s	97.74%	5.33 s	98%	0.056	98.15%	5.98 s
Fashion-MNIST	90.88%	133.37 s	90.35%	8.34 s	91%	0.236	91.39%	9.3 s	90.83%	109.86 s	88.58%	5.78 s	89%	0.315	89.45%	5.35 s
EMNIST-Letters	86.19%	82.59 s	89.13%	10.57 s	93%	0.193	93.65%	10.62 s	79.45%	25.65 s	81.81%	3.93 s	89%	0.344	89.29%	3.40 s

, the CNN model achieved 99% accuracy on the MNIST test set, 91% on the Fashion-MNIST, and 93% on the EMNIST-Letters test set, while the MLP model achieved 98% on the MNIST test set, 89% on the Fashion-MNIST, and 89% on the EMNIST-Letters test set. Furthermore, Table 7 reports the training and testing time of the models on the different datasets. The training time for the CNN model ranged between 82 and 133 s, while for the MLP model, it ranged between 25.6 and 118 s.

In general, this conventional collaborative scenario represents a server-based framework or direct collaborative deep learning [6]. In this approach, each user uploads a local dataset to the central server during training to train the deep learning model. Since this framework uses all training data, it is expected to achieve the best classification performance. However, it comes with privacy risks; this scenario is susceptible to direct privacy leakage [1] by exposing data to another party. Moreover, as it transfers all client data to the server, it may result in significant network overheard.

In our privacy-preserving learning framework, the server gathers the encoded data sent by all clients and trains a global model.

Table 8. The sizes of the datasets used in the privacy-preserving collaborative models experiments.

	Training Data Size	Percentage of the Reduced Training Data	Test Data Size
MNIST	54,351	90.58%	10,000
Fashion-MNIST	41,355	68.92%	10,000
EMNIST-Letters	83,282	57.19%	20,800

displays the sizes and percentages relative to the full dataset sizes of the reduced encoded datasets, reduced by the ENN algorithm, and used to train and test privacy-preserving collaborative models.

Table 9. The performance of CNN and MLP on MNIST, Fashion-MNIST, and EMNIST-Letters datasets in the privacy-preserving collaborative models experiments.

Dataset	CNN								MLP
	Training		Validating		Test				Training		Validating		Test
	Accuracy	Time	Accuracy	Time	Accuracy	Loss	F-Score	Time	Accuracy	Time	Accuracy	Time	Accuracy	Loss	F-Score	Time
MNIST	97.56%	99.39 s	97.69%	5.30 s	96%	0.175	96.43%	7.24 s	99.44%	87.87 s	99.44%	3.19 s	97%	0.108	97.40%	3.99 s
Fashion-MNIST	96.41%	68.51 s	96.65%	4.40 s	84%	0.637	84.17%	7.14 s	97.48%	57.85 s	97.29%	2.85 s	85%	0.626	85.14%	4.51 s
EMNIST-Letters	73.94%	35.24 s	77.87%	4.15 s	82%	0.673	82.91%	9.80 s	89.56%	9.18 s	91.72%	1.00 s	85%	0.530	85.60%	2.17 s

shows that the MLP model in the proposed framework achieved 97% accuracy on the MNIST test dataset, down from 98%. It also reached 85% on the Fashion-MNIST dataset, a decrease from 89%, and 85% on the EMNIST-Letters test set, also down from 89%. Furthermore, Table 9 indicates that the CNN model in the proposed framework achieved 96% accuracy on the MNIST test set, down from 99% in the conventional collaborative framework. It also achieved 84% on the Fashion-MNIST, down from 91% in the conventional framework, and 82% on the EMNIST-Letters test set, down from 93%. The larger gap between the CNN and MLP models in the two frameworks may be attributed to the fact that CNN, as a deep learning method, can make better use of the training data. Consequently, any reduction in the training data has a more significant impact on the CNN models compared to the MLP models. However, this trade-off in accuracy may be considered acceptable in sensitive domains where privacy is a primary concern.

Our results indicate that the MLP models were less impacted by the proposed framework, which uses subsets of data encoded using RBM. One possible explanation for this is that CNN incorporates a robust feature detection mechanism that generally requires large datasets to function effectively. While RBM does select features, its selection may not be as effective for classification tasks because it is not specifically oriented toward them. RBM focuses on identifying features that are necessary for data reconstruction, rather than those that are most relevant for classification. Consequently, when CNN is trained on only a subset of the data, it may fail to accurately identify all relevant features.

It is worth noting that the proposed framework reduces training time. Table 9 also summarizes the training and testing times for the models across different datasets. For example, training the CNN model on the MNIST dataset with the proposed framework takes 99.391 s, down from the 133.25 s required by the conventional collaborative framework (see Table 7 and Table 9). Similarly, training times for the CNN model on the Fashion-MNIST and EMNIST-Letters datasets are reduced to 68.51 and 35.24 s, respectively, down from 133.37 and 82.59 s in the conventional framework. Table 7 and Table 9 also show a similar reduction in training time for the MLP models. Figure 6 and Figure 7 show the percentage of training time using the reduced datasets in the proposed framework relative to the total time taken using the complete datasets in the conventional framework (which does not protect privacy).

Through the experiments and the results, we found that our framework preserves privacy with some loss in classification accuracy. In addition to preserving accuracy, the proposed framework proved to significantly reduce the models’ training time. Furthermore, it reduces the network overhead since it sends subsets of the data instead of the whole data.

Our results indicate that the use of the ENN algorithm in our framework significantly improves training time, a finding consistent with previous studies [27,28,29]. However, reducing the size of the training dataset does slightly impact the accuracy of our deep learning models. This is noteworthy because some Instance Reduction Techniques that remove noisy and borderline instances have been found to improve classification accuracy [27,28]. The divergence in outcomes might be attributable to our use of subsets of encoded data, while previous studies used the original, unencoded data. Additionally, it is worth noting that those studies focused on simpler neural network architectures, making their findings less directly applicable to deep learning models that typically require large datasets for effective learning. Nonetheless, this limitation may be less serious than it appears, as collaborative learning often involves many clients—significantly more than the six we used in our experiments—thus making ample training data available.

Comparing our work with other Privacy-Preserving Deep Learning (PPDL) frameworks in terms of performance presents a challenge. Various studies may be specific to a particular application or domain, diverge in their implementation of privacy-preserving technology, differ in their learning framework configurations, utilize distinct deep learning architectures, or employ different datasets and parameters. Additionally, some studies assess the performance of different approaches by citing results from relevant papers, rather than re-executing the code on the same datasets [9]. However, we opted not to use this method for comparing the performance of privacy-preserving frameworks.

In general, the existing Privacy-Preserving Deep Learning (PPDL) frameworks suffer from many performance issues like an unacceptable utility–privacy trade-off resulting from the practical implementations of differential privacy techniques [42]. Also, significant computation cost and communication overhead result from using many of the MPC-based and HE-based frameworks. For instance, the MPC-based framework detailed in [72] requires 80.5 h to train a simple neural network with two hidden layers on the MNIST dataset, largely due to extensive communication and interactions. Similarly, the use of Homomorphic Encryption (HE) increases communication overhead by generating large, encrypted representations of training instances [8].

In contrast, our framework minimizes communication overhead by transmitting a reduced subset of encodings, which are smaller than the original training sets. Additionally, the computational overhead for generating these encodings with RBMs and applying the ENN algorithm is significantly lower than that of cryptographic methods like MPC and HE. For example, encoding 10,000 images from the MNIST dataset [16] using our RBM model takes each client an average of 243.8 ms, while encoding 10,000 images from Fashion-MNIST [17] takes approximately 272.3 ms. Further, applying the ENN algorithm to these 10,000 MNIST encodings takes an average of 6.28 s, and for Fashion-MNIST encodings, it takes an average of 7.63 s.

To analyze and evaluate the privacy of our framework, we employ qualitative metrics, as suggested by [9]. These metrics include Privacy of the Client (PoC), Privacy of the Model (PoM), and Privacy of the Result (PoR). PoC ensures that neither the server nor other clients can access individual client data. PoM ensures that clients remain unaware of the deep learning model and its parameters, while PoR ensures that clients cannot access information about predicted outcomes. Our framework guarantees all three privacy metrics. We secure the privacy of training datasets by training global models on subsets of user-encoded data rather than the complete raw data. While it is possible that a determined server could decode this subset, our framework, like most privacy-preserving frameworks, assumes that participating parties are honest but may be curious. Also, the server is not required to disclose the trained model or its parameters to the training parties, also known as the donors of the training data. This ensures the Privacy of the Model (PoM). During the inference phase, clients maintain the privacy of their input data by encoding it using a Restricted Boltzmann Machine before sending it to the central server for a prediction. Additionally, we add an extra layer of protection to the prediction outcome: the server only sends back the prediction label without including the confidence score, thereby mitigating the risk of black-box privacy attacks that often leverage such scores.

The level of privacy achieved is directly proportional to the reduction rate enabled by the instance reduction technique employed. This makes it important to explore more effective reduction algorithms with higher reduction rates, such as DROP3 [14]. However, such algorithms often require substantial computational resources, especially when applied to image datasets with numerous attributes. Improving the efficiency of these algorithms and developing new, more efficient reduction methods remains an open area for future research.

Table 10. Privacy comparison [9].

Ref.	The Used Privacy Technique	PPDL Frameworks	Privacy Metrics
			PoC	PoM	PoR
[72]	MPC	SecureML	-	√	√
[73]	MPC	MiniONN	-	√	√
[74]	MPC	Chameleon	-	√	√
[54]	MPC	DeepSecure	-	√	√
[75]	MPC	ABY3	-	√	√
[55]	MPC	SecureNN	-	√	√
[76]	HE	CryptoNets	√	-	√
[48]	HE	CryptoDL	√	-	√
[61]	TEEs	Chiron	√	√	√
[62]	TEEs	Slalom	√	√	√
[64]	Instance Encodings	InstaHide	√	-	√
-	Instance Encodings	Our framework	√	√	√

compares our framework with some of the Privacy-Preserving Deep Learning (PPDL) works surveyed by [9], using qualitative privacy metrics (PoC, PoM, and PoR). In Table 10, the results listed above the bold line are those provided by [9], while the last two rows are for Instance Encodings [64] and our framework.

From Table 10, we observe that our framework and Trusted Execution Environments (TEE) frameworks satisfy all the privacy qualitative metrics compared to other frameworks. In contrast to our framework, TEEs have restricted memory and CPU resources, making it challenging to train complex deep-learning models [46]. Also, the table shows that the secure MPC-based frameworks fail to ensure clients’ data privacy (PoC), while the HE-based frameworks fail to ensure model privacy (PoM). Additionally, we compare our work against InstaHide [64] since it proposes a similar idea of encoding the training instances before using them to train machine learning models. Nevertheless, InstaHide [64] does not guarantee model privacy since it is based on federated learning that shares parameters. Further, it does not guarantee the client’s data privacy as claimed by the authors, since it is vulnerable to privacy attacks that recover the training data [65].

In summary, our framework protects the learning framework privacy by satisfying all privacy metrics and maintaining utility. This approach ensures that clients do not need to disclose their entire raw training data to the server, but only subsets of them. Moreover, only the relevant features of subsets that are extracted using an RBM are sent to the server. Furthermore, our framework extends its privacy protection to the trained model and its parameters, safeguarding them from exposure to other clients, as the server does not need to share them with the clients. During the prediction phase, the clients encode the data for classification using the RBM model before transmitting it to the central server. The central server then executes the deep learning models, returning only the predicted output label to the client.

It is evident that our model adheres to the privacy-preserving framework requirements outlined by Tariq et al. [8]. These requirements focus on preserving the privacy of client training data, the model and its parameters, and individual client data points during the classification phase.

Furthermore, it reduces communication costs by sending a reduced subset of encodings that are smaller than the original images. Additionally, the use of RBMs for data encoding and ENN for data reduction incurs lower computational overhead compared to cryptographic techniques like Multi-Party Computation (MPC) and Homomorphic Encryption (HE). Accordingly, we posit that our work fills a notable gap in the existing literature. It offers a comprehensive solution that addresses the privacy-utility trade-off, encryption challenges, and overheads in communication and computation, while also accommodating the demands of complex deep learning models.

On the other hand, our work, like most research initiatives, has some limitations that need to be acknowledged. One significant constraint is that the framework is currently designed to work only with grayscale image datasets. This limitation is a direct consequence of utilizing Restricted Boltzmann Machines in our methodology. Furthermore, our reliance on Google Colab [68] for development and evaluation limited our computational resources. The platform’s inherent restrictions also prevented us from accurately assessing the actual communication time between clients and the server.

7. Conclusions

This study aims to propose and evaluate a privacy-preserving deep learning framework that combines Instance Reduction Techniques with the Restricted Boltzmann Machine to preserve privacy in collaborative deep learning frameworks and overcome the limitations of other privacy-preserving solutions. In such a framework, each participating party applies a Restricted Boltzmann Machine model to their data, encoding it in a manner that retains the important features relevant for training. Subsequently, a reduction technique (specifically, ENN) is used to select a subset of the data to send to the central server. These reduced sets of encodings are transmitted without concerns about data privacy or computational and communication overheads. Experimental evaluations on MNIST [16], Fashion-MNIST [17], and EMNIST-Letters [18] datasets using CNN and MLP architectures compare the proposed framework with a conventional collaborative learning scenario. The results show that our framework preserves privacy with a little loss of accuracy and a large reduction in training time. The study highlights the framework’s suitability for privacy-preserving collaborative deep learning in real-world applications, emphasizing its potential in sectors such as healthcare, finance, and smart security and surveillance systems, where both privacy and scalability are of utmost importance. For future work, we aim to address framework limitations, optimize reduction algorithms for diverse datasets, enhance model performance, and deploy the framework in real-world applications for comprehensive evaluation.