A new way to look at data privacy.

Imagine that a group of researchers has created a machine-learning algorithm that can determine from lung scan pictures if a patient has cancer. In order for professionals to begin using this model for diagnosis, they want to distribute it to hospitals all around the world.

However, there is an issue. They trained their model by exposing it to millions of actual lung scan pictures in order to teach it how to anticipate cancer. It is possible for a hostile agent to retrieve the sensitive material that has been encoded into the model's internal workings. Scientists may avoid this by making it more difficult for an opponent to predict the original data by adding noise, or more general randomness, to the model. But perturbation decreases a model's precision, so the less noise one can add, the better.
A method created by MIT researchers allows users to possibly provide the least amount of noise while yet guaranteeing that sensitive data is preserved.

The researchers developed a system based on a novel privacy metric they call Probably Approximately Correct (PAC) Privacy that can automatically estimate the least amount of noise that has to be added. Additionally, this framework is simpler to utilize for many models and applications since it does not require understanding of a model's internal workings or training procedure.

In a number of instances, the researchers demonstrate how much less noise is needed with PAC Privacy compared with other strategies to secure sensitive data from attackers. This could aid engineers in developing machine-learning algorithms that can indubitably conceal training data while retaining accuracy in practical contexts.

"PAC Privacy meaningfully uses the entropy or unpredictability of the sensitive data, allowing us to add, in many circumstances, an order of magnitude less noise. With the help of this framework, we can automatically privatize arbitrary data processing without making any unnatural changes. Although we are still in the early stages and working with simple cases, Srini Devadas, the Edwin Sibley Webster Professor of Electrical Engineering and co-author of a recent publication on PAC Privacy, is enthusiastic about the potential of this method.

Together with graduate student in electrical engineering and computer science Hanshen Xiao, Devadas co-authored the study. The study will be presented at Crypto 2023, the International Cryptography Conference.

Defining privacy

How much sensitive data may an adversary retrieve from a machine-learning model with noise introduced is a basic concern in data privacy.
According to the differential privacy definition of privacy, privacy is attained when an adversary seeing the model after it has been released cannot determine if any random person's data was utilized for the training process. However, it frequently takes a lot of noise to obfuscate data consumption to prove that an adversary cannot differentiate it. The precision of the model is decreased by this noise.


PAC Privacy approaches the issue quite differently. Instead than concentrating just on the distinguishability issue, it specifies how difficult it would be for an adversary to reconstruct any portion of randomly selected or created sensitive data once noise has been introduced.

For instance, differential privacy would concentrate on whether the adversary could discern if a specific person's face was in the dataset if the sensitive data were photos of human faces. On the other hand, PAC Privacy may examine if an opponent could extract a silhouette—an approximate—that someone could identify as a specific person's face.

In order to prevent an adversary from confidently recreating a close approximation of the sensitive data, the researchers developed an algorithm that automatically instructs the user how much noise to add to a model. According to Xiao, this technique ensures anonymity even if the attacker has unlimited computational power.


The PAC Privacy algorithm uses the entropy, or uncertainty, in the initial data from the adversary's point of view to determine the ideal quantity of noise.
The user's machine-learning training algorithm is performed on the subsampled data using this automated approach, which randomly selects samples from a data distribution or a huge data pool to create an output learnt model. This is repeated on several subsamples, and the variance of all outputs is compared. The variance dictates the amount of noise that must be added; a lower variance indicates a need for less noise.

Algorithm advantages

Different from other privacy approaches, the PAC Privacy algorithm does not need knowledge of the inner workings of a model, or the training process.

When implementing PAC Privacy, a user can specify their desired level of confidence at the outset. For instance, perhaps the user wants a guarantee that an adversary will not be more than 1 percent confident that they have successfully reconstructed the sensitive data to within 5 percent of its actual value. The PAC Privacy algorithm automatically tells the user the optimal amount of noise that needs to be added to the output model before it is shared publicly, in order to achieve those goals.

"The noise is ideal in that if you add less than we advise, everything might go wrong. However, the impact of adding noise to neural network parameters is complex, and we cannot guarantee that the model's utility would not decline as a result of the added noise, according to Xiao.

This highlights a drawback of PAC Privacy: the method doesn't inform the user of the accuracy loss the model would experience after noise is introduced. PAC Privacy can be computationally costly since it requires repeatedly training a machine-learning model on numerous data subsampling's.

One method for enhancing PAC Privacy involves altering a user's machine-learning training procedure so that the output model it generates does not change much when the input data is subsampled from a data pool. The PAC Privacy algorithm would need to run fewer times to determine the ideal amount of noise, and it would also need to add less noise as a result of this stability, which would result in less deviations between subsample outputs.


Devadas continues, "An additional advantage of more stable models is that they frequently have less generalization error, meaning they can make more accurate predictions on data that hasn't been seen before. This is a win-win situation for machine learning and privacy."

We would love to go a bit more deeply into the connections between stability and privacy in the next years, as well as the connections between privacy and generalization mistake. Although it is not yet evident where the door goes, we are pounding on it here.


"The key to safeguarding an individual's privacy is to obscure how their data is used in a model. However, doing so would reduce the usefulness of the data and, consequently, of the model, warns Jeremy Goodsitt, a senior machine learning engineer at Capital One who was not involved in the study. "PAC offers an empirical, black-box method that, while providing comparable privacy assurances, can cut down on the extra noise compared to present techniques. Additionally, its empirical methodology broadens its scope.

This research is funded, in part, by DSTA Singapore, Cisco Systems, Capital One, and a MathWorks Fellowship.