The Cost of Privacy in Asynchronous Differentially-Private Machine
  Learning
  We consider training machine learning models using Training data located on
multiple private and geographically-scattered servers with different privacy
settings. Due to the distributed nature of the data, communicating with all
collaborating private data owners simultaneously may prove challenging or
altogether impossible. In this paper, we develop differentially-private
asynchronous algorithms for collaboratively training machine-learning models on
multiple private datasets. The asynchronous nature of the algorithms implies
that a central learner interacts with the private data owners one-on-one
whenever they are available for communication without needing to aggregate
query responses to construct gradients of the entire fitness function.
Therefore, the algorithm efficiently scales to many data owners. We define the
cost of privacy as the difference between the fitness of a privacy-preserving
machine-learning model and the fitness of trained machine-learning model in the
absence of privacy concerns. We prove that we can forecast the performance of
the proposed privacy-preserving asynchronous algorithms. We demonstrate that
the cost of privacy has an upper bound that is inversely proportional to the
combined size of the training datasets squared and the sum of the privacy
budgets squared. We validate the theoretical results with experiments on
financial and medical datasets. The experiments illustrate that collaboration
among more than 10 data owners with at least 10,000 records with privacy
budgets greater than or equal to 1 results in a superior machine-learning model
in comparison to a model trained in isolation on only one of the datasets,
illustrating the value of collaboration and the cost of the privacy. The number
of the collaborating datasets can be lowered if the privacy budget is higher.
