Test-time Collective Prediction
  An increasingly common setting in machine learning involves multiple parties,
each with their own data, who want to jointly make predictions on future test
points. Agents wish to benefit from the collective expertise of the full set of
agents to make better predictions than they would individually, but may not be
willing to release their data or model parameters. In this work, we explore a
decentralized mechanism to make collective predictions at test time, leveraging
each agent's pre-trained model without relying on external validation, model
retraining, or data pooling. Our approach takes inspiration from the literature
in social science on human consensus-making. We analyze our mechanism
theoretically, showing that it converges to inverse meansquared-error (MSE)
weighting in the large-sample limit. To compute error bars on the collective
predictions we propose a decentralized Jackknife procedure that evaluates the
sensitivity of our mechanism to a single agent's prediction. Empirically, we
demonstrate that our scheme effectively combines models with differing quality
across the input space. The proposed consensus prediction achieves significant
gains over classical model averaging, and even outperforms weighted averaging
schemes that have access to additional validation data.
