{'fraction_non_alphanumeric': 0.029995175799540284, 'fraction_numerical': 0.0077754760350747755, 'mean_word_length': 4.5592364726297525, 'pattern_counts': {'":': 0, '<': 1, '<?xml version=': 0, '>': 0, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Contextual HyperNetworks for Novel Feature Adaptation

While deep learning has obtained state-of-the-art results in many applications, the adaptation of neural network architectures to incorporate new output features remains a challenge, as neural networks are commonly trained to produce a fixed output dimension. This issue is particularly severe in online learning settings, where new output features, such as items in a recommender system, are added continually with few or no associated observations. As such, methods for adapting neural networks to novel features which are both time and data-efficient are desired. To address this, we propose the Contextual HyperNetwork (CHN), an auxiliary model which generates parameters for extending the base model to a new feature, by utilizing both existing data as well as any observations and/or metadata associated with the new feature. At prediction time, the CHN requires only a single forward pass through a neural network, yielding a significant speed-up when compared to re-training and fine-tuning approaches. To assess the performance of CHNs, we use a CHN to augment a partial variational autoencoder (P-VAE), a deep generative model which can impute the values of missing features in sparsely-observed data. We show that this system obtains improved few-shot learning performance for novel features over existing imputation and meta-learning baselines across recommender systems, e-learning, and healthcare tasks.


Introduction
In many deep learning application domains, it is common to see the set of predictions made by a model grow over time: a new item may be introduced into a recommender system, a new question may be added to a survey, or a new disease may require diagnosis. In such settings, it is valuable to be able to accurately predict the values that this feature takes within data points for which it is unobserved: for example, predicting whether a user will enjoy a new movie in a recommender system, or predicting how a user will answer a new question in a questionnaire.
On the introduction of a new feature, there may be few or no labelled data points containing observed values for it; a newly added movie may have received very few or even no ratings. The typically poor performance of machine learning models in this low-data regime is often referred to as the cold-start problem [31,18,16], which is prevalent not only in recommender systems but also in applications where high quality data is sparse. This presents a key challenge: the adaptation of a deep learning model to accurately predict the new feature values in the low data regime. On one hand, it is often required to deploy the model in applications immediately upon the arrival of new features, so it is impractical for the adaptation to wait until much more data has been acquired. On the other hand, simply retraining the model every time a new feature is introduced is computationally costly, and may fall victim to severe over-fitting if there are only a small number of observations available for the new feature.
Few-shot learning [33,28,37,9] has seen great successes in recent years, particularly in image classification tasks; however, these approaches typically treat all tasks as independent of one another. We wish to extend these ideas to the challenge of extending deep learning models to new output features, using a method which captures how a new feature relates to the existing features in the model. Furthermore, we seek a method that is computationally efficient, ideally requiring no fine-tuning of the model, and that is resistant to over-fitting in the few-shot regime.
To address these needs simultaneously, our contributions are as follows: • We propose an auxiliary neural network, called Contextual HyperNet (CHN), that can be used to initialize the model parameters associated with a new feature (see Section 2). CHNs are conditioned on both a context set made up of observations for the new feature, and any associated content information or metadata. CHNs amortize the process of performing gradient descent on the new parameters by mapping the newly observed data directly into high-performing new parameter values, with no additional fine-tuning of the model being required. This makes CHNs highly computationally efficient and scalable to large datasets. • We use a CHN to augment a partial variational autoencoder (P-VAE) and evaluate the system's performance across a range of applications (see Section 4). While CHNs are applicable to a wide range of deep learning models, in this work we choose a P-VAE as the evaluation framework. The result is a flexible deep learning model able to rapidly adapt to new features, even when the data is sparsely-observed, e.g. in recommender systems. We show that this model outperforms a range of baselines in both predictive accuracy and speed across recommender system, e-learning and healthcare tasks.

Problem Setting
Our goal is to enable fast adaptation of a machine learning model when new output features are added to augment the originally observed data. Specifically, we consider the original observations as a set of vector-valued data points D = {x (i) } m i=1 , where each of the feature values in a given data point may be missing. We denote x j as the jth feature of a data point x ∈ D and group the observed and unobserved features within a data point as x = [x O , x U ]. In many scenarios, such as recommender systems, a machine learning model p(x U |x O ) aims then at predicting the unobserved features x U given observed ones x O . Now suppose a new output feature x n becomes available, so that each data vector x ∈ D is augmented to becomex = [x; x n ]. This happens when e.g. a new item is added to a recommender system, or a new type of diagnostic test is added in a medical application. We note that not every data point x ∈ D receives an observed value for the new feature: a newly added movie may have received very few ratings, or a new diagnostic test may have yet to be performed on all of the patients. We refer to the set of data points where the new feature is observed as the context set for the new feature n, i.e.
The context set is shown in yellow in Figure 1. Its complement, the target set T n , is the set of those data points for which there is no associated observation for the feature: One can also split the augmented data into observed and unobserved parts, i.e.x = [x O ,x U ]. Using this notation, it is clear x ∈ T n . In addition, we may also have access to some metadata M n describing the new feature. This could be categorical data such as the category of a product in a recommender system or the topic of a question in an e-learning system, or some richer data format such as images or text. We wish to adapt the machine learning model However, in deep neural networks, the number of model parameters θ may be extremely large, so that maximising this log-likelihood is very expensive, particularly if new features are being introduced on a regular basis. Furthermore, optimising θ for one particular feature may lead to poor performance for another, as is the case in catastrophic forgetting [15] in continual learning tasks. In order to address both of these concerns, we divide the model parameters into parameters θ 0 inherent from the old model, and feature-specific parameters θ n associated solely with the new feature. In other words, we use p θ0 (x U |x O ) as a base model and pose a factorisation assumption on the augmented model as , which together yield a predictive model for the new feature. We then hold θ 0 fixed and only seek MLEs for θ n . While this greatly reduces the dimensionality of the parameter space over which we optimize for a new feature, and decouples the optimization of parameters for one new feature from another, several issues still exist. This factorization still requires a gradient descent procedure, which can be computationally costly and risks severe overfitting when there is little data for the new feature. Furthermore, it is not immediately clear how to make the estimation of θ n depend on the feature metadata M n . To address these problems, we introduce a Contextual HyperNetwork (CHN) H ψ (C n , M n ), an auxilliary neural network that amortizes the process of estimating θ n . The goal is that when a new feature x n * is added at test time, the CHN will directly generate "good" parametersθ n * = H ψ (C n * , M n * ) such that the new predictive model p θ0 (x n * |x O ; θ n * =θ n * ) can predict the values of the new feature accurately.

Contextual HyperNetworks
CHNs aim to map the context set C n and metadata M n into an estimate of the new model parameters θ n . Since the size of C n is variable for each feature, CHNs require an architecture that is able to adapt to a varying input dimension. This challenge is addressed through the use of a PointNet-style set encoder [27,40]. For each context pointx (i) ∈ C n , we concatenate the new feature x ... to produce a single, fixed-length vector. This output is passed through a second neural network g(·) to produce a fixed-length encoding of c n we term a "context vector". This architecture is displayed in Figure 2a.
The fixed length encoding z (i) of the observed features x Additionally, any feature metadata M n is passed through a neural network h(·) to produce a fixed length metadata embedding vector m n . In the case of image or text metadata, specialized architectures such as convolutional or sequence models can be used here. The concatenated vector [c n ; m n ] is then input into a final feed-forward neural network which outputs the new feature-specific parameters θ n . CHNs can be applied to any predictive model with dynamically added output features: Figure 2b shows the application of CHN to a predictive model. As an example, we illustrate how CHN is used with an autoencoder-style model in Figure 3. Since it is possible to parallelize the encoding of the context set C n both across each datapoint x (k) and across the observed features within each datapoint x i , CHNs are able to scale efficiently in both the context set size and the number of observed values within each datapoint in the context set. These properties, combined with the lack of any costly fine-tuning procedure, make CHNs an extremely efficient choice for parameter initialization.

Training CHNs with Meta-Learning
We adopt a meta-learning approach to training the CHN, treating the prediction of the values of each new feature as an individual task, with the aim of producing a model that can "learn how to learn" from C n and M n . First, a base model p θ0 (x U |x O ) is trained on the data observed before the adaptation stages; this model is then frozen during CHN training. To implement the training strategy, in the experiments we divide the dataset into three disjoint sets of features (see Figure 4): a 'training' set for base model training in the first stage, a'meta-training' set for CHN meta-learning in the second stage, and a meta-test set for CHN evaluation in the third stage.
Meta-Training of the CHN During meta-training, the parameters θ 0 of the base model are frozen, and we learn the parameters ψ of the CHN. We iterate the following training steps in mini-batches of features B sampled from the meta-training set for every step: 1. For each feature n in B, sample k n data points in which this feature is observed to form the context set C n , and reveal the associated feature values to the model. In our experiments we sample k n ∼ Uniform[0, ..., 32] to ensure that a single CHN can perform well across a range of context set sizes. 2. For each feature n ∈ B, compute feature-specific parameter predictions using the CHN, 3. For each feature n ∈ B, estimate the log-likelihood of the CHN parameters ψ given the ground truths for the hidden values of the feature n in the data points in its target set T n , using the augmented model p θ0 (x n |x O ,θ n ): 4. Update the CHN parameters by taking a gradient ascent step in ψ for l(ψ).
Note that the log-likelihood is only computed for the hidden values of the new feature in the target set T n , and not for the observed values in C n -this is to ensure that the CHN produces parameters which generalize well, rather than overfitting to the context set. This approach is consistent with many meta-learning methods such as MAML [4], where the meta-learning model is updated based on its performance on a "test set" of previously unseen examples for each new task. : Data splits when meta-training a CHN. Meta-testing proceeds analogously, using features from an additional meta-test set of features, but using a constant set of observed values for each feature on every meta-test set evaluation.
Evaluating the CHN At evaluation time, the parameters of both the base model and the CHN are now frozen. A fixed context set and metadata are provided for each feature in the meta-test set, and these are used to initialize feature-specific parameters for the meta-test features using the trained CHN. These parameters are then used to make predictions for all of the target set values for the new features, from which evaluation metrics are computed.

Related Work
CHNs aim to solve the problem of adapting to a new feature with very few available observations, and thus relate to few-shot learning and related fields such as meta-learning and continual learning. From a technical point of view, we use an auxiliary neural network to amortize the learning of parameters associated with the new feature, which falls under the domain of hypernetworks. Furthermore, in the context of recommender systems, a number of related methods have been proposed to address the cold-start problem. We thus discuss related work in these three areas.
Few-Shot Learning Few-shot learning is the problem of designing machine learning models that can adapt to new prediction tasks given a small number of training examples. A popular approach to this problem is gradient-based meta-learning, such as MAML [4] and Reptile [24], which seek a parameter initialisation θ that can rapidly adapt to tasks drawn from a task distribution p(T ).These methods do not directly condition the parameter initialisation for a new task on any associated data or metadata, instead relying on fine-tuning,which can be both computationally expensive and lead to severe overfitting when little data is available. Another line of methods seek to adapt a classifier to a task based on a context set of class examples. For instance, by embedding class examples to provide a nearest neighbours classifier [33], learning an attention mechanism between class examples and a new example [37], or modulating activation functions [22] within a feature extractor conditioned on the context set. Conditional Neural Adaptive Processes [28], which are based on Conditional Neural Processes [5], adapt both classifier and feature extractor parameters based on the context set for a task. Similarly, [6] generate classifier weights for a new image class based on features extracted using the base model. However, in all cases, each task or image class is treated as independent from all others, whereas CHNs explicitly utilize all previously-observed features in the base model when adapting to a new feature.
A closely related field is continual learning [15,23], where a model seeks to adapt to new tasks or a shifting data distribution while avoiding catastrophic forgetting of old tasks. Continual learning does not necessarily address the few-shot scenario and is commonly applied in classification settings where the classifier/heads can be either shared or independent. CHNs can be seen as a means of addressing continual learning in the few-shot learning regime, by generating parameters for a new feature conditioned on all of the features already learned by the model.
Hypernetworks Hypernetworks [11] are auxiliary neural networks which generate the parameters of a neural network. They were introduced in [11] as a form of model compression, with the hypernetwork taking as input only structural information about the weight matrix they are currently predicting, such as its index in a sequence of layers. By contrast, CHNs are explicitly conditioned on data relevant to the weights currently being predicted. Bertinetto et al. [2] train a hypernetwork to predict all of the parameters of a binary classifier for a class of images, conditioned on a single exemplar image for the class. It is found that the output dimension of the hypernetwork grows extremely large for even small classifiers-in order to mitigate this, the authors propose a factorisation of the parameters, whereas we instead choose to learn only a small number of feature-specific parameters θ n . Task-conditioned hypernetworks [38] provide an application of hypernetworks to multi-task continual learning, where weights for the entire neural network for different tasks are predicted using a hypernetwork, based on a learned task embedding. This setting differs from our work as the continual learning tasks are assumed to be independent, and the hypernetwork is not conditioned directly on data for the new task, instead requiring a gradient descent process to learn the associated task embedding.

Cold Starts in Recommender Systems
Cold starts [31,18] occur when there is little or no data associated with a novel item or user in a recommender system. Collaborative filtering approaches to recommender systems have enjoyed great success for many years [30,29,13], but can fail completely when there is very limited rating data associated with a new user or item [16]. One potential solution to cold starts is given by content-based methods [26,19], which use any available descriptive information about the new user or item. Hybrid approaches [1,34,7] seek to marry these two approaches, making use of both collaborative and content-based methods. Meta-learning approaches also show promise for solving cold starts, including MAML-like approaches [3] for initialising new items, or adapting either the weights of a linear classifier or the biases in a neural network based on a user's history [36]. When applied to recommender systems, CHNs combine the strengths of all of these approaches, using content information, ratings data and latent representations of the associated users to generate accurate parameters for novel items.

Experiments
We demonstrate the performance of the proposed CHN in three different real-world application scenarios, including recommender systems (Section 4.2), healthcare (Section 4.3) and e-learning (Section 4.4). Our method exhibits superior performance in terms of prediction accuracy across all these applications. We also perform timing experiments to demonstrate the computational efficiency of CHNs compared to other methods.

Experiment Settings
In all our experiments, we apply a CHN to a partial variational autoencoder (P-VAE) [21,20] as an exemplar model. This is a flexible autoencoder model that is able to accurately work with and impute missing values in data points, allowing us to model sparsely-observed data such as that found in recommender systems. For each new feature n, we augment the P-VAE's decoder with a new decoder head consisting of an additional column of decoder weights w n and an additional decoder bias term b n which extend the model's output to the new feature, so that θ n = {w n , b n }. Figure 3 illustrates how a CHN is used to extend a P-VAE to make predictions for a new feature x n .
For all experiments, we train the CHN to output accurate feature parameters based on a range of context set sizes k ∈ [0, ..., 32] by randomly sampling k on each occurrence of a meta-training set Full details of all of the baselines used in the experiments can be found in Appendix C. Full details of the model architectures and hyperparameters can be found in Appendix D.

Recommender Systems
In real-life recommender systems, new users and new items are continuously added as new customers join and new products are launched. In deep learning based frameworks such as [32,17,8,20], the deep neural networks are commonly used in a user-based manner. In this approach, each new user is treated as a new data point, while each new item is treated as a new feature. To add a new item, one must extend the network architecture to incorporate the new feature, and we propose CHNs as an efficient way to predict the parameters associated with the new feature.
We evaluate the scenario above with MovieLens-1M dataset [12]. The dataset consists of 1 million ratings in the range 1 to 5 given by 8094 users to 5660 movies, and is thus 2.2% observed.
For each movie, we have associated metadata M n giving a list of genres associated with the movie, such as Action or Comedy, which we encode in a binary format, alongside the year of release which we normalize to lie within [0, 1]. For each random data split, we sampled 60% of movies as training data to train the base P-VAE model, used 30% as a meta-training set for CHN training and used the remaining 10% as a meta-test set.
The plot in Figure 5 (left) shows the performance of our proposed CHN, comparing with all other baselines in terms of RMSE (lower is better). Our method shows an advantage over all considered baselines other than MAML in the few-shot regime (k < 8), while achieving competitive performance with MAML without requiring costly fine-tuning.

Healthcare
In healthcare applications, a new question is often added to an existing health-assessment questionnaire, and in hospitals, new medical devices may be introduced to make physiological measurements. In this case it is desired for a model to quickly adapt to the newly added feature for health assessment, even when relatively few tests have been administered and so data is scarce.
We assess the utility of CHNs in a healthcare setting using synthetic data generated by the Neuropathic Pain Diagnosis Simulator [35]. This simulator produces synthetic data using a generative model to simulate pathophysiologies, patterns and symptoms associated with different types of neuropathic pain. The data is binary, where a 0 represents the a diagnostic label that is not present in a patient's record, and a 1 indicates a diagnostic label that is present. We simulated 1000 synthetic patients, and removed features with fewer than 50 positive diagnoses, resulting in 82 remaining features, with 17.3% of the values in the dataset being positive diagnoses. We used 50% of the features as training set; 30% of the features as the meta-test set and 20% of the features as the meta-test set.
The plot in Figure 5 (middle) shows the results in terms of AUROC (higher is better), as the dataset is highly imbalanced. Our method consistently outperforms all baselines across all values of k, while many methods including MAML suffer from severe overfitting when k is small. In contrast to the MovieLens-1M result, here the 10-nearest neighbour approach does not seem to leverage more datapoints in the context set. This shows that our method is desirable in the cost-sensitive healthcare environment, even for highly imbalanced medical tests where results are largely negative.

E-learning
We foresee CHNs being valuable in online education settings, potentially allowing teachers to quickly assess the diagnostic power of a new question given a small number of answers, or to gauge whether a new question's difficulty is appropriate for a particular student.
We assess the performance of the CHN in an e-learning setting using a real-life dataset provided by the e-learning provider Eedi for the NeurIPS 2020 Education Challenge [39]. In particular, we use the dataset for the first 2 tasks, filtered so that all students and questions have at least 250 associated responses. This results in a dataset of for 6797 students across 4792 questions, detailing whether or not a student answered a particular question correctly. The dataset contains approximately 2.7 million responses, making it 8.2% observed. We treat each student as a data point and each question as a feature, and use a binary encoding of each question's associated subjects as metadata. We used 60% of the questions as training set; 30% of the questions as the meta-test set and 10% of the questions as the meta-test set.
The right panel in Figure 5 illustrates the performance on prediction on the unseen meta-test set in terms of AUROC. Our method shows a significant improvement over all of the considered baselines over the entire range of k, suggesting real promise for applying CHNs in educational settings. In addition, we use this real-world dataset to compare the time taken to generate new feature parameters at meta-test time for a number of methods. The results are shown in Table 1. We see that the CHN offers nearly a 4-fold speedup compared to the nearest-neighbours based approach. We see a similar difference in performance when compared to training the new heads on a single observation for just 10 epochs. Moreover, while this training time grows rapidly with the number of observations in the context set, the time taken for a CHN remains nearly constant since it can efficiently parallelize along these observations, making CHNs an extremely efficient initialisation choice for larger context set sizes.

Conclusion
We introduce Contextual HyperNetworks (CHNs), providing an efficient way to initialize parameters for a new feature in a model given a context set of points containing the new feature and feature metadata. Our experiments demonstrate that CHNs outperform a range of baselines in terms of predictive performance across a range of datasets, in both regression and classification settings, and are able to perform well across a range of context set sizes, while remaining computationally efficient. In the future work, we will evaluate CHNs in streaming setting with large-scale real-world applications.

A.1 Partial Variational Autoencoders
For our experiments, we base our model on the Partial Variational Autoencoder (P-VAE) [20] -this model combines a traditional variational autoencoder (VAE) model with a PointNet-style set encoder [27], allowing it to efficiently encode and reconstruct partially observed data points. The P-VAE is based on the observation that typically the features in a VAE are assumed to be conditionally independent when conditioned on the latent variable z. That is, Then, given a data point x with observed features x O and unobserved features x U , we have that Hence, if we can infer a posterior distribution over z from the observed features, we can use this to estimate p(x U |x O ). The P-VAE infers a variational posterior distribution over z using an amortized inference network (or encoder network) q θ (z|x O ) and approximates the conditional data distribution given a value of z using a decoder network p φ (x O , x U |z).
In our model, we extend the decoder to decode the value of a new feature x n by initialising an additional subnetwork in the decoder which we term a decoder head, with parameters φ n , to extend its output dimension by one. In principal this head could be of any architecture which takes as input the output of the shared layers of the decoder, but in practice we found that simply extending the final layer of weights and biases to accommodate a new output dimension yielded good results while remaining parameter-efficient as the number of output features grows.

A.2 Training P-VAEs
The P-VAE is trained to reconstruct observed features in the partially-observed data point, and in the process learn to infer a variational posterior q θ (z|x O ) over the latent variable z. The P-VAE is given batches of data points where features from both the meta-train and meta-test sets are hidden from the model. Additionally, each time a particular data point is input, some additional features are also randomly hidden from the model using a Bernoulli mask, in order to ensure the model is robust to different sparsity patterns in the data. The P-VAE is then trained by maximising the Evidence Lower-Bound (ELBO) [21]: Context Set Size  Figure 6: k-shot learning performance on MovieLens-1M where features are arranged into training, meta-training and meta-testing set chronologically by movie release data from oldest to newest.

B Appendix: Chronological Feature Ordering
Throughout our experiments training CHNs, we use random splits of each dataset's features into training, meta-training and meta-testing. While we do not believe that this represents information leakage from future to past in an asymmetric way, we performed an additional experiment on MovieLens-1M where the training, meta-training and meta-testing sets are arranged chronologically by movie release date from oldest to newest. Figure 6 shows the results of this experiment. We see that the overall performance is somewhat worse for all baselines (although this may simply be random noise), but that the relative ordering of the baselines appears largely unchanged.

C Appendix: Baselines
Here we provide additional details and results for the baselines used in our experiments.

C.1 Overview
We consider the following baselines for generating the new feature parameters θ n = {w n , b n }. All methods are applied to the same base trained P-VAE model to ensure a fair comparison.
• Random: Generate a random value for θ n for each new decoder head using Xavier initialisation for weights and 0 for biases. • Mean Imputing: Set weights and biases to always predict the mean of the observed values for the new feature in the context set, i.e. w n = 0 and b = σ −1 1 k i∈Cn x (i) n . • Mean Head Parameters: Generate the new head parameters θ n as the mean of all of the head parameters learned on the training set features. • Mean Head Parameters (Matching Metadata): As above, but averaging only over parameters of heads whose associated feature has metadata categories matching those of the new feature. • k-Nearest Neighbour Head Parameters: Generate the new head parameters θ n as the mean of the head parameters of the k-nearest neighbour features in terms of Euclidean distance, where column-wise mean imputing is used to fill in unobserved values. • Train from Random: Initialize the new feature head parameters randomly, and then fine-tune these parameters on the data in the context set C n for a fixed number of epochs. • MAML: We meta-learn an initialisation of θ n using Model-Agnostic Meta Learning [4], where we treat the prediction of each feature as a separate task and fine-tune these parameters on the context set. In all experiments, we compare with the MAML baseline which has the best-performing number of fine-tuning epochs. For full details, see Appendix C.

C.2 MAML
We adapt the Model-Agnostic Meta Learning [4] technique as a baseline. The decoder head parameters θ n are adapted using the MAML algorithm in the 'meta-training' stage. Each new feature X is viewed as a separate MAML task, with some observed and unobserved values. We sample the tasks in batches of size M and train the inner (a.k.a. fast) model over N steps. The inner model training loss is the ELBO of the PVAE on the observations L X O . The meta-model (a.k.a. the slow or outer model) is trained by being given the context set observations, and computing a reconstruction loss on the target set,L T ,C (f θ , X ). The gradient for the meta-model update is taken over the batch reconstruction losses mean. The full algorithm is detailed in Algorithm 1.
Notably, since MAML aims to fit parameters that adapt quickly to new tasks, it allows for fine-tuning at evaluation time, that is, training the model for several iterations from the MAML parameter initialization. Here, we evaluate the model with and without fine-tuning.
In the MAML baseline experiments we use M = 4, N = 10, ADAM [14] with learning rate α = β = 10 −2 for inner and outer model optimization. The model fine-tuned performance is evaluated over {1, 3, 5, 10} epochs and the best results are used. We make use of the higher order optimization facilitated by the higher library [10] in the implementation of this baseline. Figure 7 shows the performance of the MAML baseline for different numbers of fine-tuning epochs and with no fine-tuning. As expected, the baseline with no fine-tuning is outperformed by those where fine-tuning is employed. For the Neuropathic Pain and E-learning datasets, the increase in the number of fine-tuning epochs corresponds to improvement in performance (greater AUROC), whereas in case of MovieLens-1M, performance drops (RMSE increases) with longer fine-tuning, particularly for the smaller context set sizes.  Figure 9: Comparing the predictive performance when training decoder new head parameters in a P-VAE on a range of context set sizes k, on the E-learning dataset (higher is better).

C.3 k-Nearest Neighbour Head Parameters
We consider k-Nearest Neighbour Head Parameters baselines for the values k ∈ {1, 5, 10}. Figure 8 shows the performance of this baseline for the different values of k across a range of context set sizes. We expect that as k is increased further, and the number of head parameters averaged over grows, the behaviour will approach that of the mean head parameter baselines. In the main text, 10-Nearest Neighbours is used throughout, as it yields good performance in both the low and high-data regimes.

C.4 Fine-Tuning
In our experimental results, we show the performance of training the new decoder heads on their context sets from randomly initialized parameters for 10 epochs, in order to provide a trade-off between predictive accuracy and computational cost. In Figure 9a, we show the predictive performance of the P-VAE on the meta-test set after training randomly initialized head parameters for an increasing number of epochs, for a range of context set sizes k. We see that the performance improves with training in all cases, with better performance achieved as the context set size k increases, and thus the effect of over-fitting is lessened.
Furthermore, in Figure 9b, we perform the same experiment but instead initialising the heads with the CHN parameters. We see that in all cases except k = 0 and k = 32, training by gradient descent leads to a decrease in performance due to over-fitting, suggesting that the CHN has an implicit regularising effect on the parameter initialisation. We note also that in all cases, the untrained CHN parameters substantially outperform those trained from the random initialisation for all values of k, even after 25 training epochs, with many of the training curves appearing to approach convergence.

D Appendix: Experiment Details
All models were implemented in PyTorch [25]. All experiments were performed on a single Nvidia Tesla K80 GPU. For training both the P-VAE and the CHN's parameters, the ADAM [14] optimizer was used with β 1 = 0.9, β 2 = 0.999 and ε = 10 −8 . Training and evaluating a CHN for the specified number of epochs took around 3 minutes on the Neuropathic Pain dataset, around 1.5 hours on the E-learning dataset, and around 8 hours on MovieLens-1M.
Details of hyperparameters and model architectures used for each dataset can be found in Table 2.