{'fraction_non_alphanumeric': 0.028633953889779313, 'fraction_numerical': 0.01183577857230921, 'mean_word_length': 4.945574491478834, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 0, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Memory Based Attentive Fusion

The use of multi-modal data for deep machine learning has shown promise when compared to uni-modal approaches, where fusion of multi-modal features has resulted in improved performance. However, most state-of-the-art methods use naive fusion which processes feature streams from a given time-step and ignores long-term dependencies within the data during fusion. In this paper, we present a novel Memory Based Attentive Fusion (MBAF) layer, which fuses modes by incorporating both the current features and long-term dependencies in the data, thus allowing the model to understand the relative importance of modes over time. We define an explicit memory block within the fusion layer which stores features containing long-term dependencies of the fused data. The inputs to our layer are fused through attentive composition and transformation, and the transformed features are combined with the input to generate the fused layer output. Following existing state-of-the-art methods, we have evaluated the performance and the generalizability of the proposed approach on the IEMOCAP and PhysioNet-CMEBS datasets with different modalities. In our experiments, we replace the naive fusion layer in benchmark networks with our proposed layer to enable a fair comparison. Experimental results indicate that MBAF layer can generalise across different modalities and networks to enhance the fusion and improve performance.


Introduction
Multi-modal deep learning has become a major research area with an increasing number of applications that generate/use multiple modalities such as autonomous driving [1,2], emotion analysis [3,4], and biometrics [5]. Researchers have sought to develop different approaches to combine learned features from different modalities and obtain a common feature space which maximizes the overall performance of the system [6,7,8]. Careful selection of the fusion stage, the method and parameters, has enabled researchers to obtain higher performance compared to using a single mode. However, hardships associated with fusion have created new challenges, and thus fusion has emerged as a separate field of research.
The majority of prior multi-modal fusion research has used naive approaches such as feature concatenation and summation, or fusion methods using attention to filter out uninformative features from the combined space [9,10]. However, with a naive fusion approach the fusion process only uses information from the current time step, and ignores historical information which can play a vital role in multi-modal fusion by means of capturing long-term dependencies between the data domains. With multiple modalities, information from one modality may be occluded or undetected, leading to an information loss which adversely affects the fusion. However, reasonable adjustments can be inferred by understanding the relationships between the modes and the events captured within them.
Long-Short Term Memory (LSTM) networks provide a mechanism to capture historical relationships for fusion [11]. Despite their feedback connections and ability to process sequential data, these methods have limited capacity, especially when considering long-term dependencies. Memory Networks [12], however, are capable of learning long-term dependencies, especially in sequential data [13,14], by means of an explicit memory representation. The original concept of an explicit memory has been improved [15,16] and used in applications not limited to sequential data [17]. Since memory networks consider historical information explicitly, they reduce the chance of forgetting important historical information. Therefore, we argue that by incorporating an external memory to store historical data relevant to the fusion, fusion in future iterations can exploit this information. However extending existing memory architectures to enable fusion is not trivial, and may add more complexity to the network which can degrade the performance if not carefully designed. Furthermore, consideration needs to be given to the information stored in the memory, and how best to extract information from the memory to assist decision making.
The main objective of this research is to evaluate the performance of a memory based architecture for multi-modal fusion. We have proposed an improved memory network which receives multi-modal data and outputs fused features, which incorporates both current and historical data. We conduct experiments on different modalities and architectures, and show the superior performance and generalizability of the proposed method.

Literature Review
Multi-modal deep learning has been extensively used in research applications such as emotion recognition [18,19,20] and autonomous driving [21,22] over the past decade. Different modalities including video, multi-spectral data and sensor data have been used in the above applications [23,24,25]. Within these applications, deep networks are frequently used to learn highlevel semantic features from each modality, and then the learned features are combined through fusion. State-of-the-art methods in multi-modal fusion will be discussed in Section 2.1, and memory networks will be outlined in Section 2.2.

Deep Multi-Modal Fusion
The fusion of multi-modal features can be carried out at different depths in a deep neural network, and methods can be broadly categorized into three stages: early, intermediate and late [26,27]. A widely used fusion methodology is concatenation, where the features are combined via concatenation (horizontal or stacked) [28,29], summation or multiplication. In intermediate fusion, concatenation is widely used where features vectors from each modality are appended, while element-wise summation and multiplication of feature vectors is also common.
Haghighat et al. presented a customized intermediate feature fusion approach using discriminant correlation analysis for biometric recognition [30]. It incorporates class associations into the correlation analysis of the feature sets and performs fusion by maximizing the pairwise correlations across the two feature sets. The proposed method eliminates the inter-class correlations and strengthens the intra-class correlations. Dong et al. proposed an improved late fusion approach based on matrix factorization [31]. In general, late fusion leads to performance degradation as the predictions from different features can contradict each other. Such methods leverage a hard constraint on the matrix rank to preserve the consistency of predictions by different features. Kim et al. used locally adaptive fusion networks to fuse tri-modal (original image, disparity and matching cost) confidence features (using scale and attention maps) for stereo matching to reconstruct the geometric configuration of a scene [32]. They have used attention networks to encode the importance of features and then stacked them together in an adaptive and dynamic fashion.
Even though the above stated feature fusion frameworks achieve significantly better results compared to uni-modal networks, they can introduce irrelevant and redundant features within the fused feature space, which reduces the potential performance improvement. Therefore, feature refinement techniques have been used after the fusion.
Pouyanfar et al. proposed a residual attention based fused feature refinement method, where the encoded features from each modality are fused using a weighted Support Vector Machine to handle imbalanced data [33]. Lv et al. introduced a feature refinement unit using a combination of a CNN and an RCNN which can correct the networks own identification errors based on the acquired knowledge, and adapt the RCNN to compensate for the lack of feature extraction in the CNN [34]. Park et al. introduced feature refinement blocks from multiple stages of a deep net to achieve more accurate prediction [35]. Feature fusion blocks learn residual features from each modality and their combinations to fully exploit the complementary characteristics in the data. Wang et al. suggested that the fusion of global features from two modalities (image and 3D point cloud) blindly would degrade the performance of the estimation [36]. They have proposed a pixel-wise dense fusion network that effectively combines the extracted features by performing local per-pixel fusion instead of global fusion.
However, these feature fusion and refinements methods are application specific and fail to generalize across applications. This illustrates a recent trend of introducing fusion methods specific to individual tasks, rather pursuing a general fusion architecture.
Beard et al. proposed recursive multi-attention with a shared external memory [37] which is updated over several iterations as an alternative to naive attention [38], where attention on one modality exploits other modalities. They generate a context vector representing attended features from all modalities and a vector representing the previous iterations, which is passed to the next iteration. This method can easily be extended for multi-modal fu-sion; however, it is limited by the simple vector representation used to store and retrieve historical patterns. Zadeh et al. proposed a dynamic fusion graph for multi-modal fusion by defining n-modal dynamics as a hierarchical process [39]. They have suggested that each fusion combination from the mode set has a contribution towards the final fusion outcome, and derived a "fusion set" graph. The graphs' node connections are weighted by efficacies (to measure how strong the connection between to mode sets) which indicate the contribution. Wang et al. proposed a multi granularity fusion approach to fuse information from attentive and global features [40]. Yang et al. proposed a dynamic fusion method that randomly repels the representations from less significant data sources for fusion [41]. This enables the network to dynamically select informative modes of information and eliminate uninformative modes from fusion and increase the performance.
The majority of multi-modal fusion research has limited the developed methods to using the combination of input features at a given time step as described above. The notion of historical context has been applied to a few of the approaches [38,11], but those methods fail to learn long term dependencies in fusion. Therefore, we propose the use of an explicit memory network to address the above limitations in multi-modal fusion.

Memory Networks
Memory networks utilize a storage block and inference components (reader, writer and composer) together, and learn to use these components jointly [12]. The memory can be read and written to, with the aim of using it for prediction tasks. Memory networks were introduced to alleviate the issue of learning long-term dependencies in sequential data. Compared to a Long Short Term Memory (LSTM) unit which updates an internal fixed-size memory representation, memory networks consider the entire history explicitly, eliminating the chance to forget, and the size of memory becomes a hyper-parameter to tune. Sukhbaatar et al. has extended the above idea by introducing a recurrent attention model over the external memory which is trained end-to-end [37]. This approach can be applied to realistic settings since it needs less supervision during training.
Rae et al. have shown that naive memory networks scale poorly in time and space as memory grows [42]. They have proposed a memory access scheme which is end-to-end differentiable: Sparse Access Memory (SAM). They have shown that this method retains the representative power of naive methods while training efficiently with large memories. Miller et al. proposed a key-value memory network for question answering to overcome the limitations of knowledge bases [14]. Their approach makes document reading viable by using a different encoding in later stages (output) of the memory read and addressing.
Kumar et al. proposed a dynamic memory network for question answering that processes questions and input sequences using episodic memories [16]. An iterative attention process is triggered by input questions and it allows the network to condition the attention based on the history of previous iterations and the inputs. Fernando et al. introduced a Tree Memory Network which modeled inter-sequence (long-term) and intra-sequence (short-term) relationships using memory modules [43]. The memory was implemented as a recursive tree structure, compared to a naive approach which uses a sequence of historical states. This shows that memory networks can be applied to a diverse set of applications not limited to natural language processing.
Fernando et al. have used individual memories on two modes before fusion [44], to help extract salient features for fusion. However, the joint learning of the two modalities does not allow modes to see the information within their histories, and determine how this can be utilized in decision making.
Even though significant advances have been made in areas of multi-modal fusion and memory networks, the applicability of memory networks to multimodal fusion is not well explored. To the best of our knowledge, [44] is the only work which investigates how history can be used to augment feature fusion. However, no investigation has been made regarding: 1) what information from each modality should be stored; 2) how to combine different modalities using fusion; and 3) how to effectively extract information from the stored memories to augment the decision making. Our proposed system uses a memory-based fusion layer to model the relationship between data sources and uses it as supportive data for naive fusion. Our proposed layer addresses the above mentioned research questions.

Methodology
The proposed Memory-Based Attentive Fusion (MBAF) layer (see Figure 1) consists of four major modules: read, compose, write and a memory encoder. The memory is represented as M ∈ R k,l , with a variable number of memory slots where k is the memory length (slots) and l is the hidden dimension. The size of the hidden dimension was set to the shape of the concatenated multi-modal features that are the input to the memory. During initialization, the memory is filled with values from a normal distribution where µ = 0 and σ = 1. All other weights are initialized using a uniform distribution. The proposed layer takes two dense feature vectors from both modalities as the input, and outputs a tensor with the same shape as naive concatenation to obtain a similar number of parameters for deeper layers.
Consider the dense feature vectors from mode 1 and mode 2, which we denote x m 1,t and x m 2,t respectively. The memory controller concatenates the feature vectors to obtain x t (Equation 1), which is used with the state of the memory from the previous timestep (M t−1 ) to generate a key vector (z t ) for a corresponding memory slot which is semantically associated with x t . The read module receives z t and M t−1 and retrieves data from the selected slot m r,t . The slot location r is defined by z t which is obtained by attending over memory slots [45]. The process is defined by Equations 2 and 3.
Similar to the work by Munkhdalai et al. [46], the read function, f r , maps the concatenated dense feature vectors (x t ) to internal memory space and the memory slots related to x t are determined. The degree of association is calculated and transformed to a key vector z t , and the composing slot, m r,t , is determined by the weighted sum of all slots. Then m r,t and x t are passed to the memory composition component and the composition operations concatenate them followed by passing the representation through a Multilayer Perceptron (MLP), f M LP c , and self attention. The resultant composition feature vector, c t , is returned. We have used fusion to incorporate both the input and memory in composing the output, and an MLP to make the composition learnable. With this, the composition component is capable of learning semantic features from fused vector space. Attention is applied to the generated features from the MLP to filter uninformative features from the composition. The composition process is defined by Equations 4, 5 and 6, where α andb are the attention score and attentive vector respectively. Figure 1: Proposed MBAF layer: Inputs are the dense feature vectors from two modalities, and the output is a feature vector of the same dimension as the concatenation of the input features. The inputs are concatenated and the corresponding memory locations' key is calculated (green box). The resultant key is used to read the memory slot (orange box) and it is fed to the composer (blue box) along with the concatenated input to compose. Self-attention is applied to filter uninformative features from the composed vector. The composer output is transformed (learnable over iterations) and written to the memory slot (pink box) using the pre-calculated key by the controller. The transformed vector is added to the concatenated input and outputs from the layer.
Then the composition output, c t , is transformed to the encoding memory space, h t , with weight w t and a RELU activation. In the write module, the resulting feature vector, h t , is written to the selected slot location of the memory, r, using the pre-calculated key, z t . This updates the memory to the new statem M t . First the data in the corresponding slot is erased and then it is replaced with h t as indicated in Equations 7 and 8, where 1 is a matrix of ones, e l ∈ R l and e k ∈ R k are vectors of ones and ⊗ is the outer product. Finally, the output of the layer (o t ) is calculated using x t and h t as indicated in Equation 9 where f SU M o refers to axis wise summation of elements where the output of the MBAF layer is determined by the composition of the current input and previous memory state. This operation is carried out to ensure the output dimension of the layer is similar to that achieved by naive fusion, and the inclusion of memory output h t ensures that the long term dependencies are incorporated in the fusion operation, The number of parameters of the proposed fusion layer changes with the input dimensions and it is calculated using, where p mbaf , s x 2,t , s x 2,t , q refers to the total parameters of the layer, the vector dimension of first and second mode, and the batch size respectively. Similar to the naive fusion layer, this can be extended to n modalities without significant changes to the architecture. Even though this layer has a significantly higher number of parameters (Equation 10) compared to a naive fusion layer, operations are fast and no significant time increase can be observed during inference.
In the proposed layer, we have developed a novel approach in composition, memory update and output generation to incorporate historical features and the current input for feature fusion. Compared to other methods, Beard et al. [38] uses a single vector as the memory (context) while we utilize a memory (M ) with l slots, which ensures that our model has more capacity to store and retrieve informative facts for different contexts. Fernando et al. [44] utilized two separate memory layers for two different modalities, (M tr t and M sp t ), while we use a single memory for the combined features. Due to this representation, we are able to build a memory which stores historical relationships among the two modalities that are learned by the network, and use that information during fusion, giving the network a better intuition of how modalities behaved in the past compared to the above works.
In memory composition (c t ), both the above works have used naive fusion (concatenation) where we have used an MLP and attention (applied to inputs and memory) to encourage the memory to learn relationships among historical feature representations and current features, and to filter-out uninformative details. In output generation, both the prior methods have used either a transformed c t (i.e. h t ), or c t itself. However, if the input feature vector contains different information from the historical vectors, it will be augmented and the same information will not be propagated to deeper layers. However, in the proposed method, we use both the input and the transformed features in output generation to minimize the above effect.

Datasets and Experimental Setup
In this section, we describe the datasets used and the experimental setup to evaluate the performance gain in multi-modal deep learning networks by using a memory enabled fusion, and to establish it's generalizability. We have selected benchmarks and baselines from the literature for two different domains: emotion recognition and physiological signal analysis. We have implemented baseline network architectures similar to original works, and replaced the naive fusion layer with the proposed Memory-Based Attentive Fusion (MBAF). The same hyper-parameter configurations were used to evaluate the performance.

Experimental Setup for IEMOCAP
The Interactive Emotional Dyadic Motion Capture (IEMOCAP) dataset for emotion recognition was selected and our architecture follows the approach of [47]. IEMOCAP contains five sessions of utterances for 10 unique speakers along with the transcripts. We follow the evaluation protocol of [48,47], and select utterances annotated with four basic emotions: anger, happiness, neutral and sadness; to achieve an approximately even sample distribution over classes. Samples with excitement are merged with happiness as per [47]. The resultant dataset contains 5531 utterances {anger:1103, happiness:1636, neutral:1708, sadness:1084}.
Initial training is carried out on both acoustic (AE) and textual (TE) encoding networks separately before fusion as described bellow [47]. In AE, a 250ms audio utterance is passed through a SincNet layer [49] followed by a Deep Convolution Neural Networks (DCNN) with "Convolution1D" and "Dense" layers. We obtain a 2048-D feature vector as the output of AE.
In TE, the input vector is passed through an embedding layer followed by two parallel branches (Left: bi-RNN + DCNN; Right: DCNN) for textual feature extraction. The output vector from the Bi-RNN is passed through three parallel convolutional layers with a filter sizes of 1 ,3 and 5; and convolutional layers with the same filter sizes are used in the right branch. Crossattention is applied on convolution layers with the same filter size from the right branch as the attention for the left branch. The corresponding convolution layers from two branches are concatenated together and passed through a dense layer to obtain 4800-D feature vector as the output of TE. The resultant dense feature vectors from each modality are then passed through the proposed MBAF layer as illustrated in Figure 2.
Similar configurations have been followed where the sampling rate of each utterance (A), audio segment length (A), window size (A), window shift (A), max sequence length (T), embedding (T) were set to 16000Hz, 250ms, 250ms, 10ms, 100 and Glove-300d respectively (A-Audio network configuration, T-Text network configuration). We utilize a 8:1:1 dataset split for training, validation, and testing sets respectively. The learning rate and the batch size in each network are fixed at 0.001 and 64 respectively, and the Adam optimiser is used. Experiments were conducted on the baseline architecture with naive fusion and using the proposed fusion.

Experimental Setup for PhysioNet-CMEBS
We have selected the "combined measurement of ECG, breathing and seismocardiogram (CMEBS)" dataset from the PhysioNet data ban. The dataset contains electrocardiogram (ECG), breathing (through respiratory signals -RS) and seismocardiogram (SCG) for 20 healthy people (physiological signals) [50]. Subjects are asked to listen to music and the physiological signals were measured at three states; Basel state, while listening to music and after the music. The recordings are 5min, 50min, and 5min in length for each state respectively. The data has been acquired by a Biopac MP36 [50] where channels 1,2,3 and 4 were devoted to measuring ECG-I, ECG-II, RS, and SCG respectively. Each channel was sampled at 5 kHz.
For experimental purposes, we divide and annotate the data according to the acquired state (3 classes). Furthermore, due to the high class imbalance in the original dataset, we selected a single 5min segment from the second state (first 5 mins). Then each signal was segmented into 200ms chunks (each chunk with 1000 data points). The obtained dataset consists of 14, 977 samples {class 1: 4977, class 2: 5000, class 3: 5000}. Due to the lack of multimodal fusion research on this dataset, we created a benchmark model with LSTMs and a MLP as illustrated in Figure 3. We considered 6 consecutive chunks from each modality as a single input to the network. First, the inputs are passed through LSTM-1 with 512 hidden units which returns sequences, followed by LSTM-2 with 512 hidden units which returns a single dense output. The resultant features from both modalities are then fused using the proposed MBAF layer. Then the fused features are passed through a dense layer with 1024 units and 0.5 dropout rate followed by a classification dense layer with 3 units and Softmax activation.
Experiments were conducted on the proposed baseline architecture with naive fusion and the proposed fusion. The learning rate and the batch size in each network is fixed at 0.001 and 32 respectively, and the Adam optimizer is used. Leave One Subject Group Out (LOSGO) cross-validation is used in this experiment with 4 groups of 5 subjects in each.

Results and Discussion
The experimental results on both IEMOCAP and PhysioNet-CMEBS datasets are described in Section 5.1 and 5.2 respectively.

Experiments on IEMOCAP Dataset
Following [47,48], we have measured the performance of our system with weighted accuracy (WA) and unweighted accuracy (UA). Table 1 and Figure  4 presents the performance of our approach for emotion recognition compared with the baseline method. The only architectural differences between our model and the baseline model is the proposed memory-based attentive fusion layer used instead of the naive fusion layer. We have kept all the corresponding layers and parameters same throughout the experiment. Our proposed model has achieved a substantial improvement in overall accuracy, with a 1.7% increase (in MBAF-III) compared to the best baseline NF-III. Furthermore, we have achieved a performance increase of 0.6% and 0.5% from MBAF-II and MBAF-I respectively. This performance increase is achieved only by changing the fusion layer.

Experiments on PhysioNet-CMEBS Dataset
For PhysioNet-CMEBS, we have measured the performance of our system only with unweighted accuracy (UA) since the modified dataset has an approximately equal class distribution. Table 2 and Figure 5 present the performance of our approach for physiological signal state recognition compared with our baseline method. Our proposed model achieves a substantial improvement in overall accuracy, with approximately a 6.5%, 2% and 2.5% A + T 71.8% − MHA-2 [48] A + T 76.5% 77.6% NF-I [47] A + T 79.27% 77.85% NF-II [47] A + T 80.01% 78.98% NF-III [47] A  increase in F-IV, F-V and F-VI respectively compared to the corresponding baselines with naive fusion. Confusion matrices are provided in Figure 5. Table 3 shows the inference time for the above mentioned models with MBAF compared to the NF for 500 test samples on PC with 20 CPU cores and 60GB of memory. Since we have used the same benchmark network with the same number of parameters for F-IV, F-V and F-VI (different modalities with same input dimensions as inputs), we have averaged the inference time for each fusion (F − IV − V I).
A significant difference in inference time cannot be observed between MBAF and NF for IEMOCAP, even with the high parameter difference. We observe that the model used on the PhysioNet-CMEBS database is much faster, due to the complexity of encoder networks and higher feature dimensions required for processing IEMOCAP. With these results, it is evident  that the proposed MBAF layer has a negligible impact on the inference time, even with high dimensional inputs.

Ablation Studies
Several ablations studies have been carried out to identify the impact of memory size, location of the memory unit, and the read function on the final classification performance. The best model (MBAF-III) for the IEMOCAP dataset is selected for these experiments and results are described below. Table 4 illustrates the variation in recognition accuracy with the number of memory slots, l, measured with unweighted and weighted accuracy. Approximately similar unweighted accuracies can be observed with lower  memory sizes while higher memory sizes achieve comparatively lower accuracies (but higher than naive fusion, refer to Table 1). However, fluctuations can be observed with weighted accuracy where highest accuracy is obtained with a memory size of 30. A fluctuating relationship between memory size and the accuracy is observed for the PhysioNet-CMEBS dataset as illustrated in the Table 5.

Impact of Memory Size on Recognition Accuracy
Even though the recognition accuracy changes with the memory size, all the recognition rates have surpassed the naive fusion accuracy (refer to Table  1 and 2). A drop in recognition accuracy can be observed for higher memory sizes since when more history is stored, it is harder for the memory read operation to find salient information. Similarly, when the memory is too small information is lost and performance degrades slightly.

Impact of Read Function on the Recognition Accuracy
The above described MBAF layer uses an attentive approach to select the memory location in the read module. We have evaluated the performance of using cross-attention over naive attention to retrieve the memory slot since we are inputting two modalities to the layer. For this task, we have defined an additional feature fusion by swapping the order of the fusion (x s t ) and used it to compose (c t ), the output which is written back to the memory slot after transformation, as given in Equations 11, and 12. With this, when the attentive memory selection is carried out, the original fused features (x t ) are used as an attention input to the memory which contains transformed features of (x s t ), which behaves as a cross-attention function. All other equations described in Section 3 are unchanged.
The resulting accuracy variations with the memory size are illustrated in Table 6. The results indicate that the use of cross-attention is unable to achieve higher performance compared to general attention. The main reason behind this would be the significantly different features in the two modes.

Impact of Memory Location on Recognition Accuracy
We have evaluated the impact of the location of the memory based fusion layer to highlight the importance of using the memory in fusion. For this experiment, we have used the proposed MBAF layer on individual channels by altering Equations 1 to 13 as shown. All the other internal calculations were kept the same.
The experiments were carried out for the MBAF-III architecture with a memory size of 30 and the highest accuracy was 78.31% for UA and 80.29% for WA. Using memory for individual modes couldn't outperform the MBAF and NF accuracies. Therefore, it is evident that the long-term dependencies learnt through a memory network are capable of increasing the performance of multi-modal fusion over using a naive fusion scheme.

Conclusion
This paper proposed a novel deep learning architecture for multi-modal data fusion. Instead of using naive fusion which considers only the current input features, we have proposed a memory based attentive fusion layer which incorporates both the current input and long-term dependencies to generate the fused feature vector. We have evaluated our work on multiple datasets with multiple modalities to show the generalizabilty of the proposed fusion. The experimental results demonstrate that the proposed MBAF is capable of achieving significant improvements, outperforming the naive fusion used by state-of-the-art baselines in terms of classification accuracy.

Acknowledgements
The research presented in this paper was supported partly by an Australian Research Council (ARC) Discovery grant DP170100632.