{'fraction_non_alphanumeric': 0.0302799131506367, 'fraction_numerical': 0.02306202687635702, 'mean_word_length': 4.798570942497448, 'pattern_counts': {'":': 0, '<': 1, '<?xml version=': 0, '>': 1, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
A Transfer Learning Study of Gas Adsorption in Metal-Organic Frameworks.

Metal-organic frameworks (MOFs) are a class of materials promising for gas adsorption due to their highly tunable nano-porous structures and host-guest interactions. While machine learning (ML) has been leveraged to aid the design or screen of MOFs for different purposes, the needs of big data are not always met, limiting the applicability of ML models trained against small data sets. In this work, we introduce an inductive transfer learning technique to improve the accuracy and applicability of ML models trained with small amount of MOF adsorption data. This technique leverages potentially shareable knowledge from a source task to improve the models on the target tasks. As demonstrations, a deep neural network (DNN) trained on H2 adsorption data with 13,506 MOF structures at 100 bar and 243 K is used as the source task. When transferring knowledge from the source task to H2 adsorption at 100 bar and 130 K (one target task), the predictive accuracy on target task was improved from 0.960 (direct training) to 0.991 (transfer learning). We also tested transfer learning across different gas species (i.e. from H2 to CH4), with predictive accuracy of CH4 adsorption being improved from 0.935 (direct training) to 0.980 (transfer learning). More importantly, transfer learning is shown to effectively improve the models on the target tasks with low accuracy from direct training. However, when transferring the knowledge from the source task to Xe/Kr adsorption, the transfer learning does not improve the predictive accuracy, which is attributed to the lack of common descriptors that is key to the underlying knowledge.


INTRODUCTION
Metal-organic frameworks (MOFs) are a highly tunable class of nano-porous materials with exceptional host-guest properties. The nearly limitless combinations of metal nodes and organic linkers that can be used to create MOFs result in a class of materials that is enormously diverse, thus leading to equally diverse applications of interest where MOFs can be applied, such as gas storage, [1] catalysis, [2] nonlinear optics, [3] and light-harvesting. [ Advanced molecular simulation techniques have made calculating structural and functional properties of MOFs reliable given only the crystalline structure of the material. [5] For example, the gas uptake capacity of a MOF can be predicted by grand canonical Monte Carlo (GCMC) simulations on a modern workstation in under half an hour. [6] However, recent in-silico MOFs design has given rise to databases of materials consisting of hundreds of thousands of MOF structures that are ready for high-throughput computational screening. [7] The exhaustive screening of such immense search spaces can be prohibitive, even when using rapid methods to generate the simulation parameters and characterize the materials. [8] Given the large databases of materials and the data that can be generated by analyzing those databases, machine learning (ML) algorithms are currently used in the materials field to analyze the data and speedup material development due to their potential to reveal new insights into structure-property relationships and drastically accelerate material evaluation while maintaining accuracy. For example, ML can be used to generate a surrogate model to preselect high-performing candidates prior to intensive simulations, and such a process only take a fraction of seconds, which makes the virtual screening of intractably large search spaces feasible. In addition, ML techniques are also shown to be capable of extracting important features and descriptors that influences the structure-property relationships. [9] In this manuscript, we will focus on ML efforts that leverage surrogate models towards the prediction of gas adsorption and separation in MOFs. For the readers interested in further reading of the applications of ML algorithms in MOFs, we refer them to recent reviews in the literature. [10][11][12][13][14][15] Hydrogen storage has been an area of particular interest where ML has been applied to aid MOF performance evaluation. For example, Anderson and coworkers [16] performed molecular simulations to calculate volumetric hydrogen storage in MOFs containing catecholates to improve the material-hydrogen interactions. They varied the strength of interaction and performed over 18,000 GCMC simulations to calculate hydrogen adsorption at various temperature and pressure conditions. They used the data to train a neural network (NN) which they used as a surrogate model to calculate hydrogen deliverable capacity in a nonisothermal, pressure swing scenario. Most recently, Anderson and coworkers [17] performed molecular simulations to calculate adsorption isotherms of alchemical species to train a DNN. They then successfully showed that they could use that same DNN to calculate adsorption isotherms of real molecules, including hydrogen, nitrogen, argon, and methane adsorption. In a similar vein, Bucior and coworkers [18] devised an energy-based descriptor that captures the material-hydrogen interactions from fast molecular simulations. The information was used as input to a surrogate model from a LASSO regression. They were then able to use that surrogate model to evaluate MOF performance 1000x faster than GCMC simulations. [10] In recent work, Gharagheizi and coworkers leveraged ML algorithms to predict adsorption of mixtures in MOFs to identify structures that could separate azeotropic mixtures. [19] The authors used adsorption isotherm data with a multiple regression genetic algorithm to develop a surrogate model based on a given set of molecular and MOF descriptors. The authors also developed a classification model that predicts when their model is reliable. The combined use of the models was shown to match well with GCMC simulations of multiple gas molecules in a number of structures and identified promising materials for the separation of azeotropic mixtures. Similarly, Shih and coworkers demonstrated the use of hierarchical Bayesian estimation to find parameters for adsorption models based on various data sets. [20] At their core, what these various methods seek is to exploit existing or easily generated data to develop surrogate models that will allow faster prediction of the adsorption capacity of a certain gas given a MOF structure. [21] However, for a new gas species, accurate ML models need to be trained based on large datasets, which are not always available or needs significant time and resource to generate. It would be ideal if one can leverage the large datasets of one gas species (e.g., H2) to help improve the model accuracy using limited data for the target gas species of interest (e.g., CH4). Transfer learning (TL) is such a ML technique that can leverage "knowledge" learned from large datasets and transfer it to help learning, especially for constructing surrogate models, with small datasets. TL aims at improving the performance of a learner on a target domain by transferring the knowledge contained in a different but related source domain. [22] An early example of TL was applied in the area of sentiment classification, where it was used to adapt the classification model trained on certain products to improve the classification performance on other products, so that the effort for annotation reviews for various products can be reduced. [23] Preliminary attempts of TL have also been seen in materials research. For instance, DeCost et al. utilized a transferred deep convolutional network to learn representations of microstructures and then used these representations to infer the underlying annealing conditions. [24] In a recent study, TL was used to help refine a surrogate model between polymer chemistry and thermal conductivity. [25] Interestingly, the authors were able to obtain a DNN model with reasonable accuracy (R 2 =0.73) using merely 28 training data points in thermal conductivity available to them. This was realized by implementing the TL which transfers knowledge learnt from training against highly populated proxy labels (in their case, ~6,000 data points on glass transition temperature) into constructing DNN for sparsely populated labels (i.e., thermal conductivity). By directly training a DNN against the 28 thermal conducitivty data, R 2 was -0.4 (i.e., no prediction power), but TL helped improve the model accuracy to R 2 =0.73. The group later generalized such TL strategy to other properties of polymers and inorganic materials. [26] Recently, Ma et al. transferred the knowledge learned from 20 million drug-like small organic molecules to help learning polymer representations, and reported improved predictive accuracies in various surrogate models. [27] The essence of TL in materials research is that different properties are different expressions of the inherent chemistry and structure of materials; in a DNN-based surrogate model, the relationship between material structure and property is encoded in the weights and biases; when relating the material structure to another related property, the parameters of the existing DNN largely remain the same except perhaps those for the last one or few layers.
In this work, we apply TL to training DNN surrogate models for predicting adsorption capacity of gases in diverse MOF structures. We leverage our generated databases of 13,506 MOF structures and the calculated gas uptake capacity. As the first demonstration, we transfer the knowledge learned from a DNN trained using the complete 13,506 H2 uptake data points at one condition (100 bar and 243 K) (i.e., the source task) into the construction of another new DNN using limited datasets (with data size of 100) at another condition (100 bar and 130 K). Our results show improvement in the predictive accuracy can be achieved compared to direct training using small datasets. We then transfer the knowledge learned from the source task to build DNN for methane adsorption, and the improvement is again evident. We find that the excellent transferability between these cases roots from the commonality of the key descriptors of the MOF structures influencing the adsorption of these gases. Based on further analysis, TL will always work on the target tasks with low generalizability. However, the predictive accuracy in Xe/Kr adsorptions cannot be improved by TL, which can be attributed to the difference in key descriptors governing the structure-property relation. Our study demonstrates the applicability of TL in the screening of MOFs for gas adsorption, but it also identifies that including the right descriptors can be a limiting factor for applying TL.  [34] After that, the capabilities of those topologically diverse set of MOFs to adsorb H2, CH4, and Xe/Kr mixtures are calculated using GCMC, implemented using the RASPA code. [35] Additional details for generating the dataset are discussed in [28].
Transfer learning: DNN [36] with two hidden layers are used for quantifying the relationship between textural descriptors and gas adsorption, and its architecture is shown in Figure 1a. We first train a DNN using the complete 13,506 H2 uptake data points at 100 bar and 243 K (the source task) and save its weights. When training the DNNs for the other tasks, rather than learning from scratch (we also call it direct learning (DL)), we fine-tune the weights between the output layer and the last hidden layer of the pre-trained DNN from the source task while keeping the other weights fixed, using the smaller datasets available for the new target tasks (Figure 1b). The hyper-parameters of DNN are optimized on the source task before generating the final results. 5, 128, 64, and 1 are the numbers of neurons in the input layer, hidden layer 1, hidden layer 2, and the output layer, respectively. The learning rate, which serves as the step size for updating DNN parameters, is set to 0.00002, so as to make the learning process stable. The number of learning epochs is set to 10,000 with the early-stopping technique [37] applied to avoid overfitting. The whole dataset is divided into training set, validation set and test set with the ratio of 72:18:10.
The DNN is trained on the training set, and then validated on the validation set for model selection. The predictive accuracy on the validation set is calculated every 5 epochs, and the training process is stopped by the early-stopping technique once the predictive accuracy does not improve for 5 consecutive epochs.
The predictive accuracy on the test set is used for comparison between DL and TL scenarios.

RESULTS
We first train a DNN using the complete dataset (13,506 data) for H2 adsorption at 100 bar and 243 K as the source task. The trained DNN possesses a high coefficient of determination (R 2 ) of 0.998 on test set.
TL is then applied to three target tasks: (1) H2 adsorption at a different condition (100 bar and 130 K); (2) CH4 adsorption at 100 bar and 298 K; (3) Xe/Kr adsorption at 5 bar and 298 K. For each target task, 1,000 batches of randomly selected datasets (so as to make the results statistically significant), each with a size of 100 data points, are used for both DL and TL scenarios (Fig. 1c). The performances of DL and TL are compared by evaluating their respective R 2 on the same test sets.

Transfer from H2 adsorption at 100 bar and 243 K to H2 adsorption at 100 bar and 130 K.
The average R 2 on the 1,000 test sets through DL is 0.960 (Fig. 2a), while the same metric for TL is 0.991 ( Fig. 2b), which is close to that of the source task on the complete 13,506 data set (R 2 = 0.998). As can be seen from Fig. 2d for these cases as shown in the inset of Fig. 2e.
When the target tasks have very high model accuracies from DL already, the transferred knowledge from the source task does not always help further improve the model accuracy in the target tasks. TL applied to H2 adsorptions at other temperatures are also studied and found effective (Figs. S1 and S2 in the supporting information). Transfer from H2 adsorption at 100 bar and 243 K to CH4 adsorption at 100 bar and 298 K. We further generalize the application of TL from one gas species (H2) to another (CH4). The average R 2 on 1,000 test sets through DL is 0.935 (Fig. 3a), while the same metric from TL is improved to 0.980 (Fig.   3b). Figure 3d shows that most of the target cases have both high R DL . Among the 1,000 cases, TL improved the model accuracy over DL in 820 cases (i.e., R TL 2 −R DL 2 >0 ), while in the rest 180 cases, TL did not help (i.e.,

R TL
. TL is always beneficial if the case has R DL 2 < 0.891 from DL. These results show that in most cases (82%), it is beneficial to transfer knowledge learnt from H2 adsorption to modeling CH4 adsorption in MOFs, and TL is especially helpful for low performing models trained from DL. TL applied to CH4 adsorption at other condition is also found effective (Fig.S3 in the supporting information). bar and 298 K. However, in these cases, models from DL performs badly with an average R 2 on the 1,000 test sets of -0.147 (Fig. 4a), and that of TL is -0.092 (Fig. 4b).
Although TL improves R 2 in 517 out of the total 1,000 cases (Fig. 4c), neither of the two scheme yields good predictive models statistically. In general, there is still a positive relation between TL and DL performances (Fig. 4d) S4 in the supporting information). Descriptor importance quantification: To understand the distinct performance of TL in the above three examples, we quantify the feature importance of the used textural descriptors and these four gas adsorptions using random forest (RF) [38,39]. RF is a bagging method that fits a number of decision trees on various sub-samples of a dataset and uses averaging to improve the predictive accuracy and mitigate over-fitting issues. Scikit-learn [40] is used for the RF implementation and the number of trees in RF is set to 600. Here, the whole dataset (13,506 data points) are used for all four gas adsorptions, 10-fold crossvalidation are conducted and R 2 between predictions and labels on validation set are used to quantify model performance. The quantitative descriptor-adsorption relationships are shown in Knowledge is demonstrated to be very transferrable between H2 adsorptions under different conditions, and also from H2 adsorption to CH4 adsorption. TL is proved to be more effective in the target tasks with low predictive accuracies on test sets from DL, and it can help avoid very low-performing models.
However, TL fails when there is no sharable knowledge between the source task and the target task, as demonstrated by the case study of transferring from H2 adsorption to Xe/Kr mixture adsorption. Through the RF analysis on the whole dataset for all four gas adsorptions, it is understood that textural descriptors used in this study are not always appropriate for modeling the structure-property relationships. As a result, TL works well for tasks that can be described by the same descriptors and may fail for tasks that are