{'fraction_non_alphanumeric': 0.03530088431950536, 'fraction_numerical': 0.01653605579121432, 'mean_word_length': 5.143992932862191, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 0, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 2, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Materials development by interpretable machine learning

Machine learning technologies are expected to be great tools for scientific discoveries. In particular, materials development (which has brought a lot of innovation by finding new and better functional materials) is one of the most attractive scientific fields. To apply machine learning to actual materials development, collaboration between scientists and machine learning is becoming inevitable. However, such collaboration has been restricted so far due to black box machine learning, in which it is difficult for scientists to interpret the data-driven model from the viewpoint of material science and physics. Here, we show a material development success story that was achieved by good collaboration between scientists and one type of interpretable (explainable) machine learning called factorized asymptotic Bayesian inference hierarchical mixture of experts (FAB/HMEs). Based on material science and physics, we interpreted the data-driven model constructed by the FAB/HMEs, so that we discovered surprising correlation and knowledge about thermoelectric material. Guided by this, we carried out actual material synthesis that led to identification of a novel spin-driven thermoelectric material with the largest thermopower to date.


3
Recent progresses of material science technologies enable us to obtain a lot of material data in a short time [1][2][3][4] . Accordingly, the development of data-processing technologies for such material big data is becoming indispensable. Machine learning technologies are believed to be a great solution due to not only their rapid data analysis [5][6][7][8] but also their potential to discover non-trivial knowledge that is not rooted in conventional material theory.
To apply machine learning to actual materials development, collaboration between scientists and machine learning is becoming inevitable. Material scientists often try to understand the inside of the data-driven model to obtain some actionable information for materials development. However, such collaboration has been restricted so far because of the low interpretability in machine learning. For example, it is not easy for a person to understand the inside of data-driven models constructed by a deep neural network 9 , where the data-driven model is expressed as the connections between large numbers of perceptrons (neurons). Therefore, the notion of interpretable machine learning (explainable or transparent machine learning), which has not only high prediction accuracy but also high interpretability, has recently seen a resurgence [10][11][12] , especially for scientific discoveries led by machine learning.
Here, we show an actual material development success story by using one type of interpretable machine learning called factorized asymptotic Bayesian inference hierarchical mixture of experts (FAB/HMEs) [13][14][15][16] and demonstrate good collaboration between the FAB/HMEs and material scientists. In the material development, the machine learning algorithm must meet the following three requirements, "sparse modelling", "prediction accuracy", and "interpretability", as shown in Figure 1. The material data is often sparse; therefore, the sparse modelling approach, which automatically selects some of only the important descriptors (attributes), is useful.

4
One of the most popular sparse modelling methods is LASSO, which constructs a linear model with L1 regularization 17 . However, such a linear regression model does not always have high prediction accuracy because material data often includes non-linear relationships such as phase transitions. For high prediction accuracy, non-linear modelling using, for example, a support vector machine (SVM), neural network (NN), or random forest (RF) can construct a flexible model 17 . However, such non-linear modellings do not have enough interpretability. Although these non-linear machine learnings can tell us which descriptors are important for the data-driven model, they do not clarify how the descriptors actually contribute to it. Extracting actionable information from such non-linear models is not so easy.
Interpretable machine learning FAB/HMEs constructs a piecewise sparse linear modelling 13 that meets the three requirements of "sparse modelling", "prediction accuracy", and "interpretability". Therefore, the actionable information from the data-driven model provided by the FAB/HMEs smoothly leads us to discoveries of novel materials. For the case study, we applied the interpretable machine learning FAB/HMEs to develop a new thermoelectric material. Thermoelectric technologies are becoming indispensable in the quest for a sustainable future 18,19 . In particular, the emerging spin-driven thermoelectric (STE) material has garnered much attention as a promising path toward low cost and versatile thermoelectric technology with easily scalable manufacturing [20][21][22][23] . However, STE material development is hampered by the lack of understanding of the fundamental mechanism of STE material because STE phenomena are at the cutting edge of material science and physics. A data-driven approach by machine learning can exhibit its full potential in such an unexplored scientific field.
By searching for STE materials with improved thermopower SSTE (one of the most important figures of merit for thermoelectric phenomena) 24 , we have used interpretable machine learning FAB/HMEs to discover nontrivial behaviors of material parameters governing the STE phenomena. We have successfully leveraged this machine-learning-informed knowledge to discover a novel high-performance STE material, whose thermopower is far greater than that of the best known STE material to date 22 .

6
The material parameters ≡ { 1 , 2 , 3 … 14 }, whose simple descriptions are shown in Table 1, were obtained by conventional material simulation technology (high-throughput DFT calculation 3,4 ). The details about the data, experiments, DFT calculations, data-preprocessing, and the reason we use these material parameters X are shown in Supplemental Information 1. FePt binary alloy. b. CoPt binary alloy. c. NiPt binary alloy. d. Machine learning modelling by FAB/HMEs. With these data, we constructed the following data-driven model by the FAB/HMEs.
where X, I, S, and C are the material parameters ≡ { 1 , 2 , 3 … 14 }, their interaction terms ≡ { and data-regression problems at the same time by maximizing a novel information criterion (factorized information criterion, which is referred to as FIC) with an EM-like algorithm (factorized asymptotic Bayesian inference, referred to as FAB), thus constructing a piecewise sparse linear model [13][14][15][16] .  For instance, we notice that the data is firstly classified by average spin moment (X14) at gate 1. The data with small average spin moment X14 go to component 1, where the thermopower SSTE is equal to zero (SSTE = 0, as shown in Figure 3b). This is natural from the viewpoint of material science and physics. It is known that STE phenomena are not observed on non-magnetic materials with small average spin moment X14 25 . The data of magnetic materials with large average spin moment X14 only have finite SSTE values; accordingly, they go to gate 2. Similarly, from the interpretable data-driven model, we can obtain some trivial knowledge that is consistent with material science and physics (this trivial knowledge is shown in Supplemental Information 2).
Fortunately, we sometimes obtain non-trivial knowledge from the data-driven model. We notice that there is a positive correlation between SSTE and product term X2X8, where X2 and X8 are the amount of Pt atoms and the Pt spin polarization, respectively. All regression models of magnetic materials (component 2, 3, and 4) have the X2X8 term as positive, as shown in Figure 3b. This positive correlation uncovered by the machine learning models appears to be beyond our current knowledge of STE. The details of the physical interpretation underlying this relation is also discussed in Supplemental Information 2. The surprising connection between SSTE and X2X8, which was discovered by the interpretable machine learning here, will lead to a more comprehensive understanding of the fundamental mechanism of STE phenomena. As a result of material screening for large X2X8 by the conventional material simulation, we found that Co50Pt50N10 has large X2X8 (the details are shown in Supplemental Information 3). Therefore, as an initial example, we carried out actual material synthesis of Co50Pt50Nx and measured its thermopower SSTE. Figure 4 shows the results of thermopower SSTE of Co50Pt50Nx materials with different N concentration. It is clear that SSTE increases with increasing X2X8 accompanying an increase in N concentration. The SSTE of Co45.6Pt47.7N6.7 was achieved at 13.04 μV/K, which is larger than that of the current generation of STE materials 22,23 . Details of this actual material synthesis are given in Supplemental Information 4. This data-driven material development succeeded because the interpretable machine learning helped us a great deal by disclosing the non-trivial positive correlation between SSTE and X2X8 due to its high interpretability. This novel STE material with the dramatically enhanced thermopower discovered here will be a key component in thermal energy harvesting systems, leading to a more energy-efficient future. Co48.9Pt51.1N7.2 thin film was achieved at 13.04 μV/K, which is larger than that of STE materials to date. Figure 5 shows the 4-fold cross-validation root mean square error (CV-RMSE) of the FAB/HMEs and major machine learning algorithms including the neural network (NN), support vector machine (SVM), random forest (RF), least absolute shrinkage and selection operator (LASSO), and multiple linear regression (MLR). Non-linear models using the FAB/HMEs, NN, SVM, and RF have better prediction accuracy than linear models using LASSO and MLR. The prediction performance of the FAB/HMEs is comparable to that of other non-linear models.

Discussion
Although the NN and SVM exhibit high prediction performance, it is not easy for scientists to investigate the data-driven model because of their low interpretability. Even though the RF, which tells us the feature importance of each descriptor, has better interpretability than the NN and SVM, further investigation is needed to understand the more specific relationship between VANE and descriptors. For example, if you only look at the RF model, it is difficult to notice the "positive" correlation between VANE and X2X8 "on the condition of" the magnetic material data (data with large X14). Actually, this surprising correlation was not achieved directly via the MLR, LASSO, NN, SVM and RF. On the other hand, the FAB/HMEs obviously visualized such a detailed relationship, as shown in Figure 3a and 3b. As described above, the interpretable machine learning FAB/HMEs provides the data-driven model with not only high prediction performance but also high interpretability even in the case of sparse data. In summary, we have demonstrated the utility of interpretable machine learning FAB/HMEs in the material development process. Because of its high prediction performance and interpretability, material scientists can collaborate with the data-driven model obtaining non-trivial knowledge for novel materials development. Guided by the surprising correlation discovered by the FAB/HMEs, we have succeeded in developing an innovative STE material, whose thermopower SSTE is much larger than that of the current generation of STE materials. In addition, the non-trivial knowledge we found in the data-driven model can lead to a more comprehensive understanding of the mechanism of emerging STE phenomena. Thus, the enhanced collaboration between scientists and machine learning can lead to not only materials developments but also diverse scientific discoveries. [13][14][15][16] . The factorized asymptotic Bayesian inference hierarchical mixture of experts (FAB/HMEs) constructs a piecewise sparse linear model that assigns sparse linear experts to individual partitions in feature space and expresses whole models as patches of local experts. By maximizing the factorized information criterion including two L0-regularizations for partition-structure determinations and feature selection for individual experts, the FAB/HMEs performs the partition-structure determination and feature selection at the same time. To maximize the FIC, a factorized asymptotic Bayesian inference (FAB), which combines an expectationmaximization (EM) algorithm with an automatic shrinkage of non-effective experts, is used. In this paper, we set the termination condition δ = 10 −5 , shrinkage threshold ε = 0.062, and number of initial experts was 32 (i.e. 5depth symmetric tree). The 4-fold cross-validation root mean square error (CV-RMSE) was 0.188253, as shown in Figure 5.

FAB/HMEs
The neural network (NN) models the data by means of a statistical learning algorithm mimicking the brain.
Here, we have utilized a simple 3-layer perceptron. The cross-validation with "caret (nnet)" package in the R programming language decides the number r of hidden units = 8 and the decay value D = 3.91 × 10 −3 . The CV-RMSE was 0.169454, as shown in Figure 5.
The support vector regression (SVR) constructs a data-driven model with a kernel method. Here, we have used the radial basis function (RBF) kernel. We set the cost value C = 16 and sigma of the RBF σ = 3.125 × 10 −2 , which were decided by the cross-validation with "caret (svmradial)" package in the R programming language. The CV-RMSE was 0.132847, as shown in Figure 5.
The random forest (RF) is an ensemble learning method using a multitude of decision trees. The number of trees (ntree) and the number of features (mtry) were set to 1000 and 120, respectively. We have performed