{'fraction_non_alphanumeric': 0.025692457449632753, 'fraction_numerical': 0.021238311554231438, 'mean_word_length': 4.623154752031846, 'pattern_counts': {'":': 0, '<': 8, '<?xml version=': 0, '>': 0, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
FIDDLE: Ecient Assembly of Networks That Satisfy Desired Behavior

: 1 Background : Due to the complexity and redundancy of biological systems, computational 2 models are difficult and laborious to create and update. Therefore, machine reading and 3 automated model assembly are of great interest to computational and systems biologists. Here, 4 we describe FIDDLE (Finding Interactions using Diagram Driven modeL Extension), the tool 5 that we built with the goal to automatically assemble or extend models with the knowledge 6 extracted from published literature. The two main methods developed as part of FIDDLE are 7 called Breadth First Addition (BFA) and Depth First Addition (DFA), and they are based on 8 network search algorithms. To assess the advantages and limitations of BFA and DFA, we 9 applied them on Erdös-Rényi random networks (ER) and Barabási-Albert scale-free networks 10 (BA) as models of biological networks. Starting with several different baseline models and 11 different sets of candidate edges, we conducted a comprehensive evaluation of the BFA and 12 DFA algorithms and their ability to either reconstruct a specified network structure or assemble 13 models that reproduce the same behavior as defined golden models. 14 Results: When BFA and DFA are applied to random networks, we are able to show that model 15 extension with BFA is more specific and DFA is a more sensitive approach. When applied to 16 scale-free networks, both BFA and DFA demonstrated very limited success. 17 Conclusion: Results suggest that BFA is a better method for assembling abundant but mixed 18 true and false information, while DFA will better assemble fewer and more truthful interactions. 19 The limited success with scale-free networks suggests a fundamental difficulty with 20 automatically creating biological models due to the inherent redundancy and complexity of 21 biological networks. These methods represent an early attempt and a novel approach to truly

approach, the model is used to predict the behavior of individual elements in the network (6, 7). 1 Recently, there has been a push to integrate the two methods, using experimental data to 2 contextualize the bottom-up approach, and incorporating prior knowledge into the top-down 3 approach to reduce the number of potential models (8-12). Despite these efforts, modeling 4 biological systems remains a tedious, iterative, and ongoing pursuit. Thus, useful models are 5 rarely built for large systems and even more rarely updated to incorporate new information. This 6 laborious process results in slow model development and few model extensions. It is because of 7 these factors, that system and computational biologists have endeavored to automate the process 8 of model creation and extension. 9 10 In the advent of Big Data, more information is available than ever before about molecular 11 interactions. To automatically create models, information can be extracted from literature, 12 queried from databases, or taken from existing baseline models. Public databases such as 13 Reactome (13) Here, we propose FIDDLE (Finding Interactions using Diagram Driven modeL Extension), a 1 tool that conducts model assembly in a bottom-up approach. The goal of FIDDLE is to assemble 2 a model that satisfies a given performance metric (defined formally in Methods), starting from a 3 list of known biological interactions, and key indicators of the studied biological phenomena. We 4 use the key indicators as ground truths when computing the error in our model performance 5 metric. FIDDLE is able to refine models by systematically adding known biological interactions 6 into intermediate models, measuring changes in model performance, and then adding or 7 discarding interactions based on whether they improve the model performance metric. FIDDLE 8 incorporates two novel algorithms, Breadth First Addition (BFA) and Depth First Addition 9 (DFA), which utilize the same principles as the breadth-first search and depth-first search 10 algorithms in network studies (32). These methods not only represent a new approach to bottom-11 up model assembly but are also used to demonstrate the existence of key biological properties 12 which hinder automated modeling of biological systems. The model structure can be illustrated 13 using networks, where model elements (such as proteins, genes, biological processes, etc.) are 14 represented by nodes, and the molecular interactions (such as activation, inhibition, etc.) are 15 represented by directed edges. Including the direction in element connections allows for the 16 representation of causal relationships between nodes, and consequently, enables studies of the 17 complex system dynamics that arises from these causal relationships. We show a high-level 18 overview of FIDDLE in Figure 1. 19 20 Figure 1: "Overview of the FIDDLE process" FIDDLE is a robust model assembly tool 21 which requires candidate edges, an expectation of model behavior, and an optional baseline 22 model. FIDDLE iteratively assembles candidate edges into candidate models, simulates 1 them, and calculates their performance using the TME metric. 2 3 To evaluate FIDDLE and the proposed BFA and DFA algorithms, we created a number of 4 artificial networks, systematically removed network edges, and tasked FIDDLE to re-build the 5 original model. We used artificial networks so that the structure and dynamics of the original 6 network were completely known, and we could vary key parameters in their creation to observe 7 the impact it would have on reassembly. We created two different types of networks: Erdös-8 Rényi random networks (ER) (33) and Barabási-Albert scale-free networks (BA) (34). Biological 9 models, power grids and the internet follow scale-free power distributions. We will evaluate the 10 two network types to examine the role of network structure and determine the algorithmic 11 performance by comparing the behavior of the original and the reconstructed model. 12

13
We also demonstrate the advantages to using our BFA and DFA algorithms for automated model 14 assembly. Throughout our analysis, we contrast the differences between them, illustrating their 15 relative strengths, weaknesses and commonalities. By using different network structures and 16 different types of candidate knowledge, we are able to gain insight into the complex nature of 17 biological models, as well as point to the future goals and difficulties with completely automated 18 model assembly. 19 20

Dataset of True and Candidate Knowledge 22
We selected a network size of approximately 50 nodes to represent a typical biological model 1 requiring reasonable computational time. We assume that complete models, before any edges are 2 removed represent golden models, ideal representations of the actual system. When creating the 3 golden models, the components, structure, and performance are assumed to be known perfectly 4 (which is most often not the case when using existing knowledge about the biological systems). 5 Once removed from the golden model, we save the edges as candidate knowledge for model 6 assembly. Finally, to account for variable candidate knowledge, we add a list of false edges to 7 the list of true edges from the golden model and use this new set as an input to FIDDLE. We 8 created two different types of random networks, and the descriptive statistics of these networks 9 are shown in Table 1. 10 11 We removed edges from the random networks assuming different probabilities of removal, 0.1, 1 0.25, 0.5, 0.75, or 1, and stored the removed edges so they could be used by FIDDLE in the 2 model assembly process. Descriptive statistics describing the removal of edges from the 50 ER 3 networks and 50 BA networks are shown in Table 2, where Model Density is defined as the ratio 4 of directed edges to nodes, and Model Degree is the sum of all individual node degrees, divided 5 by the total number of nodes. An example of edge removal is shown in Figure 1.

Analysis of FIDDLE Model Extension 19
Next, we attempted to reconstruct the models after different percentages of edges had been 20 removed from them. We will refer to edges that were originally in the golden model, before edge 21 removal, as "true" edges, and to edges that were not in the golden model as "false" edges. To 22 determine the impact of candidate knowledge on the model extension procedure, the extension 23 method takes as input two lists of candidate edges: an Exclusive list (EXC) which contains only 1 the "true" edges that need to be in the golden model and are necessary for exact model 2 reconstruction, and an Inclusive list (INC) which contains all the edges that are in the EXC, as 3 well as an equal number of "false" edges (randomly generated edges between model nodes that 4 were not in the golden model). The creation of the two lists is meant to reflect the two types of 5 information available to modelers: interactions which are plausible and based on strong literature 6 evidence, usually well known to the scientific community and supported by curated databases, 7 and the information with less evidence, such as the candidate edges that we get from the 8 automated natural language processors. We demonstrate these concepts in To measure each method's recallthe fraction of "true" edges that are selected to extend the 18 model with, given a mix of "true" and "false" edgeswe first evaluated both methods using the 19 EXC list to extend baseline models, and then we conducted tests with the INC list. The results of 20 each method are shown in Figure 5A. of edges not added to the model). Using these classification statistics, we would want a TNR 1 close to one, and an FDR and FOR close to zero. The calculated rates of TNR, FDR, and FOR, 2 are shown in Figure 5B. 3 4 In addition to model reconstruction, we also evaluated our methods on model creation. BFA and 5 DFA algorithms were tasked with minimizing the TME using only the EXC and INC list of 6 interactions and no baseline model. The recall of both BFA and DFA are shown in Figure 6A. To 7 compare the ability of BFA and DFA to detect false edges, we compared networks created by the 8 two algorithms, with respect to TNR, FDR, and FOR, as shown in Figure 6B. the bottom row, we see the plots normalized to TME. 12 13 Next, we were also interested in visualizing the assembled network that began as an ER or BA 14 network, and that was extended with the BFA or DFA method with a goal of minimizing the 15 TME. These reconstructed networks are shown in Figure 8.

Comparison of TME by Random Network Type 22
Lastly, we were interested in explaining the difference in performance of the two network types 1 with respect to the TME. In Figure 9, we began with either an ER or BA network, and removed 2 every two-edge combination from the golden model. In this way, we created a set of models 3 similar to the golden model except for a different and unique pair of edges that we removed. We 4 then created a heatmap of the TME for each reduced model, based on which two edges were 5 removed. In cases of the diagonal, only one edge was removed and therefore serves as the golden 6 model with only one edge missing. These heatmaps comparing the TME of ER and BA networks 7 are shown in Figure 9. Automated model assembly and execution is the goal for systems and computational biologists. 17 In this paper, two novel algorithms for bottom-up model creation were evaluated, and 18 emphasized how complicated the models of biological systems are to create. We used artificial 19 networks with different network structure in order to analyze the BFA and DFA algorithm 20 strengths and weaknesses. The following sections discuss the assembly algorithms, their 21 performance on the artificial models, and future work. 22

Assembling Models 1
To investigate the ability of our proposed algorithms to assemble a model with the desired 2 behavior, we selected easily tunable random ER, and scale-free BA networks. Many biological 3 systems have been observed to follow a scale-free power distribution (34). We systematically 4 and randomly removed edges from these networks, which allowed for a varying number of edges 5 to be removed from each network. The density of the network was plotted against edge removal 6 probability, in order to understand how dense these networks were, and how edge removal 7 affected density. 8 9 In Figure 3, we can see that both ER and BA models responded linearly to the removal 10 probability. The linear line observed in BA networks is a result of the method of network 11 creation. Since BA networks were created with the same number of edges (albeit in different 12 configurations, direction, and regulation), Model Density was then a linear function of removal 13 probability. In ER networks, since the number of edges to begin with were different between 14 different networks, the obtained network population is more diverse, although the networks still 15 respond linearly to the edge removal probability. Interestingly, this linear relationship is not 16 conserved when observing model performance (TME) as a function of edge removal probability. 17 Rather than a strict linear relationship (in which each edge is equally responsible for the same dip 18 in model performance), the effect of different removal probabilities seems to have no linear 19 relationship with the TME metric. It appears that a small number of missing edges can have a 20 strong impact on model performance in some cases, while a large number of missing edges may 21 register a small impact. As we will see later, it is likely that there exist multiple local minima in 22 the model TME state space but which fail to encapsulate the total model information. In this 23 way, we are able to minimize the model TME and yet the assembled model will have a different 1 structure than the golden model. Similarly, our study shows that ER networks respond differently to edge removal than BA. 7 Rather than network diameter, the shortest path between any two nodes, we use the TME metric 8 since our models are small by comparison and our networks are directed. In models that aim to 9 capture biological phenomena, nodes with a high degree of connection dwarf the importance, 10 significance, and effect of other nodes (36, 37). 11 12 Comparison of the BFA and DFA algorithms 13 Given that we are using random networks as golden models, the random edge removal makes 14 direct comparison difficult. In order to more reliably compare the two algorithms, we applied 15 them to the exact same baseline model, and we used the same set of candidate edges. We then 16 examined the recall, TNR, FDR, and FOR of the two methods. 17

18
In Figures 4 through 7, we see that the BFA algorithm performs much better in TNR, while the 19 DFA algorithm performs much better in recall. The higher TNR of the BFA is due to the 20 scanning of all possible edge additions to select an edge that leads to largest TME reduction, and 21 therefore, more selective inclusion of edges and an increased probability of including true edges 22 (if not redundant or initially detrimental). DFA exhibits higher recall and includes any edge that 23 improves model performance. As true edges are more likely to improve model performance, 1 DFA is able to add more true edges regardless of the magnitude of improvement to model 2 performance. 3 4 The BFS algorithm appears to be a better choice for model assembly when the candidate 5 knowledge is a mix of true and false edges, as it will only incorporate edges vital to the model's 6 performance. On the other hand, the DFA method would be a better choice when correct 7 information is provided, as it performs better at reconstructing the golden model with more true 8 edges. Employing the algorithms in a parallel manner would recommend the best edges that are 9 likely to be in the model and suggest additional edges to explore experimentally. Additionally, 10 the BFA method is easily parallelized for rapid execution on large models. 11

12
To assess the algorithms, we plotted the TME metric as a function of edges as they were added 13 to the model (Figure 8). The steep decrease early on of the BFA method shows the quick 14 incorporation of highly important edges that drastically reduce TME. DFA does not exhibit the 15 same performance as any improvement is added immediately. Changes in TME are similar for 16 ER and BA networks, although BA networks appear to begin with less TME and require a fewer 17 number of edges to improve model performance when compared to ER networks. This holds 18 across different percentages of missing edges except when all edges are removed. Once all edges 19 are removed, BA networks have a worse change in TME with the number of edges added; this is 20 likely due to BA networks possessing more edges on average (an artifact of the network creation 21 method). In each case, there appear to be a small number of edges which can drastically reduce 22 TME, regardless of the overall percent of edges being removed from the model. This is best seen 23 in instances of 0.5 and 0.75 removal probability. This trait disappears when there is just too 1 much information (edges) missing from the model for complex regulations and causal 2 relationships to form. In comparing the performance of BFA and DFA when given either EXC or 3 INC list, it is interesting to note that the INC list performs almost identically (or only slightly 4 better) to the EXC list in BFA. This demonstrates that only True edges are likely to improve the 5 model performance and trigger a steep decrease in TME. 6 7 Automatically Created and Extended Models 8 Next, we were interested in exploring the effect of network structure on each algorithm's 9 performance. As we attempted to reconstruct the golden models with FIDDLE, the BFA and 10 DFA algorithms were successfully able to minimize TME; however, the assembled models were 11 not the same as the golden models (Figure 8). The key reason is that these algorithms function to 12 maximize the performance metric, i.e., minimize TME, while adding the least number of edges. 13 Thus, this approach often results in finding local minima and simplifying the model to its most 14 basic causal interactions that reproduce the expected behavior. We demonstrate some of the 15 limitations of applying a simple error function in Figure 9, where two missing true edges can 16 result in both much better and much worse model performance than either edge individually. 17 There have been similar analyses on the complicating factors of inferring biological networks 18 (38). In a comprehensive analysis of the DREAM3 challenge (38, 39), four main errors were 19 found that would drastically reduce the reconstruction method's confidence and directly impact 20 the algorithm's performance: fan-out error, fan-in error, cascade error, and feed-forward loop 21 error. These errors are inherent symptoms of characteristics of the underlying model structure: 22 highly redundant network connectivity and complex network motifs. 23 In a review by Kitano [2], he argues that robustness is a fundamental feature of evolvable 1 complex systems, necessary for them to be robust to environmental and genetic mutations. He 2 describes four key mechanisms that help ensure robustness: system control, alternative 3 mechanisms, modularity, and decoupling. Both the existence of alternative mechanism and 4 complex system controls make it difficult to understand and model these systems. The presence 5 of alternative mechanisms allows for high levels of redundancy in biological networks while 6 system control requires a complex interplay of multiple interactions where missing a small 7 subset will fundamentally change the dynamics of the model. Luckily, biological modularity may 8 hold the key into how we can best understand and model these systems. In this paper, we focused 9 on singular molecular interactions and adding one at a time. Perhaps through analyzing potential 10 interactions in the context of the system and understanding the modules as a whole, we may 11 make model assembly and extension easier as these modules can be added together. 12 13

Implications for Creation and Extension Methods 14
This paper describes an approach for the automatic assembly of biological models. The efficient 15 incorporation of novel information, especially from machine reading systems, is critical to 16 maintain updated models of biological systems. The analysis conducted in this work also enables 17 comparisons with other algorithms such as genetic (31) and clustering algorithms (30) and future 18 efforts to combine these algorithms into a hybrid approach for model extension. There has been 19 considerable work (30) showing different clustering methods of model elements and these will 20 be explored for model assembly. We plan on continuing this study by adding groups of 21 interactions together, the best number to add together and interrogate the traits that make edges 22 "good" to be added together. This is a complex problem as it not only depends on what type the 23 edges are and where they are added but how they are integrated into the network. Also, while the 1 FIDDLE algorithms do not depend on the type of update functions used to simulate the model 2 (arithmetic, Boolean, etc.), the type of update function is expected to impact model extension 3 accuracy. 4 Further improvements can be realized using a rigorous method for testing models, such as 5 statistical model checking, which allows for efficient verification of system properties (40). 6 Therefore, model checking can be used to select models that best recapitulate real biological 7 systems by estimating how well models satisfy observed or desired system properties. 8 9

Conclusions 10
We have been able to illustrate the advantages and limitations of utilizing breadth or depth first 11 addition algorithms for assembling biological networks from new candidate knowledge extracted 12 from literature. In general, automated assembly of biological models is challenging due to the 13 inherent redundancy and complexity in biological networks. In this paper, we have shown that 14 BFA has higher TNR and DFA is more sensitive when it comes to assembling biological models. 15 These relative strengths suggest different applications: BFA could be used to extend a model in a 16 more focused way, more likely to include only true biological connections while DFA could be 17 used to suggest many different interactions that could be explored for applicability. In general, it 18 appears that BFA is a better method to use if there is an abundance of inaccurate information, 19 while DFA is a better method for accurate information and assembly of a truly representative 20 model. In the future, we plan on improving our algorithms to further advance the effectiveness of 21 automated model assembly from interactions extracted from literature and public databases. 22

Methods 1
The FIDDLE tool can be used as a stand-alone tool, or as a part of a larger framework of tools. 2 In this work, we use the DySE framework (41), to simulate, and analyze the models assembled 3 by FIDDLE. Both FIDDLE and DySE are implemented in Python. In addition, the Python 4 package, NetworkX (42) was used to create all networks, while Graphviz (43) was used to 5 visualize them. In this work, we use Discrete, Stochastic, Heterogeneous simulator (DiSH) (44), which allows 1 for simulations of discrete models with various types of update functions, and has several 2 different simulation schemes, including deterministic and stochastic. For the analysis we 3 conducted here, we used a uniform, step-based, random-ordered, sequential update scheme 4 (USB-RSQ), described in detail in (44). It has been shown previously (44, 45) that the USB-RSQ 5 simulation scheme is able to recapitulate the intra-cellular network dynamics. DiSH simulates 6 the models starting from an initial state (assigned before simulations), and for a pre-defined 7 number of time steps, T (e.g., when the steady state is reached). Each such simulation run, r, 8 yields for every model element vi V, a trajectory of values, v = ( v ,1 , v ,2 , … v , ), where 9 v , is the state value of element vi at time point t (t=1,..,T), within run r. Due to the randomness 10 of the update scheme, element trajectories may vary across multiple runs that start with the same 11 initial state. Therefore, for the same time step t, following the approach from (44), we average 12 the values v , across different runs, to obtain average trajectories for all elements. More 13 formally, we compute an average trajectory of model element vi as: 14 where R is the overall number of conducted simulation runs. Additionally, we denote the average In the experiments that we describe here, for all model elements vi V, we assume that their 20 initial state value is v ,1 = 1, and for each created model, we conducted R=100 simulation runs, 21 each with T = 2,500. Given that the networks we use here are artificial (they do not represent any 22 real biological system), there is no information about initial values that should be used for 23 elements, and therefore, we used the middle value among the three possible state values, to 1 minimize the bias that the lowest and highest state values can have on model dynamics. 2 3

Model Performance Metric 4
Model accuracy is usually measured by evaluating the model against a set of ground truths, or 5 expected behaviors, that specify how the model is supposed to respond given a certain initial 6 state and input values. Since our goal is to automatically assemble models, we incorporated 7 within our assembly algorithms a metric that assesses how closely to the specified behavior 8 newly created models perform. Our model performance metric examines the states of all model 9 elements in the last time step T of the average trajectories (usually, a steady state). We denote the 10 expected final state vector of the desired behavior, either observed in a real system or assumed to 11 hold for a golden model, as QGM. Any model that we assemble is considered a candidate model, 12 and we denote its final state as QCM. We define Total Model Error (TME) as a measure of the 13 difference between the golden and the candidate model behavior, that is, we compute the 14 absolute distance between vectors QGM and QCM: 15

Eq 2 16
where N represents the number of elements in the model. The Breadth First Addition (BFA) procedure starts by evaluating the contribution of each new 2 edge from ENEW to decreasing TMEBL, that is, we simulate the model that consists of our original 3 baseline model MBL and a selected new edge, and we compute TME of that extended model 4 according to Eq 2. Next, we permanently move, from ENEW to the model, the new edge that leads 5 to the largest decrease in the original TMEBL, and then we repeat the procedure with this new 6 extended model, i.e., similar to what we did with the original model, we evaluate addition of the 7 remaining edges from ENEW to this new model by computing their TME values. This process is 8 repeated until we either find the extended model that matches the expected end values of the 9 golden model (QGM), there are no more edges in ENEW, or no edge can be added to the baseline 10 without increasing the TME. Our BFA algorithm is outlined more formally in Figure 10. The basic structure of the BFA algorithm in a pseudocode format. 14 15 The Depth First Addition (DFA) procedure, similar to the BFA algorithm, starts with 16 evaluation of edges from ENEW by computing their contribution to decreasing TME of the 17 baseline model. Different from BFA, as soon as we find an edge which leads to a TME lower 18 than the TMEBL, we add that edge to the baseline model, and remove it from ENEW. We repeat 19 this procedure using the new extended model and the remaining edges from ENEW, that is, as 20 soon as we encounter an edge that leads to a lower TME than the current model, we add that 21 edge to the model. The DFA algorithm stops when we either find the extended model that 22 matches the expected end values of the golden model (QGM), there are no more edges in ENEW, or 23 no edge can be added to the baseline without increasing the TME. Our DFA algorithm is 1 outlined more formally in Figure 11. To evaluate how well FIDDLE can assemble models and to compare and contrast the BFA and 8 the DFA approach, we apply a method commonly used for model reconstruction (46-50). The 9 first step of this method is a generation of golden models. We used two different types of random 10 networks to create our golden models: one type based on Erdös-Rényi random directed networks 11 (33), and another using the Barabási-Albert scale-free network (34). To generate an Erdös-Rényi 12 Network (ER), a random network was seeded with 50 nodes and a probability of connection of 13 0.035. To create a Barabási-Albert scale-free network (BA), each network was seeded with fifty 14 nodes and two initial edges. For both the ER and BA networks, these parameters were chosen to 15 create artificial networks of roughly the same number of nodes, edges, and density (as shown in 16 Table 1). Models were kept around 50 nodes to best replicate the synthetic models used in the 17 DREAM3 challenge (39). We used the Python library NetworkX (42) to generate these networks 18 and to calculate the network statistics reported in Table 1. Next, we assigned each directed edge 19 in these networks, randomly with equal likelihood, either a positive or negative sign (activating 20 or inhibiting, respectively). Finally, we simulated golden models to record the QGM values, which 21 are used by the TME function to guide the BFA and DFA algorithms. 22 In the second step of the evaluation procedure, we created baseline models by randomly 1 removing edges from the golden models. Specifically, we select the removal probability to be 2 0.1, 0.25, 0.5, 0.75, or 1, and for each edge in the model we randomly choose whether to remove 3 it or not with that selected probability. For example, in a model with 0.5 edge removal 4 probability, an edge was selected, given a 50-50 chance of being removed, either removed or not, 5 and then the next edge was selected. If an edge was removed in a model of low removal 6 probability, it was also missing from a model of higher removal probability. The removed edges 7 were stored in a list called the Exclusive list (EXC) since it only contains edges that were 8 removed from the golden model. We also create an   Figure 1 "Overview of the FIDDLE process" FIDDLE is a robust model assembly tool which requires candidate edges, an expectation of model behavior, and an optional baseline model. FIDDLE iteratively assembles candidate edges into candidate models, simulates them, and calculates their performance using the TME metric. Figure 2 "Illustration of edge removal with varying probabilities" Example of the process of randomly removing model edges with different probabilities. If an edge is removed with 0.1 probability, that edge will also be removed from the network for all higher probabilities of edge removal.   "Comparison of performance statistics of model extension" (A) Box plots comparing the quartile ranges of the recall of both algorithms under different conditions. The top row is for models based on the ER network while the bottom row is based on BA networks. The left column is comparing the two algorithms when using only the EXC list, while the right column is comparing the algorithms when using the INC list, with equal number of true and false edges. Statistical signi cance was determined by t-test (* p < 0.05, *** p < 0.001). (B) Box plots comparing the quartile ranges of the TNR , FDR, and FOR of both algorithms under different conditions. The top row is for models based on the ER while the bottom row is based on BA. In all cases, both algorithms were given the INC list, with equal number of true and false edges.
Statistical signi cance was determined by t-test (* p < 0.05, *** p < 0.001). Figure 6 "Comparison of performance statistics of model creation" (A) Box plots comparing the quartile ranges of the recall of both algorithms under different conditions. The top row is for models based on the ER random network while the bottom row is based on BA networks. The left column shows the results obtained when using the EXC list in both algorithms, while the right column shows the results when the algorithms were given the INC list, with equal number of true and false edges. Statistical signi cance was determined by t-test (* p < 0.05, *** p < 0.001). (B) Box plots comparing the quartile ranges of the TNR, FDR, and FOR of both algorithms under different conditions. The top row is for models based on the ER random network while the bottom row is based on BA networks. In all cases, both algorithms were given the INC list, with equal number of true and false edges. Statistical signi cance was determined by t-test (* p < 0.05, *** p < 0.001). "Model performance as edges are added back into the model" In the top row of this gure, we see plots of the model TME (y-axis) with number of edges added (x-axis). In the bottom row, we see the plots normalized to TME.  "Heatmap of network response to combinatorial edge removal" These heatmaps compare the differing responses of ER and BA networks under random edge removals. Columns represent the TME for a model with the same edge missing, while rows also represent the TME for a model with a different edge missing. The diagonal represents models with only one edge missing. Figure 10 "Pseudocode of the BFA Algorithm" The basic structure of the BFA algorithm in a pseudocode format. Figure 11 "Pseudocode of the DFA Algorithm" The basic structure of the DFA algorithm in a pseudocode format.