{'fraction_non_alphanumeric': 0.030301799423209715, 'fraction_numerical': 0.00893618749746131, 'mean_word_length': 4.935985533453888, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 0, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 3}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 2, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
TFix+: Self-configuring Hybrid Timeout Bug Fixing for Cloud Systems

Timeout bugs can cause serious availability and performance issues which are often difficult to fix due to the lack of diagnostic information. Previous work proposed solutions for fixing specific type of timeout-related performance bugs. In this paper, we present TFix+, a self-configuring timeout bug fixing framework for automatically correcting two major kinds of timeout bugs (i.e., misused timeout bugs and missing timeout bugs) with dynamic timeout value predictions. TFix+ provides two new hybrid schemes for fixing misused and missing timeout bugs, respectively. TFix+ further provides prediction-driven timeout variable configuration based on runtime function tracing. We have implemented a prototype of TFix+ and conducted experiments on 16 real world timeout bugs. Our experimental results show that TFix+ can effectively fix 15 out of tested 16 timeout bugs.


INTRODUCTION
Timeout is commonly used to handle unexpected failures in complex distributed systems. For example, when a server A sends a request to another server B, A can use the timeout mechanism to avoid endless waiting in case B fails to respond. Timeout bugs can severely impact system availability and performance, causing system hang and performance degradation [11], [14], [19]. Our previous bug study [11] shows that 78% of real world timeout bugs are caused by missing timeout mechanisms or mis-using timeout schemes (e.g., setting a too small or a too large timeout value). For example, a misused timeout bug caused Amazon DynamoDB to experience a fivehour service outage in 2015 [3]. The root cause of this bug is an improper timeout value setting under unexpected workload increase. Timeout bugs are often difficult to fix because of the lack of diagnostic information and the correct timeout value often depends on the runtime execution environments and application workloads.
Previous work [16], [17] has proposed bug fixing schemes for specific bug types (e.g., hang bugs, data corruption bugs) using static code analysis methods. However, an effective generic timeout bug fixing solution requires both runtime knowledge (e.g., network bandwidth for setting a proper RPC timeout variable) and root cause function localization [16], [17]. In this paper, we propose a hybrid approach to fixing both misused and missing timeout bugs. We introduce predictiondriven timeout configuration scheme to realize self-configuring fully automated timeout bug fixing.

A Motivating Example
To better understand how real-world timeout bugs happen, and how they can affect cloud services, we use the Yarn-1630  (v2.2.0) 1 bug as one example shown by Figure 1. This bug is caused by missing timeout settings on asynchronous polling operations in YarnClientImpl. When submitting an application, the Yarn client periodically polls the ResourceManager to get the application status (line #154), until the application is submitted successfully (line #155). With a slow submission process in the ResourceManager end, the Yarn client gets stuck, causing system hang indefinitely. Figure 1 also shows the patch generated by TFix + . To break the Yarn client from the infinite waiting while still guaranteeing that the asynchronous communications between the Yarn client and the ResourceManager can succeed during normal runs, 1. We use "system name-bug #" to denote different bugs.
TFix + inserts a timeout mechanism in the loop body (added lines between line #156 to #170) with user configurable timeout variables (added lines before line #142). The automatically generated path by TFix + saves users diagnosing and development effort to fix the bug. To further ease the users' testing work and better accommodate their runtime environments, TFix + provides dynamic timeout value predictions based on runtime monitoring data. In this example, we configure the timeout value as 1130 ms based on the runtime available resources and our workloads.

Contribution
In this paper, we present TFix + , a self-configuring hybrid timeout bug fixing system. Compared with its preliminary version TFix [16], which focuses only on fixing misused timeout bugs using static analysis methods, TFix + provides selfconfiguring holistic fixing solutions for both misused and missing timeout bugs. TFix + proposes dynamic timeout value prediction scheme using both runtime tracing and static taint analysis. When a timeout bug is detected by a runtime bug detection tool such as TScope [15], TFix + executes a drill-down bug analysis protocol to automatically narrow down the root cause of the detected bug and produce bug fixing patches with sound timeout value recommendations. Specifically, TFix + first determines whether the detected timeout bug is caused by misused timeout bugs (i.e., incorrectly used timeout variables) or missing timeout bugs (i.e., lacking timeout mechanisms). For misused timeout bugs, TFix + identifies timeout affected functions from an application performance trace and then uses those functions to pinpoint misused timeout variables using taint analysis. For missing timeout bugs, TFix + identifies the root cause as either an infinite loop or a blocking function call via stack trace analysis, and inserts different timeout mechanisms with configurable timeout variables, respectively. To achieve high efficiency and obtain strong robustness, TFix + recommends a proper timeout value to adapt to the user runtime environments based on the historical execution time of the pinpointed timeout affected functions during normal runs. TFix + validates the generated patch by checking whether the bug still occurs under the same workload and all the test suites are passed after adopting the patch. Specifically, our paper makes the following contributions.

•
We describe a holistic drill-down bug analysis framework which can automatically narrow down the root cause of a timeout bug and provide a complete bug fixing patch. • We describe a hybrid scheme that combines dynamic application performance tracing and static taint analysis to find the timeout affected function and localize the misused timeout variable when the bug is caused by mis-using a timeout mechanism.
• We propose a dynamic timeout value prediction scheme to provide proper timeout value recommendation based on the historical execution time of the timeout affected function.

•
We have conducted an empirical study over 91 real production timeout bugs to quantify the coverage of our fixing strategy. TFix + can fix 79% bugs completely. We have implemented a prototype of TFix + and conducted experimental evaluation over 16 real world reproduced timeout bugs. The results show that TFix + can successfully fix 15 out of 16 tested bugs and provide correct timeout configurations in seconds. The rest of the paper is organized as follows. Section 2 describes design details. Section 3 presents the experimental evaluation. Section 4 discusses the limitation of TFix + . Section 5 discusses related work. Finally, the paper concludes in Section 6.

SYSTEM DESIGN
In this section, we present the design details of the TFix + system. We first provide an overview of TFix + , followed by the fixing strategies of misused and missing timeout bugs, respectively. Next, we describe our prediction-driven timeout value configuration and the patch validation.

Approach Overview
As shown in Figure 2, when a server system experiences software hang or performance slowdown, TFix + leverages TScope [15] to identify whether the anomaly is caused by a timeout bug after analyzing a window of system call trace collected by the kernel tracing module LTTng [12]. If TScope confirms that the performance anomaly is caused by a timeout bug, TFix + is triggered to conduct further drill-down analysis. TFix + first performs timeout bug classification to determine whether the timeout bug is caused by missing timeout check or incorrectly using timeout variables, which is described in our previous work [16].
If a misused timeout bug is confirmed, TFix + identifies those timeout-affected functions by checking the abnormality of their execution time and frequency from application function traces (Section 2.2.1). TFix + further checks whether the timeout-affected functions use a too large or a too small timeout value when the bug happens. (Section 2.2.2). For both cases, TFix + uses corresponding checking conditions to pinpoint the misused timeout variables associated the timeoutaffected functions through static taint analysis (Section 2.2.3).
If a missing timeout bug is confirmed, TFix + identifies the root cause as either an infinite loop or a blocking function call by examining the stack traces (Section 2.3.1). TFix + then adopts different patching strategies to add timeout mechanisms, either inside the loop body or surrounding the blocking function call (Section 2.3.2).
Lastly, TFix + produces prediction-driven timeout value configuration to adapt to the user runtime environment automatically based on historical execution data and performs dynamic validations over the generated patch (Section 2.4).

Fig. 3: A trace example of Dapper.
The whole drill-down bug diagnosis protocol is executed automatically without requiring any human intervention. We will describe each component in detail in the following subsections.

Misused Timeout Bug Fixing
In this subsection, we discuss how TFix + identifies the timeout variable for a misused timeout bug. TFix + first identifies the timeout affected function, then differentiates whether the bug is caused by a too large or a too small timeout value. For each case, TFix + takes different matching criteria to identify the misused timeout variable via static taint analysis.

Timeout Affected Function Identification
After classifying a detected bug as a misused timeout bug [16], TFix + identifies the timeout affected functions by checking the abnormality of the functions' execution time and frequency. To achieve this goal, TFix + leverages a commonly used application performance tracing tool, i.e., Google's Dapper framework [27]. Dapper allows us to trace the beginning and ending timestamps of all function calls and the control flow graph for the diagnosed bug. We choose Dapper tracing tool because it supports distributed systems and incurs low runtime overhead to production systems. The existing implementations of Dapper tracing can only be applied to RPC related functions. TFix + augments the Dapper tracing tool to support all timeout related functions, such as IPC functions and synchronization functions [16]. After retrieving a Dapper trace for a target bug, we first extract the execution time and frequency of all the functions invoked when the bug happens. Specifically, we calculate the frequency of each function by simply counting how many times it is invoked in the Dapper trace. We calculate the execution time of each function by subtracting the beginning time from the ending time. Figure 3 shows a Dapper trace example with various labels indicating different information. Among them, "b" and "e" indicate the beginning timestamp and the ending timestamp of a function, respectively. "d" represents the function name and "r" represents the process name. We identify the timeout affected functions by checking the abnormality in the functions' execution time and frequency.

Misused Timeout Bug Differentiation
TFix + checks whether timeout affection functions use a too large or a too small timeout value based on the following rationales. If the timeout value is set to be too large, the execution time of the timeout-affected functions become longer than their normal execution time, causing system hang or slowdown. In contrast, if the timeout value is set to be too small, the system will experience repeated failures due to frequent timeout. Therefore, the frequency of the timeoutaffected functions become higher than their normal execution frequency.
The  messages with the HBase server (e.g., HMaster, RegionServer) within tens of seconds. However, when the HBase server fails, the HBase client hangs for about 24 days, significantly increasing the execution time of the HBase client's RPC function. TFix + identified the RPC function as the timeout affected function which uses a too large timeout value.
The HDFS-4301 bug shows the example where the timeout value is set too small. The timeout value is set to 60 seconds. During normal run, NameNode transfers file system image to the Secondary NameNode within 60 seconds. However, when the image size is too large or network congestion occurs, the transmission process cannot be finished in 60 seconds. Under this circumstance, the system experiences repeated timeout failures because the system retries the image transfer but fails each time. TFix + identified the doGetUrl function as the timeout affected function because their invocation frequencies significantly increase. Besides, the execution time of doGetUrl function is nearly the same as the timeout variable value setting (i.e., 60 seconds).

Misused Timeout Variable Identification
TFix + correlates the pinpointed functions with different timeout variables included in the system configuration files in order to identify the specific misused timeout variables that are attributing to the timeout bug.
To localize which timeout variable is used when the bug happens, we first retrieve all the timeout variables in the target system. In large scale distributed systems, timeout variables along with other configurable parameters are often stored in specific configuration files [33]. For example, in a Hadoop system, all the configurable variables are defined with default values in configuration files, such as HConstant and DFSConfigKeys classes. These variables' value can be overridden and customized by users in .xml configuration files. Thus, all the variables appear in systems' configuration files and contain "timeout" keyword in their names are potentially  related to misused timeout bugs. Next, we taint all these timeout variables using static taint analysis tools [1] and conduct data flow dependency analysis on them to extract all related variables. We then check whether the timeout affected functions use the timeout related variables. If a timeout affected function f uses a timeout variable v t , we consider v t as a misused timeout variable candidate. We then compare the execution time of f with the value of v t to identify the misused timeout values.
If the bug is caused by a too large value, the execution time of f should be no larger than the value of v t , considering the system experiences a long delay and we may terminate the system tracing earlier. If the bug is caused by a too small value, we consider v t as the misused timeout variable if the execution time of f matches the value of v t .
For example, Figure 4 shows how TFix + uses the static taint analysis to identify the misused timeout variable for the HDFS-4301 bug. In this bug, the default timeout value is set to 60 seconds in DFS_IMAGE_TRANSFER_TIMEOUT_DEFAULT in DFSConfigKeys.java.
If users configure the timeout variable dfs.image.transfer.timeout in hdfs-site.xml, the system uses the configured value. Otherwise, the system uses the default value. We annotate both dfs.image.transfer.timeout and DFS_IMAGE_TRANSFER_TIMEOUT_DEFAULT as tainted. After applying static taint analysis, we find that the timeout affected function doGetUrl uses both tainted variables at line #271-273. Since the user configures the value of dfs.image.transfer.timeout in hdfs-site.xml, we determine that the misused timeout variable is dfs.image.transfer.timeout. We also perform cross validation between the timeout variable value and the execution time of the timeout affected function to confirm whether our timeout variable identification is accurate.

Missing Timeout Bug Fixing
In this subsection, we describe how TFix + identifies the root cause of missing timeout bugs as either an infinite loop or a blocking function call and inserts proper timeout mechanisms to fix the missing timeout bugs.

Root Cause Function Pinpointing
Our previous study [11] shows that missing timeout usually causes system stuck at a certain state, i.e, repeating on executing a block of code inside of an infinite loop or endlessly waiting for an asynchronous function call to finish. We use the stack trace analysis to pinpoint the hanging functions with infinite loops and the blocking function calls. After a timeout bug is classified as missing timeout, TFix + dumps the stack trace of the running system continuously. Figure 5 shows an example of the dumped stack trace on a process. The trace contains each thread's status, (e.g., runnable or waiting) and the detailed information of each Java function, (e.g., function name and the line number). The per-thread trace also shows the explicit control flow that for the two contingent functions, the top function is the callee of the bottom function.
To localize the hang/blocking functions, TFix + first extracts the common function(s) in multiple traces. This is because hang and blocking are long-term stable status. The same function call appears in all the dumped traces, indicating the manifestation of the hang or blocking. TFix + then extracts the inner-most common functions as the candidate root cause functions.
TFix + pinpoints the root cause functions after filtering out those candidate functions triggered by long-running background processes, e.g., the heartbeat process, by analyzing an interrupted application performance trace using Dapper [27]. Specifically, TFix + forces the hang/blocking function to terminate abruptly after a period of tracing time. TFix + then localizes the root cause function based on the following criteria: 1) the function's execution time is larger than its normal execution time; 2) the function's start time coincides with the time when the performance alert is reported by the timeout bug detection tool such as TScope [15], and 3) the function's ending time is the termination time forced by TFix + . For example, in the MapReduce-5066 bug, the candidate root cause functions are Listener.run(), Responder.doRunLoop(), Reader.run(), and JobEndNotifier.httpNotification(). However, the first three functions are triggered by background processes, thus are not root cause functions. TFix + excludes those three functions because their execution time is not longer than that in normal runs and their start time is much earlier than the bug detection time. After pruning, TFix + identifies the JobEndNotifier.httpNotification() function as the root cause function.
Next, TFix + compares the line numbers of the inner-most common functions in different traces to identify the bugs as infinite loops or blocking function calls. For infinite loops where a block of code executes repeatedly, the line numbers of the candidate root cause function in each trace should be different, as shown in Figure 5. For blocking function calls, the line numbers of the candidate root cause function are always the same, as shown in Figure 6.

Adding Timeout Mechanism
TFix + adopts different strategies to add the timeout mechanisms for the system hangs on an infinite loop or a blocking  For missing timeout induced infinite loops, we add a timeout checking in the loop body to ensure the system jump out the loop after the pre-defined time elapses. Specifically, TFix + adds a configurable timeout variable and marks down the starting time point before the loop is executed. In every loop iteration, TFix + checks whether the elapsed time exceeds the configured timeout value using an if branch. If the checking result is yes, TFix + breaks out of the loop by throwing an exception. For example, the Yarn-1630 (v2.2.0) bug, shown in Figure 1, and described in Section 1.1 falls into this category.
For missing timeout induced blocking function calls, TFix + reuses existing timeout settings if applicable, or adds new timeout mechanisms surrounding the blocking function call. Existing timeout mechanisms can be provided by overloaded functions or existing methods inside the same class. Overloaded methods are easily to identify. For example, if the function is stuck on Socket.connect(SocketAddress endpoint), TFix + examines all the methods in Socket class and finds the overloaded function Socket.connect(SocketAddress endpoint, int timeout) with timeout settings. TFix + replaces the original one with the supplementary method to add a timeout setting. Besides the overloaded functions, TFix + examines all the methods of the invoked class and searches the keywords "set" and "timeout" in all the methods' names. If the method name contains both keywords, we consider it as a candidate timeout setting. TFix + calls the method to add the timeout setting to the function call. For example, for HDFS-3180 bug, the system hangs on URLConnection.connect() function. We search all the methods inside the URLConnection and find connection.setConnectTimeout() and connection.setReadTimeout() methods. TFix + leverages the two existing methods to add timeout mechanisms.
If there is no existing configured timeout mechanisms, TFix + puts the function in a callable thread and leverages ExecutorService and Future interfaces provided by standard Java library to monitor and manager the execution of callable threads. For example, Figure 7 shows the patch of HDFS-4176 bug. When the bug occurs, the system hangs on the getActiveNodeProxy().rollEditLog() function. TFix + does not find any pre-configured timeout mechanisms, so TFix + replaces the getActiveNodeProxy().rollEditLog() function with rollEditLogWithTimeout() function and puts the getActiveNodeProxy().rollEditLog() function into a callable thread. TFix + configures the timeout mechanism using future.get(timeout, TimeUnit.MILLISECONDS) method to terminate the callable thread when the elapsed time exceeds the value of timeout.

Prediction-driven Timeout Value Configuration and Patch Validation
After pinpointing the misused timeout variable or adding a configurable timeout variable, TFix + recommends a proper timeout variable value, to fix the timeout bug. During the normal run, a correct timeout value should be consistent with the expected execution time of the timeout affected function. This runtime execution time depends on the application workload (e.g., input file size) and the runtime available resources (e.g., CPU availability and network bandwidth). TFix + employs an online timeout value prediction scheme to adapt to dynamic runtime environments.
To achieve model-driven prediction, we first perform continuous collection of normal execution time of the identified function when the timeout bug is not triggered. We also collect the corresponding runtime system metrics and function input parameters when the function is invoked. We then employ polynomial regression schemes to establish the mapping from the application workload and runtime environment parameters S 1 , S 2 , ..., S k , ..., S N to function's execution time T . The polynomial fitting function is calculated by Equation 1, where β j is the coefficient for the combinatorial term S 1 p1 S 2 p2 ...S i pi ...S N pN . Each input parameter S i has the polynomial degree p i . The sum of the degree p i should be no larger than P which is the highest polynomial degree. This equation contains all the combinatorial terms constructed by all the input parameters when the polynomial degree is no greater than P .
Based on our experimental evaluation results, we set the highest polynomial degree P to 3 to both avoid overfitting and achieve fast online prediction. TFix + changes the P from 1 to 3 and selects the best fitting model using the least-square error.
In practice, any prediction model can hardly achieve perfect prediction. In order to achieve effective timeout bug fixing, TFix + includes built-in prediction error handling mechanisms. Particularly, under-estimation error (i.e., predicted timeout value is less than expected timeout value) of the prediction model has serious impact to our solution, which causes our timeout bug fix to fail. Thus, we introduce a padding to avoid the under-estimation error. We calculate the estimated timeout values T est produced by the regression model and measure the exact historical execution time T hist during the normal run. We calculate the relative fitting error, which is (T est − T hist ) over the exact execution time T hist , i.e., Test−T hist T hist . Suppose there are M measured data points, the padding ratio is calculated by Equation 2.
Our experimental results show that our padding scheme can avoid all under-estimations without imposing too much timeout delay. Finally, TFix + predicts the timeout value T predict using Equation 3.
The timeout value recommendation considers three different cases: 1) the timeout value is set too large, 2) the timeout value is set too small, and 3) timeout mechanism is missing. For the bugs caused by too large timeout values, the regression model is an interpolation model, since the predicted timeout value is within the range of the historical execution time of the timeout affected functions during normal runs. For the bugs caused by too small timeout values, the predicted timeout value T predict is beyond the range of the function's execution time T hist in normal runs. In this case, TFix + builds an extrapolation prediction model to suggest a timeout value. For the missing timeout cases, the regression model can be either interpolation or extrapolation model, which depends on the current workload and system environment.  Figure 8a shows the mapping model of the transferred image size and execution time when we fix the network traffic. Figure 8b shows the mapping model of the file size and network traffic when we fix the image size to 20 megabytes. Figure 8a shows an extrapolation model and Figure 8b shows an interpolation model, respectively.
Our online prediction scheme can adapt to different applications, because the predicted value is calculated from the normal run statistics on the specific applications. Our previous work [16] adopts a profiling-based heuristic method to provide timeout value recommendation, which might take a long time and high profiling cost to derive the right timeout value configuration.
After the patch is generated, we validate whether the patch is correct or not. A correct patch should fix the bug without introducing new bugs. To check whether the bug is fixed, we apply the patch to the program and re-run the bug triggering test cases. We trigger the bug under the same triggering condition and workload to observe whether the bug occurs again. To check whether our patch introduces new bugs, we run all the test suites provided by the release package (e.g., [2]) to check whether all of them are passed. Normally, a mature system is packaged with test cases to test all the functionalities. For example, Hadoop system is equipped with its JUnit tests to test each class's functions. We determine that TFix + 's patch is

EVALUATION
In this section, we present our evaluation details. We first introduce our evaluation methodology. We then present our empirical study results on 91 real world misused or missing timeout bugs. we then present the experimental evaluation results on 16 reproduced timeout bugs. Finally we present several case studies.
All the experiments are conducted in our research lab in a cluster of hosts which are equipped with quad-core Xeon 2.53Ghz CPUs and 16GB memories and running 64-bit Ubuntu v16.04. The system call trace is collected using LTTng v2.0.1. The function call trace is collected using Google's Dapper framework. We adopt existing static taint tracking framework, i.e., Checker [1], to perform misused timeout variable identification. We use jstack to dump the stack traces for all the Java processes. TFix + is implemented on top of Soot compiler [5] to patch binary code. We use Python's scikit-learn [4] package to build the polynomial regression model to predict timeout values as described in Equation 1 to 3.

Evaluation Methodology
We first conduct an empirical study on 91 real world misused or missing timeout bugs which are collected by our previous study [11] to manually evaluate the fixing strategies of TFix + . We check each bug to determine whether TFix + can fix the bug without introducing new problems. We should note that, our benchmarks do not contain concurrency bugs, e.g., race conditions and deadlocks. Even though TFix + can fix some of them by adding timeout mechanisms, in most cases, fixing concurrency bugs requires specific techniques, e.g., order checking [22], atomicity checking [21] and etc.
To the best of our effort, we have reproduced 16 timeout bugs from six commonly-used open-source systems, listed in Table 1. We set up four systems in the distributed mode to investigate timeout issues occurring on the communications among different nodes. The 16 reproduced bugs are listed in Table 2, including 11 misused and 5 missing timeout bugs. They cover timeout problems in both network communications and intra-node coordinations, representing common realworld missing and misused timeout bugs.
Workload: To replay real-word scenarios, each system runs a specific workload, listed in Table 2. Specifically, for the Hadoop, HDFS, MapReduce and Yarn systems, we run word count job on a 765MB text file. For the HBase system, we use the YCSB workload generator to make insertion, query and update operations on a table. For the Flume system, we write log events to the log collection tool and distribute the   logs repeatedly. These workloads invoke the timeout affected functions all of our tested timeout bugs. Expected timeout value measurement: In order to validate the prediction accuracy, we measure the exact expected timeout value during buggy run. We run the same workload under same runtime computing environment without triggering the bug and retrieve the timeout affected function's execution time in the function call trace, which is the exact expected timeout value. Figure 9 shows the fixing coverage of TFix + on 91 studied timeout bugs. TFix + fixes 72 bugs (i.e., 79%) completely. TFix + fixes 12 (i.e., 13%) bugs partially because their fixes require not only proper timeout mechanisms but also retry schemes and program semantic repairs. For the rest seven (i.e., 8%) bugs which cannot be fixed by TFix + , they are all caused by improper hard-coded timeout values.

Empirical Study Results
The seven misused timeout bugs cannot be fixed by TFix + are caused by hard-coded timeout values, i.e., no configurable timeout variables. Those timeout bugs often occur in early versions of a system, such as Hadoop 0.x version and HBase 0.x version. TFix + cannot identify the misconfigured timeout variables in our taint analysis since the timeout affected functions do not use any configurable parameter or configuration constant. Although TFix + cannot localize misused timeout variables, TFix + identifies the bug as a misused timeout bug and pinpoint the timeout affected function. The hard-coded timeout value should be located in the timeout affected function or its callee functions. Developers can perform a cross validation to check whether any hard-coded value equals to the timeout affected function's execution time. Then developers can expose a configurable timeout variable for users.

Experimental Results
In this subsection, we present the experimental results on 16 reproduced timeout bugs. we first present the results of fixing misused timeout bugs and fixing missing timeout bugs, and then we describe the results of timeout value prediction and patch validation. Finally, we present the negative case study to show why TFix + cannot fix one timeout bug completely. Table 3 shows the results of adding timeout mechanisms on the 5 missing timeout bugs. TFix + fixes the HDFS-3180 bug by first identifying that it hangs on a blocking function call and then adding timeout mechanisms using existing timeoutsetting functions, i.e., connection.setConnectTimeout() and connection.setReadTimeout(). For the HDFS-4176 and MapReduce-5066 bugs, there are no existing timeout mechanisms that can be set on corresponding blocking function calls. TFix + fixes them by isolating the blocking functions in callable threads with timeout settings on the future.get() method. TFix + fixes the Yarn-1630 (v2.2.0) and Flume-1819 bugs by first identifying that they hang on infinite loops and then adding timeout checking on the loop body. Specifically, TFix + checks whether the elapsed time exceeds the pre-defined timeout value in each loop iteration. Table 4 shows the bug fixing results by TFix + . We list the expected timeout value under current workload for each bug. After adopting TFix + 's patch with the predicted timeout value in the system, we find that 15 out of 16 bugs do not occur anymore under the same workload with all the test suites passed successfully.

Prediction-driven Timeout Value Configuration and Patch Validation Results
We list the timeout values in the bugs' patch files in Table 4. We observe that the timeout values in the manual patches are not always correct. 11 out of 16 patches of the bugs need user inputs to set manual thresholds. It means that when patching misused timeout bugs, developers usually make the timeout variable configurable for users and set a default value which requires the user to set the timeout value based on his or her own application workloads. However, it is challenging to make the correct configurations for different production workloads and computing environments, even for experienced engineers. For example, in the patch of Yarn-1630 (v2.3.0) bug, the default value of the yarn.client.application-client-protocol .poll-timeout-ms variable is configured to be -1 milliseconds, which equals infinity. Developers expose the variable for users to configure. If users do not properly configure the timeout variable, the timeout bug still happens in the patched version.
We also observed that TFix + 's fixing schemes sometimes are different from the manual patches. We use HDFS-4301 bug as an example. In the patch of HDFS-4301 bug, the default value of dfs.image.transfer.timeout is still set to 60 seconds, which is identical with the timeout value before patching. However, the patch partitions the fsimage file into multiple chunks of a equal chunk size. The timeout variable limits the maximum transferring time for each chunk. In contrast, TFix + changes the timeout value to 138.68 seconds, that successfully fix the problem.
We should note that, the predicted timeout value by TFix + might be different under different workloads. This is our design choice, because a fixed timeout setting cannot handle unexpected workload changes or environment fluctuations. For example, in HBase-15645 bug, the misused timeout variable hbase.client.operation.timeout defines the time to block a certain table to prevent concurrency issues. Since the table size is small for YCSB workload in our evaluation, the predicted value by TFix + is only 3.47 seconds. If we use a large timeout value under the same YCSB workload, the user will still experience a noticeable delay in the system. On the other hand, if we increase the table size, the predicted value will increase correspondingly.
Although TFix + introduces timeout value over-estimations to avoid too small timeout bugs, the differences between the predicted values and the expected values are no more than 20 seconds for all the tested bugs.
We also list the diagnosis time for each bug in Table 4, which is the total execution time for all the fixing components of TFix + . The diagnosis time is within ten seconds, which makes it practical to apply TFix + in real-world systems. The runtime overhead comes from three tracing modules, i.e., system call tracing, function call tracing and system metric tracing. As discussed in our previous work [16], function call tracing incurs low overhead because TFix + only traces a small number of functions (functions related to timeout configuration, network connection and synchronization). Our prototype implementation experiments show that TFix + incurs less than 3% runtime overhead to the monitored system.

Negative Case Study
As shown in Table 4, Flume-1819 bug cannot be fixed by TFix + since not all the test suites are passed after adopting TFix + 's patch. Flume-1819 bug is triggered when the system hangs on an infinite loop to read the data into channels. To fix the bug, a timeout mechanism is needed to flush the cache data into disk when the bug is triggered. Otherwise, the system experiences data loss. TFix + can partially fix the bug by enabling the system to jump out of the infinite loop. However, in order to fix the bug completely, we need to trigger cache data flushing, which requires application specific knowledge. In addition, TFix + provides a proper value for the timeout mechanism, which complements the manual patch. As shown in Table 4, the default value of the manual patch is not applicable under current workload. The expected timeout value (i.e., 30 seconds) is significantly larger than the one (i.e., 3 seconds) in the manual patch.

Case Study
In our previous study paper [11], we further divide the misused and missing timeout bugs into multiple subcategories based on the root causes, e.g., incorrectly reused timeout values, ignored timeout values, missing timeout for network communication and missing timeout for synchronization. We list several cases to show how TFix + fixes bugs belonging to different subcategories. HBase-16556 (misused): this bug is caused by reusing rpcTimeout in get(), delete() and existsAll() functions. When this bug occurs, get() function invokes batch() function and invokes submitAll() function. submitAll() leverages the rpcTimeout variable, whose value causes severe performance degradation to the system. TFix + can identify submitAll() function as the timeout affected function and further determine the misused variable rpcTimeout. TFix + then adopts the prediction scheme to suggest a proper value for rpcTimeout. The manual patch leverages the timeout mechanism of batch() function to introduce a new variable with a recommended value, which has the same effect of TFix + 's bug fix.
HBase-13104 (misused): this bug is caused by ignoring a pre-configured timeout variable to adjust the session timeout for the standalone HBase system. Instead, HBase system only uses the configuration constant ZK_SESSION_TIMEOUT as the session timeout. The constant is defined in the configuration file HConstants.java and TFix + can localize it by performing taint analysis. TFix + then predicts a proper session timeout value for the constant ZK_SESSION_TIMEOUT instead of introducing another variable.
Hadoop-4659 (missing): this bug is caused by missing IPC timeout for network communication. TFix + can identify an infinite loop inside the waitForProxy() function. TFix + then adds a timeout mechanism to jump out of the loop, which is the same as the manual patch.
Phoenix-2496 (missing): this bug is caused by missing timeout for synchronization. The bug happens when a query thread on QueryServer holds a read lock to connect with HBase and another shutdown hook thread is acquiring the lock. HBase system cannot respond to QueryServer, due to heavy workload or unexpected events. Then the query thread does not release the lock and the shutdown hook thread hangs. TFix + identifies the systems hangs on closeInstance() inside the addShutdownHook thread. Since no pre-configured timeout mechanism is found, TFix + then leverages future.get() to add a timeout mechanism on closeInstance() function. The manual patch is the same with TFix + 's patch.

Negative Case Study
We describe two cases to show why TFix + fix the bugs partially or TFix + cannot fix the bugs.
HBase-3456 (misused): this bug is caused by ignoring the configurable timeout variable ipc.socket.timeout. HBase client uses a hard-coded 20 seconds as the socket timeout in NetUtils.connect() function. TFix + can identify setupIOstreams() inside the HBaseClient class as the timeout affected function. However, since no pre-configured timeout variable is used, TFix + cannot identify the root cause since the 20-second socket timeout is hard-coded.
MapReduce-4813 (missing): this bug is caused by missing timeout for synchronization. The bug occurs when the Application Manager (AM) calls commitJob method synchronously during JobImpl state transitions. The JobImpl write lock is held by the commitJob thread. If committing the job takes too long, the state transition hangs on acquiring the lock. TFix + can partially fix the bug by introducing a timeout mechanism to terminate the hanging job commit thread. However, TFix + cannot reset the JobImpl state. In comparison, the manual patch adds the COMMIT_COMPLETED and COMMIT_FAILED states and enables the job to transit to a proper state after the commit succeeds or fails.

LIMITATION
TFix + fixes both misused timeout bugs and missing timeout bugs which cover 78% timeout bugs in total. When fixing misused timeout bugs, TFix + can localize the misused timeout variable if the cloud system uses timeout variables in timeout handling operations. However, we observe that some timeout bugs are caused by hard-coded timeout values. The hard-coded cases are discussed in Section 3.2. Although TFix + cannot localize misused timeout variables, TFix + can correctly identify them as the misused timeout bugs and pinpoint the timeout affected functions. TFix + provides useful hints to localize the hard-coded values.
TFix + currently cannot cover the other three categories of timeout bugs, i.e., improper handling, unnecessary timeout, and clock drifting. For the bugs caused by improper handling, we need to add or revise application-specific functions to fix them. For the bugs caused by unnecessary timeout, TFix + can identify the timeout affected functions when the bugs cause performance issues. However, whether removing the unnecessary timeout alters the control flow and brings any unwanted effects is out of the scope of this paper. For the bugs caused by clock drifting, the fixing strategy is to enhance the synchronization of distributed systems, rather than to modify any timeout mechanisms.
TFix + currently only supports cloud systems written in Java since the implemation of fixing misused and missing timeout bugs only works on Java platforms now. However, our approach is agnostic to programming languages. TFix + can be easily extended to support other programming languages by replacing Java specific Dapper, static taint analysis, timeout mechanism instrumentation components with other programming language counterparts.

RELATED WORK
Automatic bug fixing. Work has also been done to automatically fix different bugs. For example, AFix [21] and CFix [22] proposed automatic patching strategies for concurrency bugs. ClearView [26] identified violated invariants from erroneous executions and generated candidate repair patches to change the invariants. Tian et al. [29] presented an automatic bug fixing patch identification tool to maintain older stable versions. Tufano et al. [30] applied an Encoder-Decoder model based on neural networks to mine the existing patches and automatically generate new ones. Narya [23] predicted a host failure by adopting domain knowledge and machine learning models and took appropriate actions to mitigate failures. In comparison, our work fixes timeout bugs with the drill-down bug analysis approach that can not only identify bug root causes but also apply proper timeout fixing strategies.
Our previous work HangFix [17] focused on fixing hang bugs caused by infinite loops and blocking function calls. HangFix proposed to resolve some hang problems by inserting a timeout mechanism with a pre-set timeout variable. In contrast, TFix + focuses on fixing both misued and missing timeout bugs which could cause performance slowdown, job failures, or software hang. In HangFix, the newly introduced timeout variable is heuristically configured, thus cannot handle dynamic workloads. TFix + provides the prediction-driven timeout variable configuration scheme which can adapt to user runtime environment and dynamic workloads. TFix [16] is the early version of TFix + , which focuses on only fixing misused timeout bugs covering 47% timeout bugs. In contrast, TFix + fixes both misused and missing timeout bugs that cover 78% timeout bugs. We have conducted extensive empirical study on the fixing coverage of TFix + . Moreover, TFix + provides a new prediction-driven approach to achieving self-configuring timeout bug fixing.
Tracing-based bug detection and diagnosis. Previous work has extensively used tracing techniques to detect and diagnose bugs. For example, X-ray [6] dynamically instrumented binaries to trace the inputs and outputs of different components and inferred the traces to diagnose performance bugs. Chopstix [9] collected low-level OS events such as CPU utilization and I/O operations online and reconstructed these events offline for troubleshooting intermittent and non-reproducible bugs. REPT [10] utilized hardware traces to reconstruct the program's execution and employed record-and-replay techniques for debugging. Magpie [8] instrumented middleware and packet transfer points to record fine-grained system events and correlated these events to capture the control-flow and resource consumption of each request for debugging. Crash-Tuner [24] detected crash-recovery bugs by injecting faults in the test suites and exposing the bugs via log-based program analysis. Gong et al. [13] conducted a study on applying machine learning techniques to detecting malwares from a major Android App market. Different from TFix + , these tracing techniques do not apply for timeout bug diagnosis or fixing.
Tracing techniques have also been used by previous work to detect timeout bugs specifically. For example, TScope [15] detected timeout bugs using timeout related feature selection and machine-learning based anomaly detection on system call traces. SafeTimer [25] checked whether a timeout bug is caused by network delay or packet delay at the OS level. Those tracing-based timeout detection techniques are complementary to and can be used with TFix + as an automated system for timeout detection, diagnosis and fixing. In our experiment, we use TScope as the front-end to identify whether an anomaly is caused by timeout.

Configuration bug detection and diagnosis.
Different techniques have been used to detect and diagnose misconfiguration issues. Violet [18] adopted symbolic execution to detect specious configurations. Ctests [28] generated new test cases based on original test suites to detect errors caused by configuration changes. SPEX [32] studied configuration constraints and exposed potential configuration errors by injecting errors that violate the constraints. ConfValley [20] introduced a new language to define system validation rules and checked configurations against those rules before the application was deployed in production. PCheck [31] analyzed the application source code and automatically emulated the late execution that used configuration values to detect latent configuration errors. CODE [34] detected configuration bugs by identifying the abnormal program executions using invariant configuration access rules. ConfAid [7] adopted dynamic taint tracking methods to instrument the binary application and analyzed the information flow to pinpoint the root causes of configuration errors. ConfDiagnoser [36] extracted the control flow of configuration options, instrumented the application code for profiling and analyzed the configuration deviation to detect the erroneous configuration options. EnCore [35] applied machine learning techniques to model the correlation between the configuration settings and the executing environment and correlations between configuration entries, in order to learn and detect configuration bugs. In summary, existing approaches focused on detecting misconfiguration issues that caused functional bugs. They cannot be readily applied to detecting performance issues caused by misconfigured variables that are triggered during system runtime due to input data or environment changes. Moreover, existing tools cannot recommend applicable configurations to fix the bugs. In comparison, TFix + adopts the prediction-driven scheme to suggest proper timeout variable configurations.

CONCLUSION
In this paper, we have presented TFix + , a self-configuring hybrid timeout bug fixing system. TFix + proposes two hybrid schemes to fix misused and missing timeout bugs, respectively. TFix + adopts a prediction-driven timeout configuration recommendation scheme based on runtime tracing and static taint analysis to provide end-to-end bug fixing. Our empirical study shows TFix + can fix 79% of 91 real world misused or missing timeout bugs. The experimental results on 16 reproduced timeout bugs show that TFix + can produce effective fixes for 15 out of 16 tested timeout bugs.