{'fraction_non_alphanumeric': 0.019192796777505725, 'fraction_numerical': 0.0067135297369876, 'mean_word_length': 4.954150011756408, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 0, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 4, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Software Test Automation with Robot Framework

Automated testing allows for releases to be quickly verified and regression tested multiple times over a product’s development period, providing a level of quality assurance unmatched by a purely manual process. This article discusses the driving objectives, implementation strategy, and general overview of a test automation framework created using the open source project Robot Framework as its test automation core. The framework described in this article has been used to provide test automation capabilities to both Graphical User Interface (GUI) based Windows applications and embedded systems. In addition to an overview of the developed framework, this article also provides the results of automated testing for multiple releases and a discussion of selected lessons learned from its creation and maintenance. General Terms Automated Testing, Software Testing, Test Automation, Test Engineering.


INTRODUCTION
Automated testing refers to the use of software and/or machinery to perform evaluation of a produced item without a human performing the actions necessary to stimulate the item [1]. Generally, automated testing is preplanned by a human using a scripting language or framework to set up scenarios with which to exercise the functionality covered by the requirements [2]. The tester uses their knowledge of the system to define inputs and expected outputs to be covered by the automation code [3]. The automation framework executes the test scenarios and provides detailed feedback as to the results of the testing [4]. It takes the procedures that would be performed by a human via manual means and executes them automatically through the use of specific equipment or applications, requiring the creation of test scripts to take the place of manual test procedures [5].
The incorporation of automated testing on a project can produce several benefits. Test documentation organization can improve due to an increase in maintainability of the test procedures and better management of test procedures [6]. Quality of the system under test may improve due to increased confidence of the product, reduction of human errors during test execution, and the ability to perform much more test executions than could be possible with a manual-only approach [7]. Finally, it provides opportunities by allowing new types of testing that were impossible to perform manually and it allows for more dedicated exhaustive testing of the most important components [8].
With the rise of iterative development methodologies coming out of the ideas presented by Rapid Application Development in the early 1990s and the Agile Manifesto of the early 2000s, automated testing has continually been a key component of rapid product development [9]. Essential to support an incremental development process, it allows for releases to be quickly verified and regression tested multiple times over the product's development period and can provide a level of quality assurance unmatched by a purely manual process [10].
This article discusses the driving objectives, implementation strategy, and general overview of a test automation framework created using the open source project Robot Framework as its test automation core. The framework described in this article has been used to provide test automation capabilities to both GUI-based Windows applications and embedded systems.

PATH TO TEST AUTOMATION
The decision to implement automated testing on a project can be multi-faceted, requiring a number of aspects to be considered. It may not always be the best testing approach and depends on a variety of factors such as project scope, development timelines, required test coverage, and cost [11].
For this project specifically, the main driver was the need to reduce test execution times significantly in order to support the transition to an Agile development process and to shorten the overall release schedule for the system. The system under test, an embedded airborne Electronic Warfare Management System (EWMS), could take up to 18 weeks to perform a full regression test manually. Its supporting Windows-based simulation, monitoring, and file creation tool suite could take up to an additional 20 weeks of manual testing for a full regression test. As updates to this system were usually released on a 6 to 12-month timeline, these regression test durations were prohibitive to the continued development of features and quick deployment to the field.
The path to test automation for this project was an arduous one. For the first several years, there was little support for its pursuit, due to the size of the system and the existence of a high-quality manual test suite. While the typical recurring budgets for the project were relatively high, the costs to start building automated testing infrastructure were still too high to justify. Taking time to officially build in this capability would prevent the development of necessary features that had to be released quickly.
Because of this, test automation was pursued as a side project targeting one of the system's file creation tools. This tool was a data entry application with a large number of configurable settings for the main system, resulting in a text-based file that would be loaded onto the embedded system for a specific scenario. The large amount of options in the tool resulted in its manual test procedures numbering approximately 2,200 pages, requiring six weeks of testing time to execute. This system was targeted first for automation because of the simplicity of its environment; it was a single software application with no additional software or hardware interfaces. It could be executed in a virtual environment, making easy use of parallel testing and continuous integration tools [12].
The development of test automation capabilities for this project started initially as an experiment in 2013 with a team primarily composed of engineering student workers. This helped keep initial costs low while providing a worthwhile student project that created many opportunities for learning. This initial version, while somewhat successful, was not developed to be sustainable, leading to a largely unorganized codebase that was scrapped and rebuilt multiple times along the way. Over the next three years, the effort suffered a number of self-imposed issues from less than ideal coding practices and implementation strategies. Regardless, the automation project produced several significant test execution time savings and provided evidence for the value of a test automation capability.
Due to this evidence, the author secured internal funding in 2016 to start development of an official test automation framework. Using the knowledge gained through the various false starts over the previous several years, an initial design was developed that centered around the use of Robot Framework, along with an initial set of guiding objectives and an implementation strategy for the targeted system. This prework was necessary to build a proper foundation that would guide the creation of a sustainable automated testing capability.

Objectives
The initial set of objectives for the test automation framework is provided below.

Strategy
The strategy employed when implementing test automation was to utilize as much existing functionality and components as possible in order to: 1-Minimize costs.
2-Limit the impact to the system under test.
3-Maintain the ability to execute existing manual testing. 4-Provide an easier transition to automated testing.

5-Provide automated testing capabilities faster.
These goals provided for the establishment of an automated test framework that supported the system under test without the need to change its development path or schedule. The approach focused on providing automation capabilities through the existing suite of support tools that were used to interact with the system. These tools were used manually through a GUI by the test team to provide simulation and monitoring of the system's execution environment. The initial approach was to create automation libraries that could control and monitor these tools via their respective GUIs, which would mimic the exact actions of a test engineer performing a manual test.
While this approach succeeded at providing test automation capabilities while adhering to the goals provided above, it was not the most impactful approach. However, other initial approaches would have been too costly or could have affected the development of the system. These observed inefficiencies combined with the difficulties of implementing test automation on a project as complex and established as the system under test necessitated a long-term automation strategy that would eventually result in an optimized test automation solution.
This led to the concept of a phased approach to test automation, which is described below. Phase 4: Develop lower-level test harnesses that can talk directly to software components within the system, providing the ability to perform testing on each component in isolation from the rest of the system. Develop automation capabilities to stimulate and monitor lower-level components. Create headless test cases that target component-level functionality.
Phase 5: Introduce high-level commands that provide more capability to the tester (e.g., move threat 1 to location 5 within 20 minutes using route 2, maintain aircraft altitude and fly in a circle with radius of 10 miles), allowing more natural and realistic test scenarios in a lab environment.
It should be noted that these phases are not necessarily completed sequentially; some could be worked concurrently, while some could be skipped altogether. For example, currently for this system, a mixture of Phase 0 through Phase 4 has been implemented, depending on the component. This specific system may never need test automation capability at Phase 5, but it has been discussed as a future option.

FRAMEWORK OVERVIEW
The following section will provide an overview of the test automation framework that was created. First, a discussion of Robot Framework is necessary to understand the core automation capability. Next, the basic architecture of the framework is described. Finally, an example test case is provided with commentary to show a test case from the end user's perspective.

Robot Framework
Robot Framework is a Python-based automation project that provides the infrastructure for developing a test automation capability, including the translation of code to user-defined actions, automatic test execution, and results reporting [13]. It allows automated tests to be written in simple English to support their easy creation and understanding. The basic idea of Robot Framework is to define libraries that include functions for performing necessary actions required by the test case and then call them from the high-level test case using a simple English keyword or phrase.

Example 1
To illustrate how Robot Framework works, a simple test case named "notepad_save_file.robot" is defined in Figure 1 below.

Fig 1: Simple Robot Framework Example
The test case above verifies Notepad's ability to save a file with a specific line of text. Items in blue are referred to as keywords, while items in red are referred to as keyword inputs. The code to execute the keywords is stored in the library "notepad.py," which is called at the beginning of the test case. As can be seen above, the power of Robot Framework comes with being able to define the functionality of a keyword once and use it multiple times within a test case [14]. This is even more apparent with keywords such as "Add Text" in the above example, as the keyword input can be any string, depending on the restrictions of the implemented function.
To execute the test case, a tester would only need to run the command "robot notepad_save_file.robot" from a command prompt, assuming Robot Framework is installed and the user has navigated to the correct directory. Robot Framework would the output the results of the test case within the same directory in a variety of formats for later review.

Example 2
Keyword text can also be expanded and modified to give a more natural feel to the automation language within test cases. For example, consider the following keyword in Figure 2.

Fig 2: Keyword Usage
The entire line in Figure 2 is considered a keyword with three keyword inputs. For ease of comprehension, keyword inputs are now required to be surrounded by square brackets. This is customizable in the definition of the keyword, which can be seen below in Figure 3.

Fig 3: Keyword Definition
The keyword refers to the "Select Combobox Item" method defined elsewhere, which takes three inputs. Robot Framework is able to interface with almost any language or automation tool to provide more powerful automation capabilities. In this case, the Windows UI automation language AutoIt, translated to a Python module PyAutoIt, was used to provide UI interaction capabilities. The code to perform this action of selecting an option within a combobox can be seen in Figure 4 below.

Fig 4: Keyword Function
The method above accepts the parameters passed from the keyword usage through the keyword definition to the final method. It uses the passed values to identify the actual names associated with those values as stored in the code by looking them up in a JavaScript Object Notation (JSON) file that associates aliased labels (i.e., the word/phrase that the test engineer wants the specific item to be known by) with the code-based name (i.e., the name that the developer used for the specific object). An example of a JSON table to perform this aliasing can be seen in Figure 5 below.

Fig 5: JSON Combobox Label Association
The main purpose of the JSON file is to: 1-Maintain a list of supported objects outside of the code to interact with them.
2-Allow aliasing so that the final test procedures are not reliant on standardized control names to make sense to an external representative.
3-Define an object type to use in the event of a collision of preferred name (e.g., a textbox and combobox that use the same label property).

Basic Architecture
A basic architectural view of the test automation framework can be seen in Figure 6 below.

Fig 6: Basic Architecture
Starting at the bottom of the diagram, to create a Test Automation Capability targeting a specific application or system, Robot Libraries are written to interface with the system under test. These libraries contain functions that define methods for interacting with the system. In the Robot Framework Keywords file, the methods from the Robot Libraries are mapped to commands that will be visible to the test engineer to use for building test cases in the Robot Test Case file. This essentially means that an automated test case (a .robot file) is a sequence of pre-defined keywords written in natural language that reads like a set of manual test steps. This allows the test case to be executed manually for debugging purposes or if for some reason the automated capability breaks, and it allows the automated test case to be easily translated to a more presentable form for delivery to a customer.
The test automation framework is fundamentally a collection of custom developed automation libraries that control interaction with the system, keyword definitions that provide the standardized interaction language used by test engineers, and Robot Framework that provides the management of those connected pieces. Additionally, several usability features such as syntax highlighting, automatic keyword completion have been incorporated to make the creation of test cases faster and more user-friendly. Figure 7 is an example automated test case, shown as a screenshot from Visual Studio Code using the custom highlighting scheme developed for the test automation framework. Specific sections of the test case are described here to provide an overview of the organization and flow of a typical test case.

Settings Section (Lines 1-5)
The Settings section is used to declare keyword libraries that are expected to be used in this test case. The test automation framework architecture encapsulates keyword libraries by function or component for easier management. In this section, the test engineer imports the libraries that include keywords they need to perform the actions of the test case.

Variables Section (Lines 7-10)
The Variables section provides the ability to declare groups to be used later in the test case. Currently, the only use of this is to group requirements traced to this test case and aircraft platforms that are applicable for this test case. These labels are used to make it easier to identify these items quickly.

Test Cases Section (Lines 12-51)
The Test Cases section contains the body of the test procedure that will be executed. Subsections are further discussed below.

Title (Line 13)
The title of the test case is declared after the Test Cases section header. In this example, a global test case number, its specific name, and the platform this test case uses (Plat_1) during the execution of the test case are listed.

Tags (Line 14)
Tags are a feature of Robot Framework that allow test cases to be queried/executed based on custom phrases that are included in the test case. For example, a test engineer could target the entire test suite and command all test cases that include the tag "Requirement-01" to be executed. Robot Framework is able to identify which test cases those are and execute them.
In this specific example, the variables previously defined are also tagged so that a query can be run based on requirements, functionality, or platform applicability.

Setup (Line 24)
This line is a custom keyword that sets up a test case for the system under test after being provided a configuration. There are many actions rolled into this singular keyword that prepare the system under test for user input, such as loading the system hardware, verifying that configuration loaded correctly, and clearing the catalog of errors. Configurations are defined by test engineers in a separate file that the test automation framework can read. This keyword also prints the details of the specific configuration used for this execution of the test case for future reference. 16-17, 20-22, 26-28, 36-37, 43-44, 50) These lines are typical actions that would be defined in a manual test case as steps that a user should perform.

Requirements Declarations (Lines 30, 34, 39, 41, 46, 48)
While requirements are traced to the test case using the Tags section, more specificity is often required by the customer or project. The test automation framework provides a method for defining the specific steps that satisfy requirements through defining where the verification of the requirement begins and ends.

RESULTS
The test automation framework described herein has been used successfully during the formal test execution of the targeted system for two releases, as well as two releases of the data entry support tool described previously. The results of the automation effort for these releases are summarized in Table 1 below and further described in the following subsections.

Data Entry Tool Release 1
The data entry tool was the first application targeted by the test automation framework, due to its limited interactions with other components. During the first release, 95% of the manual test procedures were automated, equating to more than 2,200 pages and six weeks of execution time. The entire suite of manual and automated test cases for the final release was able to be executed in two days. Typically, the manual test procedures were run twice during a release effort, once for a dry run and once for a formal test. Factoring that time in, the test automation framework saved approximately $60,000 of customer funds due to test execution costs. This initial use of the test automation framework significantly reduced the overall cost of developing the tool.

Data Entry Tool Release 2
The second release of the data entry tool was similar to the first, in that 95% of the test procedures remained automated even with additional functionality being added. With the additions, running the testing manually would have taken approximately seven weeks, but the entire test was still able to be accomplished within two days. The approximate cost savings for this effort equated to roughly $70,000, increasing the value of the test automation framework on this effort.

System Release 1
As the embedded system is significantly more complicated than the data entry tool, the transition to automated testing is still ongoing, even after two years of a functional test automation solution. This is partly because feature development of the system was not stopped to implement this transition and partly due to the reworking of test cases to better support automation. Due to the phased approach limiting impact to the system under test, the transition was less impactful than it could have been or will be in the future.

Qualitative Results
Several qualitative results were discovered during this process. A selection of key discussion topics is provided below.

Team Morale
The field of software engineering can be subject to a high level of burnout, due to a variety of factors [15]. For a test engineer, running the same set of regression tests manually over and over again can become tedious, especially if that effort takes significant time in comparison to other, more creative or skill-intensive tasks. Test automation can reduce this burnout, as it removes some of the monotonous aspects of the job of a test engineer. In place of the time usually spent on regression testing, it may provide the potential for more exploratory testing to be performed, which has been found to identify more defects that scripted testing [16].

Test Quality
The quality of the test procedures increased, due to improved flow and organization, increased maintainability, and reduced variability between approaches. When the actions that can be performed are controlled through the use of automated keywords, the test cases become more standardized, leading to a greater level of understanding. As with the development of software, different test engineers may take different approaches while still solving the problem, but the variations of how this can be accomplished are easier to comprehend.

CONCLUSION
The creation of an automated testing framework can be a valuable, but also costly, venture that should be carefully considered before pursuing. It can provide considerable reductions in test execution times, but the benefits must be weighed against the costs involved. The implementation of test automation described in this article provided significant time and cost savings, allowing more efficient use of customer funds.

Selected Lessons Learned
Several lessons were learned throughout the creation of this test automation framework. A selection of key topics is described below.

Treat Test Automation as a Product
Even if it is never intended to be delivered to an external customer, the developed automation code and the accompanying test procedures that use it should be treated like a real, releasable product. It should be designed and documented well, conforming to standard practices and procedures. This leads to a higher level of maintainability, which is a common issue with the longevity of test automation infrastructure [17]. Treating the test procedures as code also allows for the use of development tools, which can be an exceptional aid to productivity.

Start Small and Advertise Results
Management, customers, and other stakeholders that control project funding can be hard to convince that test automation is worth the effort, especially if a set of manual procedures already exist that work as expected. If it truly makes sense for the project and would provide value, find a way to incorporate automation somewhere along the process in a small way as an example of future potential [18]. Use that as evidence for the value that can be created by transitioning to an automated world.

Automate Functionality That Makes Sense
Not everything should be automated; sometimes test cases are too difficult or just too small to make sense [19]. Sometimes an automated approach does not always result in the most efficient method of testing [20]. Discretion and experience should be used to identify when functionality should be tested automatically.

Standardize the Approach
Put significant effort into the automation language being created. Standardizing the approach to the keywords that will be visible to the end users is an important part of providing value. If a test engineer trying to use the test automation framework has trouble identifying what functionality is available, it can impact productivity and result in confusion.

Future Expectations
Based on the last two years of results so far and by implementing future automation phases, it is expected that within two to four more release cycles, the execution of the test procedures for the system will be reduced to less than one week. This type of reduction, from an original timeline of 18 weeks, shows the potential improvement that developing an automated testing capability can provide.