{'fraction_non_alphanumeric': 0.024722321748477247, 'fraction_numerical': 0.006091006807595844, 'mean_word_length': 5.392254840724547, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 0, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 1, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Augmented Reality in Maintenance Training for Military Equipment

After more than 20 years of development, augmented reality technology has shown a strong development momentum, successfully applied in civil and military fields. And augmented reality maintenance system is one of the most typical application. This paper introduces the present situation of augmented reality maintenance system, illustrates the structure of a typical system, discuss key technologies of augmented reality maintenance, and investigates the insufficiencies of it in military equipment maintenance according to the contemporary condition. Eventually, this paper indicates the potential development tendency of the augmented reality military equipment maintenance system in the future.


Introduction
P. Milgram and F. Kishino first explicitly introduced the theory of augmented reality (AR) [1], that is, to combine virtual elements into real scenes, mixing real scenes with virtual scenes, and the information in the two views superimposed and strengthened each other. Eventually, it enhances people's awareness of the actual world and the capacity to process information. Compared with virtual reality technology, AR has a more natural way of interaction, a more real sense of immersion, and lower hardware requirements. Currently, it has been widely used in education [2], medical [3], industry [4], entertainment [5] and other fields. Many universities and enterprises (for example, Massachusetts Institute of Technology, Columbia University, University of Toronto, Boeing, etc.) have made in-depth research on AR maintenance training system and its key technologies, which have been successfully applied in the maintenance of complex equipment and accomplished excellent outcomes. The successful application of AR technology in the civilian field has attracted the attention of the military. The US army, the German army and the Italian army, etc. have all carried out research on AR military equipment maintenance system. The system captures the equipment in need of repair through the camera, transmits the information to the computer for processing, and forms the maintenance guidance plan, which is transmitted to the AR device in the form of video, animation, picture, text, etc., to guide the operators to maintenance. This maintenance method provides a safer operating environment, reduces the loss of equipment, shortens the training time of support personnel, and saves a lot of manpower and material resources while advancing the combat effectiveness of the army. In this paper, the application examples of AR maintenance and maintenance training in recent years are sorted out, and the key technologies are expounded in combination with typical cases. Finally, the  2 insufficiencies in the existing AR equipment maintenance are analyzed in depth, and the future development trend of AR military equipment maintenance system is given.

Related Work
After more than 20 years of development, AR has been widely applied in the military, such as command and control, situational awareness [6] and logistics support [7]. But the most representative is the application of AR in equipment maintenance.

Applications of Maintenance and Assembly
A large number of military equipment are technically intensive complex electromechanical systems, such as aircraft and spacecraft, and their maintenance and assembly require extremely high requirements of equipment support personnel. The application of AR technology reduces the risk of military equipment maintenance and improves the reliability and efficiency of maintenance and repair. In the late 1990s, the ARVIKA project funded by Germany was one of the largest AR projects in the field of maintenance and assembly [8,9], which focused on the development, production and service of complex technical products (Fig. 1). The European Aeronautic Defense and Space Company has used the ARVIKA to solve the problem of cabling fighter jets, allowing operators to easily do high-density cabling on 1m × 6m work plates with voice prompts. The project of ARTESAS is a follow-up to the ARVIKA project and is committed to the growth of AR applications for automotive and aerospace maintenance. Besides, it strengthens the application of markerless tracking registration technology, which diminishes the adverse effects of complicated surfaces and light conditions in maintenance tasks.  [8]. In 2000, the EU-funded STARMATE project [10] started to be implemented. It aims to develop a prototype system that allows computer-guided maintenance of complex mechanical equipment using AR. It mainly provides two major functions: assisting users to realize assembly, disassembly and maintenance procedures and training of relevant workers. The system seamlessly blends digital technical manuals, construction documents, and maintenance procedures into the workplace, decreasing the challenge of maintenance. Fig. 2 shows the STARMATE system for engine maintenance. In 2003, Bernd Schwald et al. proposed an AR system for auxiliary maintenance in an industrial environment based on the STARMATE project. It uses a head-mounted optical transparent display to overlay guidance information in the user's field of view, and uses an infrared optical tracking system to give the attitude of the user and the device, which improves the stability of AR contents. In addition, the system proposes a scenario-based concept, which can guide users step by step according to the current training or maintenance task, further improving the practicability of AR maintenance.  [11], aiming to improve the effect of equipment maintenance and related personnel training. The prototype ARMAR system proposes the modular design concept, which can be maintained and upgraded without modifying the system framework. The system adopts the interactive strategy of opportunity control. When the operator touches the maintenance object, the system can analyze and bring up the virtual menu of the current maintenance object according to the location marker and gesture, and the operator can use gesture to select the operation process. In 2009, researchers at Columbia University developed a prototype system for AR assisted maintenance of armored vehicle turret based on the ARMAR project [12]. With the help of the system, military mechanics are able to perform 18 common field tasks in narrow armored vehicle turrets (Fig. 3). In 2015, the U.S. department of defense led the creation of the digital manufacturing and innovation agency, which listed the layout of manufacturing workshops based on AR and wearable devices as one of seven R&D initiatives. According to conventional AR auxiliary maintenance and assembly, this project utilizes multi-channel human-computer interaction technology and cloud technology to let various people who are far away from each other to share information (physical objects, drawings and models), exchange design concepts and advance design resolutions. The implementation of this project brings AR auxiliary maintenance to a new level. ARMAR system for turret maintenance of armored transport vehicles [12]. In January 2015, Microsoft Hololens was released, and AR maintenance systems developed using Hololens gradually increased. In December 2015, the Sidekick, developed using HoloLens, was deployed to the international space station. During the complicated operation of astronauts, the Sidekick provides assistance to astronauts by displaying virtual images covering real objects and realizes the interaction between astronauts [13]. The Italian air force has also developed applications for maintaining the C4I system using Hololens [14]. As shown in Fig. 4, the effectiveness was verified through AR auxiliary maintenance cases of air traffic control radar system, radio communication system and radio navigation system.  Figure 4. Italian Air Force adopts Hololens to adjust C4I systems [14].

Applications of Maintenance Training
Military equipment maintenance requires a high level of expertise. Experienced support personnel have accumulated a great deal of valuable knowledge over the years of practical operation and can correctly handle equipment failures. However, knowledge that is mastered by only a few can hardly be passed on to the novice in its entirety. The application of AR can integrate knowledge into the system and save it in the form of animation, voice and text for a long time. The use of AR technology can also solve the problem that the actual equipment cannot appear in the training site and cannot be repeatedly disassembled in the training. When the real equipment cannot appear in the training ground, virtual equipment can be used for training. When the real equipment cannot be disassembled repeatedly, AR technology can be used to display the internal structure and disassembly process of the equipment without disassembling. In the early 1990s, the European Computer Research Center annotated mechanical models applying AR. In 2003, the University of Toronto used AR to superimpose virtual images on stereo video images and successfully adopted them on teaching. By heightening the displayed structure line of the space shuttle, trainees can more intuitively know the construction of the aircraft without being influenced by conditions of observation. In 2012, Sabine Webel et al. produced a training platform of AR maintenance and assembly [15], and selected the mobile client to lead operators, and creatively introduced to practice tactile feedback to training, and carried out better training outcomes than conventional training. Summarizing the existing research work, AR equipment maintenance training mainly supports these scenarios: (1) distributed training: Utilizing spatial anchor technology, team members scattered in various geographical areas can perform collaborative training employing 3D network technology.
(2) immersion training: Real-time simulation of object behavior and multi-modal interaction helps the growth of sophisticated training systems, which can not only involve perceptual skills (e.g., how to respond in various fault scenarios). Meantime, it can include sensorimotor skills (e.g., how much force must adopt in a particular operation).
(3) mobile training: The scene can be narrowed down to the platform of smart phone and tablet computer that supports the work of support personnel. The remote consultation can connect mobile support personnel with remote experts, and various training scenarios can be easily distributed to achieve learning anytime and anywhere.

Framework of AR Maintenance System
AR maintenance system is a maintenance guidance system for electromechanical and hydraulic equipment in complex products. It uses AR technology to "enhance" display complex and diversified information such as detection data and operation steps used in the maintenance process, so as to realize the purpose of fault detection and maintenance operation for auxiliary maintenance personnel. The system mainly includes maintenance training, on-site maintenance guidance and remote expert guidance. The main technologies to implement the system include augmented reality (tracking registration technology, virtual and real fusion technology, human-computer interaction technology), interactive  Figure 5. Framework of AR maintenance system.

Display technology of AR
The display technology is the base of AR technology and one of the essential elements of the AR maintenance system. Via AR display devices, users can easily see the scene of AR. The current display devices principally divided into three varieties: head-mounted display (HMD) device, handheld mobile display device, and space projection display device. Among them, HMD can be divided into two types of optical see-through type and video see-through type according to the display principle [16]. The former enhances the display by directly overlaying virtual things in the real scene, while the latter enhanced the display by overlaying virtual things to the video of the real scene. Handheld mobile display devices regularly apply video perspective technology, while space projection display devices use 3D projection technology to project virtual things into real scenes directly. The head-mounted display device is the most generally employed one in AR maintenance system because of its portability and immersion. Currently, there are many products on the market. Among them, video see-through dispay technology has been relatively mature and is widely used in commercial applications. Between the symbolic products are HTC Vive, Oculus Rift, Varjo XR-1, and so on. However, the defects of video see-through devices are also obvious. Since the restriction of the imaging principle, the interaction is weak, the wearing comfort is not good, and 3D dizziness and other discomfort symptoms are likely to happen. In 2015, Microsoft published Hololens, which was optical see-through display device that helps gesture and voice communication. Hololens, through its own computing unit (CPU+GPU+HPU), optical elements and sensors, uses near-eye 3D technology and silicon-based liquid crystal projection technology, coupled with optical transparent digital lens, to make the virtual image enter the human eye after being projected by the micro-projector in front of it, so as to realize the display of virtual things in real space. In 2019, Hololens 2 was successfully released to support custom gesture interactions, improve visibility, and meet higher performance requirements. Compared with video see-through devices, optical see-through devices have more natural communication and faster reply. However, since real scenes are not input into the system, it is difficult to compensate for system suspension by mastering the video display rate.

Human-computer Interaction
In AR maintenance training system, human-computer interaction technology can make users communicate with virtual and actual content simply and efficiently. Gesture interaction technology allows users to adopt their hands to manage things in the AR conditions, which is one of the most generally employed interaction processes. Based on various input tools, gesture recognition technology in AR can be split into two varieties according to computer vision and sensors [17]. The former utilizes a camera to accumulate gesture images, and the computer examines and processes the images to recognize gestures ultimately. This technical interaction is direct and does not influence the regular operation of the user. Nevertheless, the algorithm is more complex, and the acceptance rate and real-time performance are inadequate. The latter employs data gloves and motion sensors [18] to distinguish and return the three-dimensional coordinates of the hand bones, through estimating the spatial position information and motion trajectory information of the human hand, and eventually carrying out the hand motion recognition. Furthermore, it can immediately capture the spatial coordinates and motion parameters of the human side without conversion, the data is more detailed than the former, and the recognition gestures are more reliable, and the accuracy is high. The disadvantage is that extra equipment requires to be installed. Voice interaction technology has also been broadly used in AR systems [19]. The voice interaction system can be separated into three subsystems: voice recognition, semantic understanding and voice output. The voice recognition system transforms the input voice into phonemes and handles the voice characteristic model for segmentation and identification. The semantic understanding system adjusts and connects the outcomes of voice recognition into words and sentences that correspond to the grammatical composition and language customs. The voice output system mixes the system vocabulary with voice feedback. Apart from gesture and voice interactions, new interaction technologies, for example, tactile sensations [20], eye movements [21], and physiological information (such as EEG, ECG, and EMG) have also been quickly expanded in the past few years, further improving the reality and immersion of AR. The continuous improvement of user interface makes human-computer interaction more efficient and natural. The sorts of user interfaces that are more generally utilised in AR cover tangible user interfaces [22,23], touch user interfaces, 3D user interfaces, and multi-channel user interfaces. The tangible user interface applies physical things in the real world to interact with the computer. It is the most commonly used user interface form in AR, which is a user interface with tactile as the main interaction method and widely used in mobile phones and tablets. The 3D user interface accepts virtual things to communicate with computers and is a significant class of user interface in AR, multi-channel user interfaces use multiple channels to interact with computing. Then, this user interface combines diverse human perception channels (for instance, vision, touch, hearing, etc.), is one of the popular research hotspots.

3D Registration
3D registration technology is the key technology of AR. It precisely calibrates virtual things in the real world; therefore, the computer-generated virtual things can be blended into real space. According to earlier analysis, Feng Zhou et al. further separated 3D registration technology into sensor-based registration, vision-based registration, and hybrid registration connecting sensors and vision [24]. Vision-based registration technology is more affordable than the other two approaches, and it is more accessible to enhance and innovate. Hence, it is one of the most broadly accepted registration techniques. Based on the different principles, vision-based registration can be split into three varieties under marker-based, markerless-based, and model-based. The marker-based registration can separate into two species: the registration technique based on artificial marker and the registration method under the natural marker. The former places the artificial sign in the actual scene, utilizes the camera to pursue and identify its location. Next, compares it with the distinguished marker, determines the camera's posture, and then calibrates the position of the virtual things by coordinating transformation to execute it correctly display in the actual scene [25], among which the most typical are ARToolkit [26] and ARTag [27]. Besides, the registration based on artificial marker usually damages the integrity of the real scene. The latter figures out this difficulty perfectly. In addition, it traces the natural characteristics in the actual scene, uses the detected natural scene objectives to measure the camera posture. Later, to complete the calibration of the natural scene things by the transformation of natural feature coordinates and camera posture [28]. Both Metaio AR SDK and Vuforia SDK [29] have chosen this way and carried out excellent tracking registration consequence. The markerless-based registration can make the camera pose by preparing the constant video image frame obtained by the camera, to accomplish the registration process of the virtual target. Hololens utilizes the markerless-based registration to fulfil the stable display of the virtual object. Model-based registration regularly employs the model with obvious characteristics (for example, CAD model or 2D template), calculates the camera's attitude by detecting the external features of the model. Additionally, it transforms the 3D coordinates of the model into 2D plane coordinates and ultimately achieves the registration of the model in the real scene [30].

Insufficient System Availability
Through investigating the current AR maintenance systems, it is common to discover that they require to understand the location of the failures in advance, and then assist the user in making maintenance operations. Also, the amount of failures that can give maintenance guidance is minimal, which rigorously limits the military practicality of AR equipment maintenance. The daily performance and maintenance of military equipment comprise a large number of electro-hydraulic equipment's daily technical state detection, state parameter adjustment, failure analysis, and equipment maintenance and repair work. At present, it still depends on experienced technicians to carry out. When encountering complex faults, the "nanny" accompanying support is often relied on in the industrial sector, but as the number of failures increases and equipment performance declines, the existing strength of the industrial sector has been unable to meet the burden of the accompanying equipment support task. In future wars, military equipment will be more decentralized, and the "nanny" accompanying support in the industrial sector will be difficult to achieve. Therefore, how to implement automatic fault diagnosis based on the existing AR maintenance system is an important obstacle that requires to be figured out. Besides, the current AR maintenance system rarely records maintenance process information in real-time. It is difficult to obtain evidence after a problem occurs retrospectively. At the moment, it is not feasible to quantitatively estimate the operation of maintenance personnel after the maintenance is carried out. Consequently, the monitoring records of the maintenance process and results in evaluation issues are required to realize.

Insufficient AR technologies
Civil AR devices cannot reach the requirements of military activities. On the one hand, current AR devices cannot prevent soldiers well in battlefield conditions. Specialists such as Eric Gans produced the ARC4 system [31] to improve the situational consciousness of soldiers and mixed them into military helmets in three various methods to present AR services while better protecting soldiers' heads. In order to adapt to different usage scenarios, the ARC4 system can directly integrate the sensor component with transparent display technology to create a separate helmet component (Fig. 6a). Also it can be collected in an individual module and established in the helmet track (Fig. 6b) or immediately connected to the helmet (Fig. 6c). The current AR military equipment maintenance system has not combined with individual equipment, and the ARC4 system has inspired researchers. On the other hand, performance parameters, for example, the use temperature, battery life, and field of view of civil AR equipment cannot meet the needs of military activities. Taking Hololens as an example, its general battery life is 2-3 hours, which cannot meet the needs of long-term military operations. Meantime, its operating temperature range cannot meet the military electronic product standard of -40 ~ 125 . Its field of view ℃ is just 30 °, which is small compared to human's natural field of vision (Fig. 7), and will seriously affect the soldier's perception of the real environment. With cooperation between AR hardware development enterprises and the military in the future, profoundly customized AR devices for military equipment maintenance will be advanced, and the previous difficulties will be efficiently solved.  Figure 6. Three different installation techniques of ARC4 system [31]. Real-time and robustness require to be further advanced. Real-time and robustness are the key evaluation criteria for the experience of using AR systems. Vision-based tracking registration technology is usually applied in AR maintenance training systems. However, all vision-based technologies are susceptible to interference from battlefield environmental conditions, such as light, oil, and dust. Using lighting as an instance, histogram equalization and other methods can effectively solve the lighting problem, but it will lead to error accumulation and eventually make tracking less reliable and less robust. To solve this problem, researchers began to use CAD models to extract edge features of virtual objects and compare them with real objects captured by cameras. This method can only be applied to the initialization of AR systems and static AR scenes. However, soldiers using AR maintenance training systems often need to quickly rotate and move their heads, which leads to the loss of tracking targets. How to accurately and rapidly relocate is also one of the difficulties to be determined. Although there are more strong tracking technologies, in specific application scenarios, their reliability still does not take into account the battlefield conditions. Human-computer interaction cannot meet the interaction requirements in complex battlefield environments Current AR maintenance training systems generally use gesture and voice interaction, and gesture interaction is the most commonly used. In battlefield repairs, soldiers need to use tools for maintenance operations, and their hands are quickly involved, so they often cannot use gestures to interact with the system. However, the battlefield is often noisy, and the efficiency of soldiers using voice interaction will also decrease greatly. Therefore, other interaction methods (such as touch, eye movement, and EEG) are required to support in accomplishing multiplied channel interactions. Also, a huge amount of information in the AR maintenance training system (such as paper, text, models, animations, etc.) increases the cognitive load of soldiers. If the amount of information presented is not limited, it may lead to confusion in the soldiers' thinking [14]. However, the information provided must be complete, and how to display the required information in a particular scenario is also a problem that needs to be addressed in the future. The adaptive human-machine interface provides a solution to this problem. It has applied to medical, command and control, banking and other fields, but the adaptive human-machine interface in AR maintenance systems has not yet matured.

Further Developing Tendency
Under the development situation of AR-related technologies, the development trend of AR military equipment maintenance training in the future can be foretold, which includes intellectualization, standardization, and modularity.

Intellectualization
Currently, the application of computer vision technology can identify and detect image characteristics in real-time, for example, the state information of instruments, indicator lights, waveforms and other error information such as fracture, displacement and deformation in the equipment. Utilizing neural network technology can carry out quick and precise fault reasoning (for instance, an expert system according to the neural network). Furthermore, adopting an open layout, a huge volume of data calculations in the AR maintenance system can be installed in the server. Also, the AR device only requires to be accountable for enhanced display and human-computer interaction. The server and the AR device can apply wireless technologies, for example, 5G and Cloud technologies [32]) for low-latency associations. In general terms, the technical base of the intelligent AR system already survives, and the intelligence of the AR maintenance system will slowly turn into mainstream. Investigators have started to attempt to apply AI to AR. In reference [33], AR and intelligent assistant system are connected to support the training of manual assembly assignment. While a worker made a mistake, a warning notification was presented to fix it. Compared with an AR training system without intelligent assistance, the intelligent AR system advances the test score by 25% and task performance by 30%. An information-centric AR maintenance system design framework has been proposed, and future development of relevant technologies will make AR systems based on this framework adaptive and intelligent [34]. In the future, the combination of AR and AI will be closer. The intelligent AR maintenance system can not only achieve fault diagnosis of intelligent equipment and automatically present answers, but also observe and record the operations of maintenance personnel throughout the process, and ultimately quantify the maintenance outcomes. Fig. 8 reveals the potential framework of the further intelligent AR maintenance system.  Figure 8. Intelligent AR maintenance system framework.

Standardization
Currently, there is no universal standard for AR equipment and AR development platform, which is not only bad for users' use but also bad for technical communication. Many researchers have raised this problem [14,32]. Thus, IEEE launched a particular working team on virtual reality and augmented reality, which is accountable for executing standardization activities, and proclaimed eight IEEE standards in 2017 [14]. Then, these eight standards comprise equipment classification and definition, immersive video file format, immersive user interface, etc.. Nevertheless, these standards essentially deal with software issues, not hardware standards, and have not been extensively adopted. In the military field, there are no distinct AR standards. Meanwhile, diverse development teams adopt various hardware and software platforms. Not only do soldiers spend numerous time to turn into familiar with them, but a lot of manpower and resources are also wasted. Consequently, how to generate AR standards to  10 consolidate hardware and software platforms based on military application scenarios is one of the essential works in the future.

Modularity
In the future, the AR military equipment maintenance system will adopt the design idea of open modularization to adapt to different scenarios. When the use scenario changes and the system needs to be upgraded or the maintenance process needs to be modified, as long as the corresponding modules are modified, there is no need to change the system framework. Also, the failing database is renewed in real-time, so when maintenance personnel need to optimize the maintenance procedure or combine new faults, they can instantly cooperate with the system and modify the maintenance content quickly. If the maintenance operators need remote assistant experts, they can also manage the maintenance immediately procedure online.

Summary
Overall, AR technology has extensive application prospects in the maintenance of military equipment and can make significant modifications to military equipment support. Although contemporary AR technology and devices are not mature enough, and there are still some uncertainties in military activities. By contrast, with the evolution of human-machine interaction, tracking registration, and artificial intelligence, the future mature AR maintenance system for military equipment is worth looking forward to.