{'fraction_non_alphanumeric': 0.022065124250214223, 'fraction_numerical': 0.009497286489574408, 'mean_word_length': 4.679237631792376, 'pattern_counts': {'":': 0, '<': 1, '<?xml version=': 0, '>': 0, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Online visual inspection of self-piercing riveting to determine the quality of the mechanical interlock

Self-piercing riveting (SPR) has become a significant joining technique for the automotive applications of aluminium sheets. Quality control in this locale has progressed at an altogether more leisurely rate than other areas of mechanical joining (e.g. spotweld) and is underdeveloped. Testing the quality mechanical interlock is often achieved by destructive testing, which results in material and time wastage. The solution is online monitoring of the self-piercing riveting process to provide non-destructive testing of the mechanical interlock. Introducing sensors into the process facilitates real time data acquisition, which can be used to determine the quality of the joint.


Introduction
Mechanical fasteners are hardware that join two or more objects together. There are two main groups of mechanical fasteners; threaded fasteners (nuts and bolts, screws etc) and non-threaded fasteners (staples, nails, rivets etc). Riveting is a fast method of permanently joining metal sheets together. Riveting can be sub-divided into hot and cold categories. In hot riveting [1] the rivet is heated before inserting it into a pre-drilled hole. The key advantage of this process is a tight join, which is achieved by impacting the rivet from both sides of the material thus forming two heads. As the rivet cools it contracts and pulls the materials together forming an airtight and watertight interlock. This is therefore best suited for boiler or shipbuilding. Cold riveting is performed at room temperature, and in the case of SPR can be applied without a pre-drilled hole [2]. There are three main types of cold riveting rivets; solid, full tubular and semi-tubular -each of which can then be further broken down into several configurations. Solid rivets have entirely solid shafts with no internal cavities. Hammering the protruding end forms a tough connection and secures the rivet. Solid rivets require access from both sides of the structure and are applied using a hydraulic or pneumatic squeezing tool, or hand held hammers. Tubular rivets have a coaxial cylindrical hole in the headless end that exceeds 112% of the rivet shank diameter. They are designed to split at the end, securing the rivet in the material and are most commonly used in self-piercing applications. They have vast industrial, aerospace and automotive applications. Semi-tubular rivets are used in impact riveting applications where the rivet end flares outward upon contact. They have a coaxial, cylindrical or tapered hole in the end opposite the head, the depth of which does not exceed 112% of the mean shank diameter. Although they are comparable to solid rivets, they require much less insertion force permitting longer rivets to be utilised without the rivet shank buckling. Semi-tubular rivets will be applied throughout the course of this research.

Riveting Hardware
Riveting machinery applies rivets to materials and is available in a wide variety of configurations, from manually operated hand riveters and handheld riveting guns to multi-head automated riveting tools that are electrically, pneumatically, or hydraulically actuated. There are three main types of riveting: compression riveting, impact riveting and non-impact riveting. Compression riveting forms the head of a rivet as a result of squeezing or pulling the rivet shank whereas impact riveting forms the head by impacting the top of the shank, often using riveting hammers. Non-impact riveting (orbital riveting) forms the head of the rivet by performing a spinning or rolling action to the end of the top of shank. The machine used in this research is an impact "120 rivet setting machine" [3]. This machine is easy to use and has a good deal of space for attaching the sensors that are used in the monitoring system.

Self-Piercing Riveting
Self-piercing riveting is a high-speed joining process that does not require pre-existing holes in the materials [4], therefore eradicating the need to align the joining materials. The materials to be joined are stacked, with a punch above the stack and die underneath it. The punch impacts the rivet, piercing the top one, and partially piercing the bottom. The die on the underside of the materials causes the rivet to flare under the force, creating a mechanical interlock [5]. This process therefore requires access to both sides of the joint [6]. The entire process of piercing and forming the joint is carried out in a single operation [7]; figure 1. Self-piercing riveting operation with a semi-tubular rivet. [8] There are many advantages associated with SPR. It allows the joining of pre-painted and galvanised materials without damaging the coatings and the joining of unlike materials (e.g. aluminium and steel). SPR also allows the joining of materials without a pre-drilled hole, which eradicates the need to align the holes and thus saves time [9]. It is a low noise process (< 80dBa [10]) and environmentally friendly. The lack of heat and fumes [11] removes the need for extra equipment such as extractors. Riv-bonding uses SPR in collaboration with adhesives to provide extra strength [12]. The entire process takes between 2 to 4 seconds [10] and can function automatically or manually, as required. However, there are disadvantages associated with SPR. Access to both sides of the material is required and sizeable robot payloads may restrict joint access further. If diverse materials are being used, the rivet is typically thrust through the thin to thick or low strength to high strength material for optimal performance. The thinner/softer material is secured by the rivet head, while the flared end of the rivet is attached to the thicker/stronger material. The completed joint is not flat; the rivet head remains on the surface of the top material, while the button formation is located on the underside of the bottom material. It is also important to note there are numerous rivet head styles, each with their own purposes, advantages and disadvantages. However, for the purpose of the research, the rivet head style is not important.

State of Industry
Much research and development has been performed worldwide concerning SPR techniques, however this has been primarily in the automotive industry. Joining techniques in the automotive industry are predominantly driven by advances in materials (joining dissimilar materials) and the call for increased automation due to a decline in the skilled labour force [13]. Inspection of joining processes has progressed at a much slower pace. A visual indication of riveting joint quality can be determined with riveting by performing dimensional checks of the button formation [14]. Throughout rivet manufacture, stringent quality control is enforced to guarantee a consistent product and performance.

Computer Vision
Current monitoring technologies use various means to acquire and analyse data, however computer vision is overlooked by all of them. This research utilises computer vision to identify the presence of a rivet and if two rivets are loaded in error. The camera is connected to a computer and read into bespoke process monitoring software. It is important to monitor the presence of a rivet before the setting force is applied as this can cause several problems. In a worst case scenario if an essential rivet were not set and went unnoticed, automotive companies may have to mass recall all vehicles on safety grounds. A nonhazardous example would be a cosmetic problem, but even this is costly to remedy. If a rivet is not loaded and the punch is fired, the punch may leave a circular imprint on the surface of the aluminium joint as shown in Figure 3.  Similarly if two rivets are loaded in error and set into the materials as shown in figure 3, the strength and aesthetics of the join are worsened. Both these problems can be identified by the process monitoring software with the use of computer vision before the procedure has commenced. Detection of these problems is achieved by converting the camera input to greyscale (R30, G59, B11) and reading the intensity values of pixels (0 to 255) in key locations. Once the intensity values are read, an algorithm is used to learn if a rivet or rivets are present. Figure 4 shows a single rivet seated in the machine jaws ready to be set. It is at this stage the process monitoring software samples a row of pixels at three key locations; the left jaw (1), the rivet (2) and the right jaw (3). Each sample consists of 50 pixels and is easily identified by the green lines.    figure 6 respectively. Based on a sample of 100 single rivets, the process monitoring software achieved 100% identification. Identifying whether a second rivet is loaded in error yields very similar results. The sample locations are repositioned to match the location of the second rivet. The average intensity level of sample 2 (the rivet) is much greater than samples 1 and 3 (the jaws) as shown in Figure 6. This will indicate if two rivets are loaded and prevent them being set as shown in Figure 3.  Again, the intensity values in sections 1, 2 and 3 of figure 8 are typical of identified double rivets. Figure  7 compares intensity levels of four identified double rivets. As previously mentioned the areas of interest are relocated to ensure the second rivet is accurately detected. The second rivet is slightly lower and further to the left than the single rivet and therefore the sample locations must be repositioned accordingly. Pixels 65 -125, 140 -190 and 220 -270 are labelled as samples 1, 2 and 3 respectively in figure 6. The intensity levels shown in figure 7 fluctuate across pixels 140 -190. This is largely due to the unpredictable orientation of the second rivet, which, depending on its orientation can reflect different levels of light and therefore return a varying intensity level. This identifies the important issue of controlling lighting; as intensity levels will of course vary if the lighting varies -this will be discussed later. However based on a sample of 100 double rivets, the process monitoring software achieved 100% identification. Figure 8 compares the average light intensities of identified missing, single and double rivets and enables the process monitoring software to differentiate between them with reference to the intensity of the jaws. The area between the jaws is again of interest. It can be seen that the average for the missing rivet is 25 and much lower than that of the single or double rivets. The average of identified single rivets is 52, while the average of identified double rivets is 62. A simplistic trend is that the higher the average intensity level, the more rivets are present.

Laser Measurement
Laser is an acronym for light amplification by stimulated emission of radiation. Stimulated emission transpires when an atom/molecule holding surplus energy is stimulated to emit the energy as light. Laser based measurement is an accepted technology for process monitoring and control [15] and is fast and accurate which lends itself greatly to on-line measurement and feedback. It is important to track the head formation as it can detect problems with the joint. For example, there are several reasons a rivet head may be too low. The setting force may be too great, the rivet may have been set into the wrong material (too weak) or incorrect depth of material (too thin). Similarly if a rivet head is too high the setting force may be too little, the rivet may have been set into the wrong material (too strong) or incorrect depth of material (too thick). The location of the rivet head should correlate with the button formation on the underside of the joint. For example, if the head of the rivet is too low, then the button formation will extend further out of the material than normal, and vice versa. Therefore by including the laser line measurement it enables The raw image can be difficult to analyse due to the varying light intensities. This is overcome using several filters to ensure only red light is allowed to pass into the camera as shown in figure 9.  The next step is to apply thresholding and find the midpoint of the laser line, which the process monitoring software achieves by labelling the start and stop point for any white pixels on the Y axis and taking the midpoint. This midpoint is plotted -as shown in figure 11 -and can be exported for further data analysis. The process monitoring software references these midpoints with those of known good quality and comes to a decision based on this and specified tolerances. In figure 11 the rivet head is shown from pixel 135 to 180. The u-shape in the graph tells us the position of the head with reference to the surface of the materials to be joined. The depression shows the rivet head is approximately 10 pixels higher than the metal surface. This coupled with the shape of the depression tells us the rivet was successfully set. The shape shown in figure 12 shows a depression approximately 10 pixels lower than the material surface. On its own this could mean the rivet was set too deeply into the materials or the rivet was not fired and therefore the depression was made by the punch. In this case the depression was made by the punch and is identified using the shape of the graph.

Conclusion and Future Work
It is clear that computer vision is another useful sensor for monitoring SPR systems. Our current process monitoring system achieves 100% accuracy in detecting missing, single and double rivets. Future work includes tracking punch position and speed, as well as employing ultrasonic testing on the post-process joint. Each sensor will be assessed and if it provides meaningful data it will be included in the process monitoring system. Each of the included sensors will be combined into a multiple sensor cluster and will be utilised in such a manner as the limitations of one sensor is negated by the advantages of another.