{'fraction_non_alphanumeric': 0.03300720352650253, 'fraction_numerical': 0.0036555209117299214, 'mean_word_length': 4.687729311047697, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 1, 'https://': 0, 'lorem ipsum': 0, 'www.': 3, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 4, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Region-Based Fusion for Infrared and LLL Images

Thermal cameras and image intensifiers are common night vision (NV) cameras, which enable operations during night and in adverse weather conditions. NV cameras deliver monochrome images that are usually hard to interpret and may give rise to visual illusions and loss of situational awareness. The two most common NV imaging systems display either emitted infrared radiation or reflected low level light (LLL). In this way the different imaging modalities give complementary information about the objects or area under inspection. Thus, techniques for fusing infrared and LLL images should be employed in order to provide a compact representation of the scene with increased interpretation capabilities. Image fusion can be classified into two types based on pixel-level: pixel-based and regionbased. The pixel-based image fusion is characterized by simplicity and highest popularity. Because pixel-based methods fail to take into account the relationship between points and points, the fused image with either of them might lose some gray and feature information. However, for most image fusion applications, it seems more meaningful to combine objects rather than pixels. The region-based fusion, on the contrary, can obtain the best fusion results by considering the nature of points in each region altogether. Therefore, region-based fusion has advantages over the other two counterparts. At present, region-based methods use some segmentation algorithm to separate an original image into different regions, and then design different rules for different regions. During the last decade, a number of gray fusion algorithms have been proposed, and the fusion methods based on the multiscale transform (MST) are the most typical. The commonly used MST tools include the Laplacian pyramid and the wavelet transform (DWT). In general, due to the perfect properties of the DWT such as multi-resolution, spatial and frequency localization, and direction, the DWT-based methods are superior to the pyramid-based methods. However, the DWT also has some limitations such as limited directions and non-optimal-sparse representation of images. Thus, some artifacts are easily introduced into the fused images using the DWT-based methods, which will reduce the quality of the resultant image consequently. The Dual-Tree Complex Wavelet Transform (DT-CWT) has been introduced by Nick Kingsbury, which has the following properties: Approximate shift invariance; Good directional selectivity in 2-D with Gabor-like filters also

true for higher dimensionality; Perfect reconstruction using short linear-phase filters; Limited redundancy: independent of the number of scales. Therefore, the Dual-Tree Complex Wavelet Transform is more suitable for image fusion. In the context of NV imaging, a number of color fused-based representations have been proposed. A simple mapping of infrared and visual bands into the three components of an RGB image can provide an immediate benefit, since the human eye can discriminate several thousands of colors but only a few dozens of gray levels. On the other hand, inappropriate color mappings may hinder situational awareness due to lack of color constancy. Hence, an image fusion method for night vision imagery must result in color images with natural appearance and a high degree of similarity with the corresponding natural scenes. To make the coloration of false-color images appear more natural, Reinhard recently introduced a method that enabled the transfer of colors from one image to another. Subsequently, Toet demonstrated that Reinhard's method could be adapted to transfer the natural color characteristics of daylight imagery into multi-band infrared and LLL images. Essentially, Toet's natural color mapping method matches the statistical properties of the NV imagery to that of a natural daylight color image. However, this particular color mapping method colors the image regardless of scene content, weights all regions of the source image by the ''global'' color statistics, and thus the accuracy of the coloring is very much dependent on how well the target and source images are matched. In this chapter, we present a region-based gray fusion method using the DT-CWT and a region-based color fusion method for infrared and LLL images. Segmentation is very important because segmentation precision has a great influence on the following fusion process. Here, we adopt two segmentation methods: the morphologic mehod and the nonlinear diffusion method. In the gray fusion method, the infrared and LLL images are decomposed by DT-CWT, the segmentation regions are mapped into each level, and fusion is carried out region by region in terms of some fusion rules. The region-based color fusion method is based on Toet's global-coloring framework.

Image fusion based on region segmentation and complex wavelets
The region-based fusion method for infrared and LLL images adopts the DT-CWT because of its approximate shift invariance and limited redundancy. Diagram of the proposed region-based method with DT-CWT is shown in Fig. 1. Segmentation is firstly performed on the infrared image and LLL images respectively with top-bottom-hat filtering and the threshold method, consequently, the DT-CWT coefficients from the different regions are merged separately. Finally the fused image is obtained by performing inverse DT-CWT.

Image segmentation using morphology
The morphologic filters have been proven as powerful methods in the denoising and smoothing of image intensities while retaining and enhancing edges. The combination of different morphologic filters makes the segmentation flexible. The top-hat transform and the bottom-hat transform are all the combination of open operation, close operation and the original image. The top-hat transform means subtracting a morphologically opened image from the original image and it can be used to enhance contrast in an image. The bottom-hat transform means subtracting the original image from a morphologically closed version of the image and it can be used to find intensity troughs in an image. The formula of the top-hat transform and bottom-hat transform are given by respectively Here f is the original image, "" and " Then the threshold method is used to segment the enhanced image E H . We can get the binary segmentation image based on this method. Because the physical significance of the pixel at the same location of the heterogeneous source images is different, the shapes of segmentation regions obtained by the former method are also different. So we must deal the segmentation region with the associate methods. The information of segmentation region should be added to the associated-segmentation image and is used to guide the fusion rules. The following steps are used to generate the associated-segmentation image: 1. If there is no overlapping area between the region (1) R and the region (2) R , then the associated-segmentation image is mapped into two regions, If there is some overlapping area between the region (1) R and the region (2) R , then the associated-segmentation image is mapped into three regions,

If region (1)
R overlaps the region (2) R completely, the associated-segmentation image is mapped into the same region, () (1) (2) j RRR ==; 4. If one region completely contains the other region, for example, (1) (2) RR ⊂ , then the associated-segmentation image is mapped into two regions, R is presented a part of one source image, (2) R is presented a part of the other source image. () j R is presented a part of associated-segmentation image. Fig. 2 is some typical examples of the associated region maps. There are some small regions in the associated-segmentation image, and they don't contain enough region information, which will cause false image in the fused image. We may merge these small regions using morphological operators.

Pixel fusion with complex wavelets
The dual-tree complex wavelet transform (DT-CWT) is a relatively recent enhancement to the discrete wavelet transform (DWT), with important additional properties: It is nearly shift invariant and directionally selective in two and higher dimensions. It achieves this with a redundancy factor of only 2d for d-dimensional signals, which is substantially lower than the undecimated DWT. The multidimensional (M-D) DT-CWT is non-separable but is based on a computationally efficient, separable filter bank (FB). For 2-D signals, we can filter separately along columns and then rows by the way like 1-D. Kingsbury figured out in that, to represent fully a real 2-D signal, we must filter with complex conjugates of the column and row filters. So it gives 4:1 redundancy in the transform. Furthermore, it remains computationally efficient, since actually it is close to a classical real 2-D wavelet transform at each scale in one tree, and the discrete transform can be implemented by a ladder filter structure. The quad-tree transform is designed to be, as much as possible, translation invariant. It means that if we decide to keep only the details or the approximation of a given scale, removing all other scales, shifting the input image only produces a shift of the reconstructed filtered image, without aliasing. The most important property of DT-CWT is that it can separate more directions than the real wavelet transform. The 2-D DWT produces three band-pass subimages at each level, which are corresponding to LH, HH, HL, and oriented at angles of 0 o , ± 45 o , 90 o . The 2-D DT-CWT can provide six subimages in two adjacent spectral quadrants at each level, which are oriented at angles of ±15 o , ±45 o , ±75 o . The strong orientation occurs because the complex filters are asymmetry responses. They can separate positive frequencies from negative ones vertically and horizontally. Therefore, positive and negative frequencies will not be aliasing. The pixel-level fusion scheme used here employs the DT-CWT to obtain a MR decomposition of the input images. The wavelet coefficients are then combined, using a maximum-selection fusion rule to produce a single set of coefficients corresponding to the fused image. This process is shown in Fig. 3. The maximum-selection scheme selects the largest absolute wavelet coefficient at each location from the input images as the coefficient at that location in the fused image. As wavelets tend to pick out the salient features of an image, this scheme works well producing good results.

Image fusion based on region segmentation and complex wavelet
Decomposed by the multi-resolution DT-CWT, low-frequency part of the images denotes their approximate components, which contains spectral information of the source image. High-frequency part of the images denotes their detail components, which contains edge detail information of the source images. So, fusion algorithms after the source images decomposed are very important for the quality of fusion. At present, the fusion rules are commonly that average operator or weighted average operator is used in low-frequency domain, max absolute operator is used in high-frequency domain. For the two fused heterogeneous source images of the same scene, spectral information of one image is usually much richer than the other. For example, spectral information of visible light image is much richer than the infrared image. If the fusion rules of weighted average is adopted, part of spectrum information of visible light images will be lost, which results in that the spectrum information of fused image is less than visible light image.

www.intechopen.com
To overcome these problems, we adopt spatial frequency to guide region-based fusion. The spatial frequency, which originated from the human visual system, indicates the overall active level in an image. The human visual system is too complex to be fully understood with present physiological means, but the use of spatial frequency has led to an effective objective quality index for image fusion. The spatial frequency of an image block is defined as follows: Consider an image, the row ( F R ) and column ( F C ) frequencies of the image block are given by Here Ω is a certain segmentation region. The total spatial frequency F S of the image is We use the fusion of two registration source images A and B as an example, the image fusion process based on region segmentation and DT-CWT is accomplished by the following steps: Step 1: Partition the source images A and B , then we get the region segmentations named A R and B R , using associated processing, then we can get the associated-segmentation image J R . Calculate F S of each region in the associated-segmentation image.
Step 2: Compare the spatial frequency of the corresponding regions of the two source images to decide fusion coefficients: Here F i R is the i th region of the fused image, A Fi S and B Fi S are the spatial frequencies of the i th region of image A and B , respectively, k is a threshold.
Step 3: Multi-level DT-CWT transform on the source images A and B , then we can get DT-CWT coefficients at different scale Layers, which contain low-frequency coefficient and high-frequency coefficient at different scale layers.
Step 4: Deal low-frequency and high-frequency part with fusion rules and fusion operators, then we get low-frequency coefficient and high-frequency coefficient at different scale Layers after fusion.
Step 5: Deal low-frequency coefficient and high-frequency coefficient at different scale Layers with DT-CWT inverse transform, then the reconstruction image is to be fused image.

Experiment results
To evaluate the presented fusion algorithm, we fuse the infrared and visible images of the same scene with this algorithm, and compare the fusion image with the fusion images with the DWT method (method 1) and SIDWT method (method 2). Fig. 4(a) is an infrared image, which presents the clear shapes such as a human being, trees and some high-temperature objects; Fig. 4(b) is a visible light (low light level) image, which provides more details than the infrared image. Besides this, it also shows some light sources. Fig. 4(c) is the segmentation region of the infrared image and Fig. 4(d) is the segmentation region of the visual image. Fig. 4(e) is associated region map of infrared/visible images, Fig. 4(f) is fused image with method 1 and Fig. 4(g) is fused image with method 2; Fig. 4(h) is fused image with the presented method. According to the fusion images, the presented fusion algorithm has better effectiveness, which preserves not only the spectral information of the visible light image, but also the thermal target information of the infrared image. The details of the fusion image with the presented algorithm is clear, which shows that region segmentation has a function of extracting targets, also shows that the DT-CWT has the capability of capturing edge information. Though the fusion images with method 1 and 2 also reserve main scene information of the two images, they lose some details slightly. Edge of objects looks blurry slightly.
I n t h i s p a p e r , w e u s e a n 1 1× 11 circular-symmetric Gaussian weighting function to modify x μ , y μ , xy σ , x σ and y σ . With such a windowing approach, the quality maps exhibit a locally isotropic property. In practice, one usually requires a single overall quality measure of the entire image. We use a mean SSIM index to evaluate the overall image quality.
In a similar fashion we compute y (w) Thus, in regions where image x has a large saliency compared to y , the quality measure SSIM( x,y , f ) is mainly determined by the "similarity" of f and input image x . On the www.intechopen.com other hand, in regions where the saliency of y is much larger than that of x , the measure SSIM( x,y , f ) is mostly determined by the "similarity" of f and input image y .
/ ABF Q is based on the idea that a fusion algorithm that transfers input gradient information into the fused image more accurately performs better. For the fusion of input images A and B resulting in a fused image F , gradient strength g and orientation ([ 0 ,] ) α π ∈ are extracted at each location (, ) nm from each image using the Sobel operator and used to define relative strength and orientation "change" factors G and A , between each input and the fused image: where M is 1 for FA gg > and -1 otherwise. An edge information preservation measure AF Q models information loss between A and F with respect to the 'change' parameters with sigmoid functions defined by constants Γ , g κ , g σ , α κ , and α σ : and ABF Q /

Region-based color fusion for infrared and LLL images
Toet demonstrated that transfer of colors could be adapted to transfer the natural color characteristics of daylight imagery into multi-band infrared and LLL images. However, this particular color mapping method colors the image regardless of scene content, weights all www.intechopen.com regions of the source image by the ''global'' color statistics, and thus the accuracy of the coloring is very much dependent on how well the target and source images are matched. Based on Toet's global-coloring framework, we present a new region-based method that the image segmentation is firstly carried out and then region coloring is realized.

Review of global-coloring method
The aim of the global-coloring is to give NV images the appearance of normal daylight color images. A false-color image (source image) is first formed by assigning multi-band NV images to three RGB channels. The false-color images usually have an unnatural color appearance. Then, a true-color daylight image (reference image) is manually selected with similar scenery (e.g., syntactic content and color appearance) to the NV images. Both source and reference images are transformed into a Luminance-Alpha-Beta ( lαβ ) color space, followed by calculating the global mean and standard deviation for each lαβ plane. Next, a ''statistic-matching'' procedure is carried out between the source and reference image. The mapped source image is then transformed back to RGB space. Finally, the mapped source image is transformed into YCbCr space and the ''value'' component (similar to the luminance component in lαβ decomposition) of the mapped source image is replaced with the ''fused NV image'', which is a grayscale image made with multi-band NV images (e.g., image intensified and infrared image). This fused image replacement is necessary to make the colorized image have a proper and consistent contrast. Notice that the ''luminance'' component in lαβ space cannot be used directly for this replacement because its dynamic range is very different from that of the fused image, whereas the "value" component in YCbCr space has the same gray-level range as the fused image. The lab space is utilized for color mapping because of its decorrelation property of three channels, whereas the YCbCr space is suitable for human interface. The fusion process can be summarized in the following steps: where I C is the colored image, S I is the source (false-color) image in lαβ space; μ denotes the mean and σ denotes the standard deviation; the subscripts ' S ' and ' T ' refer to the source and reference images, respectively; and the superscript ' k ' is one of the color components { ,, l α β }.After this transformation the pixels comprising source image have means and standard deviations that conform to the reference daylight color image in lαβ space. 6. The inverse transform from the lab space to the LMS space can be expressed by 9. The brightness f Y acquired by transforming the fusion image and reference image from RGB space to lαβ space is not usually appropriate, because f Y is the weight sum of the infrared and low-light-level images. Thereby, we adopt the fusion image by laplacian pyramid fusion method replacing f Y . 10. Transfer the adjusted rough fusion image data from BR YC C space back to RGB space, and we can get the ultimate re-staining rough fusion image.

The region-based coloring fusion method
Based on the framework of the global-coloring method as described in Section 3.1, we present a region-based coloring fusion method (see Fig. 5.) that makes the fusion images appear more like daylight imagery. The major points for this new method are as follows.
(a) The infrared and LLL images are rendered segment-by-segment. (b) The segmented regions of the two images are combined and form a new segmented map. (c) These regions are classified according to the target types and the spatial frequencies, and some valuable targets are extracted according to the luminance of the infrared images or the motion trend of the infrared and LLL video. At present, our classification is still carried out manually. (d) The infrared and LLL images are mapped into the RGB space. A lot of mapping color methods has been provided, but the simplest mapping method is still suitable. For example, the infrared image is mapped into the R channel and the LLL image is mapped into the G channel and the average of the infrared and LLL images is mapped into the B channel. (e) Some typical scene images must be chosen as the reference images. These images should include some features similar to a certain segmented region. (f) Transfer of color is run region by region in the lαβ space according to Reinhard's method. (g) The gray fusion image is used to replace the Y of the color fusion image in the YCbCr space. Here, the gray fusion method may be the fusion method in Section 2 or some other classical ones. Image segmentation is quite challenging because image contents vary greatly from image to image. We adopt two segmentation methods. One is the morphologic method in Section 2, the other is the nonlinear diffusion method. The two methods have been proven as powerful methods in the denoising and smoothing of image intensities while retaining and enhancing edges. Such an image smoothing process can be summarized as a successive coarsening of any given image while certain structures in that image are retained on a fine scale. Basically, diffusion is a PDE (partial differential equation) method that involves two operators, smoothing and gradient, in 2D image space. The nonlinear diffusion equation is Where ∇ is a vector containing gradients taken at different neighboring configurations (i.e., nearest-neighbors, second-neighbors, etc.) and () x ω are the nonlinear diffusion coefficients.
The diffusion process smoothes the regions with lower gradients whereas stops smoothing at region boundaries with higher gradients. Nonlinear diffusion means the smoothing operation depends on the region gradient distribution. In other words, the diffused result is a nonlinear function of local gradients. Diffusion must be used with clustering and region merging techniques together, which make the segmentation flexible.

Experiment result
Here two experiments have been carried out with the region-based coloring fusion method. The only difference is that the morphologic method is adopted in experiment 1 and the nonlinear diffusion method is adopted in experiment 2.

Experiment 1
To evaluate the region-based coloring fusion method, we fuse the infrared and LLL images of the same scene, and compare the fusion images with the presented method and the global-coloring method. Fig. 6(a) is an infrared image, which presents the clear shapes such as a human being, trees, building, pool and some high-temperature objects; Fig. 6(b) is a LLL image, which provides more details than the infrared image. Besides this, it also shows some light sources. Fig. 6(c) is the fusion image acquiring by Section 2. Fig. 6(e), (g), (i) and (k) are the fusion images with global-coloring method separately using Fig. 6(d), (f), (h) and (j) as reference image. Fig. 6(l) is the segmentation region of the infrared image with morphology method and Fig. 6(m) is the segmentation region of the visual image with morphology method. Fig. 6(n) is associated region map of infrared/ visible images. In the region image, that "person" was perfectly partitioned. The backgrounds such as road, building and so on are also well segmented. Fig. 6(o) is fused image with local-coloring method using Fig. 6(d), (f), (h) and (j) as reference images. Compared to Fig. 6(e), (g), (i) and (k), Fig. 6(o) has a clear color distinction between tree, person, building, pool and lawn. From Fig. 6(o) we can see that region-based coloring fusion method result can significantly improve observers' performance and reaction time.

Experiment 2
To evaluate the presented region-based coloring fusion method, we fuse the infrared and visible light images of the same scene with this algorithm, and compare the fusion image with the fusion images with global-coloring method. Fig. 7(a) is an infrared image, which presents the clear shapes such as trees, building, sky and some high-temperature objects; Fig. 7(b) is a visible light image, which provides more details than the infrared image. Besides this, it also shows some light sources. Fig. 7(c) is the fusion image using gradient pyramid method. Fig. 7(e), (g) and (i) are the fusion images with global-coloring method separately using Fig. 7(d), (f) and (h) as reference image. Fig. 7(j) is the segmentation region of the infrared image with clustering method and Fig. 7(k) is the segmentation region of the visual image with clustering method. Fig. 7(l) is fused image with local-coloring method using (d), (f) and (h) as reference color images. Compared to Fig. 7(e), (g) and (i), Fig. 7(l) has a clear color distinction between tree, building and sky. From Fig. 7(i) we can see that localcoloring method result can significantly improve observers' performance and the colors are more nature than global-coloring method result.

Conclusion
This chapter presents a gray image fusion method and a color image fusion method based on the region segmentation. The region-based fusion methods use the different feature regions of original image and compound the pixel level and feature level of image fusions. The effective way to separate target and background proves crucial for the quality of image fusion. The proposed method preserves the details of the LLL image and the legible target of the infrared image, therefore, the fused image enables the exact location of the target to be easily observed and provides all-around information for further processing tasks.
In the gray method, segmentation is firstly performed on the IR image and LLL images with top-bottom-hat filtering and the threshold method, consequently, the DT-CWT coefficients from the different regions are merged separately. Finally the fused image is obtained by performing inverse DT-CWT. This method keeps the approximate shift invariance and the limited redundancy. Region segmentation performs us to using different rules for each region of each level. Experimental results evidence this method which could provide better fusion than classical fusion methods in terms of objective fusion metric values such as entropy, average gradient, standard deviation, SSIM x y f (,, ) and ABF Q / In the color fusion method, segmentation is firstly performed on the IR image and LLL image with the morphologic method or the diffusion method. At the same time, the IR image and LLL image are mapped into the RGB space, and the gray fusion of the two images is conducted. Here, the color map rule and the gray fusion method are not very important. The false-color images usually have an unnatural color appearance, but a chance of region by region color transferring is given to ensure the fusion image similar to natural images. The fusion images are transformed into BR YC C space and the brightness is replaced by the gray fusion images. Experimental results evidence this method which could provide better sense of hierarchy than the global color fusion method in terms of subjective evaluation.