Combine Target Extraction and Enhancement Methods to Fuse Infrared and LLL Images

For getting the useful object information from infrared image and mining more detail of low light level (LLL) image, we propose a new fusion method based on segmentation and enhancement methods in the paper. First, using 2D maximum entropy method to segment the original infrared image for extracting infrared target, enhancing original LLL image by Zadeh transform for mining more detail information, on the basis of the segmented map to fuse the enhanced LLL image and original infrared image. Then, original infrared image, the enhanced LLL image and the first fused image are used to realize fusion in nonsubsampled contourlet transform (NSCT) domain, we get the second fused image. By contrast of experiments, the fused image of the second fused method’s visual effect is better than other methods’ from the literature. Finally, Objective evaluation is used to evaluate the fused images’ quality, its results also show that the proposed method can pop target information, improve fused image’s resolution and contrast.


Introduction
Image fusion namely uses redundant data and complementary information from multisensors for obtaining a image which has accurate target, owns good visual effects [1].As an important branch of image fusion, infrared and low light level (LLL) images fusion's main task is to achieve the reasonable and comprehensive description of target and scene, on the condition to retain the original data information as much as possible and avoid false information.At present, the technology has been widely used in intelligent transportation, safety monitoring, human visual auxiliary fields and so on [2]- [3].
A kind of sensors has its own feature, so they can capture some part information of the scene, so we combines the infrared and LLL sensors to describe the whole scene better.In order to better combine imaging advantages of these two sensors, many scholars have done a lot research, and put forward a series of fusion methods, including different kinds of pyramid fusion methods [5], wavelet transform methods [6]- [7], curvelet transform methods [8], the contourlet tansform methods [9], the non-subsampled contourlet transform methods [10], shearlet transform methods and so on [11].All these methods bases on multi-scale decomposition approach, first, original images are decomposed into low frequency coefficients and high frequency coefficients, then, different fusion rules are used to process the low frequency coefficients and the high frequency coefficients, respectively.All these methods can achieve a good fusion visual effects, but some shortcomings in preserving original information from the original images, especially, for the insufficient sunshine or targets concealed and so on, easy to lead targets lose or unobvious, so we can not easy to understand the scene.Therefore, in recent years, the scholars had put forward some other fusion methods [12]- [15], for extracting the target or mining more depth detail better.Literature [12] combined compressed sensing principle to fuse images, the fused image could decrease fusion time and had a better visual effects, but its target highlights unobviously.Li Shutao et al. [13] proposed a method which based on average filter to decompose the original images into base and detail layer, then, used guided filter structure weight map to fuse the original images according to the weight graph.This method quickly realizes fusion, it also had better performance in detail, but the contrast of the fused image was poor.Literature [14] introduced local histogram equalization to enhance both the infrared and LLL images, then denoised with the median filter, this method fused images fast and the fused images had clear detail, but its target couldn't show obviously.Xing Suxia et al proposed a target extraction method [15] which based on Renyi entropy to segment the infrared image for extracting thermal targets, and enhance the LLL image's detail in NSCT domain.The fused image had a high brightness, so lost its detail information at the same time.
All those methods above can achieve complementary and redundancy between different images' information, contrast or resolution has been improved to a certain extent, but contrast and definition can't compatible.In order to enhance the contrast and improve the resolution of the fused image, the paper presents a new method which segments the original infrared image and enhance LLL image, then to fuse the segmented image and the enhanced image, the fused image is called the first fused image.Aiming to prevent the incomplete image segmentation and over enhancement LLL image effectively, the paper uses the first fused image and the original infrared and the enhanced LLL image to fuse for more original image information, the fused image is called the second fused image.
The rest of the paper is organized as follows: In Section 2, it mains introduce pretreatment, infrared image segmentation and LLL image enhancement method are described.In Section 3, fusion strategy is discussed.In Section 4, experiments results and analysis are put.Finally, in Section 5, conclusions of the work are made.

Pretreatment
How to reserve important information of the scene better has been the main direction of our research.For this purpose, the paper uses the segmentation method on infrared image to extract the major target information, in addition, also apply a certain enhancement method to excavate more details deeply.Based on the analysis above, we put forward an idea of fusion treatment to fuse infrared and LLL images.We discuss the detail as follows.

2D maximum entropy
Thermal target information is important information for the infrared image, we uses the 2D maximum entropy threshold [16] to segment the infrared image, for extracting target information to better popping the infrared target.
Due to probability distribution of the segmented infrared image's target region and background region are different, so we proposes posterior probability of the gray and mean of gray region to normalize each region's occur probability ij p .Suppose the segmentation value of the image is   , s t , background region's probability and target region's probability are B p and O p , respectively.

607
The 2D entropy of background region: Where, The discriminant function of image entropy is defined as: Then, the optimal threshold vector   * * , s t must satisfy the following condition: Shown in Figure 1, demonstrated the segmentation effect, Figure 1(b) shows the whole and complete person target and some hot regions.

Zadeh transform
A LLL Image contains more detail information, its scene is dark and contrast is unobvious.In order to mine more details information for the fused image, we uses the enhancement method to intensify the LLL image.
The traditional methods including histogram equalization, gray stretching and so on, these methods can not process dynamic effect well and enhance the image noise more, we introduce the Zadeh transform theory and principle for image information mined method presented from our lab in literature [17] to intensify the original LLL image.
The image's uncertainty property is the reason for image blur processing.The image enhancement algorithm is designed in consideration of human's subjective sense.If this method can combine some visual characteristics of the human, image and video quality visual effect will be significantly increased.Here, the underlying image mining is to obtain the highest possible contrast resolution, so the original Zadeh transform [0,1] interval extended to [0,255] to define the spatial domain, which constitutes the Zadeh transform enhancement method, the paper called the method Zadeh transform.In order to mine more image information of the LLL image, the paper sets 255 represents the coefficient of the space expansion.When k is greater than the limit of human vision contrast resolution, Eq. (10) shows the performance of the underlying image mining functions; when k is less than the limit of human vision contrast resolution, Eq.( 10) achieves the image hidden features.
1   , Eq.( 10) is to achieve image binarization function. and  are mining parameters.
After Zadeh transform, Eq.( 11) may occur   , 25 5 T x y  which beyond the domain, this will cause confusion.Therefore, this need sets some constraints, namely, while the gray value is over 255, the value is set to 255; while the gray value is low 0, the value is set to 0.
We illustrate the important of the enhancement method as follows, one pair for displaying the gray image, shown as Figure 2, Figure 2 Using the absolute value of the high frequency sub-band NSCT coefficients take greatest shown as Eq.( 13): After a series process above, we get fused image (F2 x y }, finally, the NSCT inverse transform, we get the fused image F2.

Experiment results and analysis
The experiments based on the MATLAB platform, infrared and LLL image of the two groups from Holland TNO institute's picture library have been registration completely.Figures 3(a)-(b) as the first group shown, Figure 3(a) can identify a person clearly, Figure 3(b) can see road, slope, shrub and fence and other scenery, but its contrast feels not well; shown in Figures 4(a)-(b) as the second group, Figure 4(a) can identify persons, ship and other small targets, Figure 4(b) can see the ship, the sky clearly.

The experimental results and subjective evaluation
For the purpose of comfirming the correctness and effectiveness of the proposed algorithm, we uses the other four methods compared with the first fused image and the second fused image to test.Figures 3(c) and Figures 4(c) are fused based on the bior97 wavelet transform method, the source code is publically available (http://www.metpix.de/toolbox.htm).Figures 3(d 4(c)-(d) both choose the fusion rules by averaging the low frequency coefficients and getting the maximum absolute values of high frequency coefficients, their decomposition level all are 4 layers, called wavelet method and NSCT method, respectively; Figure 3(e) and Figure 4(e) are fused by the method in literature [15], its source code is publically available (http://www.xudongkang.weekly.com)called as GF; Figure 3(f) and Figure 4(f) are fused by the method in literature [15], called as Renyi entropy method; Figure 3(g) and Figure 4(g) and are the fused images by the methods which fuses the segmented infrared image and the enhanced image, as the first fused image; based on the first fused image, we fuse the first fused image and the original infrared image and enhanced LLL image with the second fused method, its visual effect shown as Figure 3(h) and Figure 4(h).Four decomposition levels with 2,8,8,16 directions from coarse scale to finer scale are adopted by the methods which uses the NSCT method.
Subjective effects are analyzed as follows.Group one's effection shown in Figure 3, Figure 3(c)'s fence seems very fussy, its contrast feels not clear and detail lost seriously; we can see clear detail from Figure 3(d), but its contrast is very poor, this state may hide some important information; Figure 3(e)'s contrast is so low that we can hard identify some important information; Figure 3(f)'s contrast seems very obviously, and we can see obvious target, due to its over enhanced, its small detail information does not very clear, the whole visual seems not very good; Figure 3(g) has good definition, its target seems obviously, its quality is better than the former methods; when compared with the Figure 3(h), it has less clear detail; so through its comparing, Figure 3(h)'s visual effect is the best.The other pair can be analyzed in the same way.

Objective evaluation
In order to further assess the fusion performance of different methods objectively.Three fusion quality metrics are applied, i.e., average gradient (AG) [18], spatial frequency (SF) [19], the edge retention metrics ( AB F Q ) [13] are adopted in the paper.All the evaluation metrics used in the paper are defined as follows: 1) Average gradient(AG) Improvement of an image's quality can be expressed by the average gradient，which reflects the clarity of the image, reflecting the small details contrast in the image and texture variation, the greater the average gradient, the image has better integration, defined as follows: Colum frequency of an image is defined as follows: where, M and N denote the number of rows and columns in the image.The spatial frequency of the image is defined as follows: Spatial frequency reflects the overall level activity of an image in the spatial domain.The greater the value, the visual effection is better.
3) AB F Q The gradient based index Q evaluates the effective keep edge information which transferred from the source images to the fused image.It is calculated as follows: where Tabel 1 shows that the proposed method has the greatest average gradient value, the greatest spatial frequency value and the greatest edge retention value, it shows that the proposed method has the best visual effect, this consistent with the subjective evaluation results.

Conclusions
On the basis of original infrared and LLL image containing different feature information, we uses segmentation method to naturally extract heat target information in infrared image and mines more depth detail from LLL image by enhancement method.According to the segmented image, we fuse the enhanced LLL image and the original infrared image.In order to optimize  ISSN: 1693-6930 TELKOMNIKA Vol. 12, No. 3, September 2014: 605 -612 612 incomplete and layering due to the incomplete segmentation or over enhancement in the first fused image, the paper uses the second fused method to fuse the original infrared image , enhanced LLL image and the first fused image in NSCT domain, so the fused image can better keep the main target information of the infrared image and mine more detail from LLL image, and also has a better visual effect.The experimental results show that the proposed second fused image can improve the resolution and contrast of the image and identify targets better compared with other methods.

Figure 1 .
Figure 1.The chart of segmentation effection

TELKOMNIKA
Vol. 12, No. 3, September 2014: 605 -612 608 The concrete description is as follows： represent the location   , x y of the target and original image, respectively.point and goal of converting the gray level image, respectively.

Figure 2 .
Figure 2. Gray image test ) and Figures 4(d) are the fused images which are fused by the NSCT method.Figures 3(c)-(d), Figures

Figure 3 .Figure 4 .
Figure 3.The first group experiments of infrared and LLL image fusion

, 2 )
yf x y  represent pixels in the x and y directions' first-order variance, respectively.TELKOMNIKA ISSN: 1693-6930  Combine Target Extracion and Enhancement Methods to Fuse Infrared …. (Yong Chen)611 Spatial frequency(SF) Row frequency of an image is defined as follows:

Q
are defined as edge strength and orientation preservation values at location   , i j , respectively.N and M are as the width and height of the images.BF  Q is similar to AF Q .