{'fraction_non_alphanumeric': 0.0382829115847042, 'fraction_numerical': 0.008637527670471809, 'mean_word_length': 4.542458503728651, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 0, 'https://': 1, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
The Information&Mutual Information Ratio for Counting Image Features and Their Matches

Feature extraction and description is an important topic of computer vision, as it is the starting point of a number of tasks such as image reconstruction, stitching, registration, and recognition among many others. In this paper, two new image features are proposed: the Information Ratio (IR) and the Mutual Information Ratio (MIR). The IR is a feature of a single image, while the MIR describes features common across two or more images.We begin by introducing the IR and the MIR and motivate these features in an information theoretical context as the ratio of the self-information of an intensity level over the information contained over the pixels of the same intensity. Notably, the relationship of the IR and MIR with the image entropy and mutual information, classic information measures, are discussed. Finally, the effectiveness of these features is tested through feature extraction over INRIA Copydays datasets and feature matching over the Oxfords Affine Covariant Regions. These numerical evaluations validate the relevance of the IR and MIR in practical computer vision tasks


I. INTRODUCTION
Advances in computer vision and image processing methods have relied on information theory as a powerful mathematical tool to determine the statistical variability of a set of images. Several saliency criteria have been proposed in the literature inspired by information theoretical concepts, such as the Kullback-Leibler divergence [1] and the Shannon entropy [2], [3]. In the following, we define two new image features motivated by information theoretical concepts: the Information Ratio (IR) and the Mutual Information Ratio (MIR). The IR is obtained as the self-information of a intensity of image over the information content of the pixels of the same intensity. The MIR is defined in a similar manner but with respect to a pair of intensity levels across two images. We show the effectiveness of these proposed image features through standard computer vision tasks: feature counting and feature matching.
Related results: In computer vision, local image features such as edges, corners and lines are extracted from an image to empower that mid-and high-level vision tasks, such as image registration, motion tracking, and 3-D reconstruction [4]. Generally speaking, image features are either handcrafted or obtained through deep learning.
Conventional handcrafted feature-extracting algorithms are FAST [5] and Oriented FAST and Rotated BRIEF (ORB) [6], Speeded Up Robust Features (SURF) [7], and KAZE (meaning "wind" in Japanese) [8]. Among deep-learning inspired feature-extracting algorithms, we mention FASTER [9] and V-FAST [10]. All the algorithms above produce specific descriptor and extract the different count of local features from the images. To evaluate the efficiency of the feature-extraction algorithms above, two criteria are usually considered: the count of extracted features and computational complexity [7], [8], [11]. Although the image features count is a helpful criterion for comparing two different algorithms, it singly and generally provides no guarantees on the quality of these features or the expected number of features extracted for different sets of images. Thus, it needs to estimate a reference value for the count before feature extracting. Similarly, the computational complexity is a relative criterion in determining the performance of a feature extraction algorithm. Accordingly, the selection and the configuration of a specific feature extraction algorithm on a specific image dataset is depended on the human decision and not a firm mathematical basis. Image features are often used for feature matching which is the procedure in which features are matched in two consecutive images, usually in gray-scale image or in one of the image channels. Feature matching, also depicted in Fig. 1, is used in advanced computer vision tasks, such as depth estimation, 3D reconstruction, and motion capture.
For image matching, that is the problem of determining object features in two different images of the same scene, the curve of "precision versus recall" is sometimes used to assess local descriptors and features. In this curve, the "precision" argument corresponds to the ratio of correct over total matches, while the "recall" argument to the portion of features that are matched between the original image and its transformation.
Another tool to evaluate feature matching algorithms for a given dataset is the average precision, as described in [11].
Contributions: In this paper, the Information Ratio (IR) and the Mutual Information Ratio (MIR) features are introduced as image features across a single and two (or more) images. We argue for the usefulness of the proposed image features through three sets of numerical evaluations: (i) by computing the IR features in two conventional image datasets: University of Oxford's Affine Covariant Regions [12] and INRIA Copydays [13], (ii) by evaluating the feature distance for first dataset to evaluate the feature matching performance using MIR features, and (iii) a pre-processing method based on the IR feature evaluated on the second dataset.
The lower bounds of the IR and the MIR as the functions of the entropy and the mutual information are also provided. This approximation is particularly useful to reduce the computational complexity in determining the features count.
From a high level perspective, our motivation in resorting to an information theoretical approach to the feature matching problem is in determining the fundamental answer to the following questions: (i) How many feature points exist in a given image, regardless of how features are described? and (ii) How many common features can be determined among two given images?
To answer these questions, we resort to a statistical formulation of the feature extraction problem and the feature matching problem which, we hope, will find general applicability.
Notation: In the remainder of the paper we use the following notation. We define [n : m] {n, n + 1, . . . , m − 1, m} ⊂ N. When n = 0, we use the simpler notation [m]. Random Variables (RVs) are indicated with capital letters, i.e. X. Random Vectors (RVs) are indicated with bold capital letters, i.e. X. The hat symbol is used to indicate the sample version and the symbol tilde for histogram versions of a random object. Logarithms are taken in any base.
The code for the numerical simulations in the paper can be found at the following web address: https://github.com/ AliKhajegiliM/IR-and-MIR.

A. System Model
In the standard RGB (Red, Green, and Blue), an image is described through a matrix of dimension N × M where each element is referred to as a pixel. Each pixel is itself a vector in [2 D − 1] 3 where D is the color depth of an image and each dimension represents one of the following colors: red, green, and blue which we indicate as R, G, and B, respectively. We assume that a set of L images X = {X 1 , . . . , 3 (that is pixels 2-D position plus 3D color vector). In the following, we refer to each X t as a frame. For a given X t , X t (C) let C ∈ {R, G, B} be the 2-D vector corresponding to the color C; we refer to X t (C) as the component image of the color C or the C channel of the image X t . Finally X t (C, n, m) for (n, m) ∈ [N ]×[M ] represents the intensity of the pixel (n, m) for the color C at frame t. Generally speaking, as the R, G, and B component images are correlated, the elements of the random vector are also correlated. This also holds for the pixels in the component images. In the following, we will be concerned with estimating the image histogram of the frame X t (C) with the distribution P X(C) : where 1 {C} is the indicator function of the condition C.
The sample self-information of the i th intensity level is then estimated as − log h i (C)/N M [nats] and the sample entropy as the In the following we refer to p i (C) as the sample probability mass function of X t (C).
The definitions in (1) and (2) can be extended to two sets of consecutive frames as follows Accordingly, the joint histogram of X 2 (C) with the distribution P X 2 (C) can be estimated analogously to (1) as follows The sample joint entropy of X 1 (C) and X 2 (C) is defined as and the sample mutual information as Remark. As defined by Shannon in [14], the entropy conceptually captures the information, or better variability, in a random source. Although there already various image processing algorithms have been inspired by information theoretical concepts, see [4], our approach is substantially different from the other approaches in the literature. We introduce the IR and the MIR as novel information measures for images and sets of images, respectively: these measures are substantially different from previously-investigated information measures.
Remark. Due to space limitations, a number of extensions to our analysis are not pursued here. For instance, (i) we don't consider the effect of correlation among the channels, (ii) the MIR for more than two images, and (iii) extensions to other image color models, such as the CMYK color model. Such extensions are left for future research.

III. THE INFORMATION RATIO (IR) AND THE MUTUAL INFORMATION RATIO (MIR) FEATURES
In this section we define two new image features based on the quantities introduced in Sec. II-A. Let us begin with the IR feature. Note that we haveh i (C) pixels with selfinformation − log(p i (C)), so that we have a cumulative infor- On the other hand, the chance of selecting each of theseh i (C) points under a uniform distribution is 1/h i (C) with a self-information log(h i (C)) [nats]. Using the definition of the random vectors, the ratio between the cumulative information and the selfinformation under uniform distribution is We refer to (7) as information ratio of the intensity level i ∈ [2 D − 1]. Next, we define the expected value of (7) as the average information ratio over the channel pixels, that is In the following, we use the histogram version of the quantity in (8) as an image feature. let ample version of (7) be so that the histogram version of (8) is We term the quantity in (10) as the Information Ratio (IR) image feature. The MIR feature is defined in an analogous way as the IR image feature in (10) but across two (or more images). First, we define the self-information ratio of two intensity level i an j in the frames X 1 (C) and X 2 (C) as for the images X 1 and X 2 as in (3). The definition in (11) is intuitively motivated as follows. Consider two images X 1 and X 2 , the self mutual information of each pair (i, j) of intensities in two image channels is log(p i,j (C)) − log(p i (C)) − log(p j (C)). Also,h i,j (C) pixels with intensity i in the first image are "matched" with pixels of intensity j in the second image. What is the probability of the random selection of involved pixels in the intensity pair (i,j) as a matched pixels? This probability of this event is 1/h i,j (C). This means that the self-information in this random selection is log(h i,j (C)). Accordingly, the ratio in (7) for the pair of intensities (i, j) in two images is as in (11). The average mutual information ratio between two images X 1 and X 2 over all pixels is defined as The histogram version of the average mutual information ratio in (13) is defined analogously to (10) as We define the quantity in (13) as the Mutual Information Ratio (MIR) image feature. From a high-level perspective, the IR in (10) captures the information contained in a singular pixel of intensity i as compared to all other pixels with the same intensity. A further interpretation of the IR is as follows: the image histogram is not a sufficient statistic of the original image, as the position information is not preserved. The uncertainty on the intensity of a pixel in the image is related to the abundance of pixel of that intensity i. Similarly, the MIR in (13) provides a estimate of the information contained in a pair of the pixels of intensity (i, j) positioned in the two images. The next theorems yield bounds on the IR and MIR for a given image channel C on one frame and two.

A. Some useful inequalities
The next inequalities are presented without proof, due to space constraints. These inequalities are meant to provide useful bounds between the IR and the entropy, and between the MIR and the mutual information.
Theorem III.1. A lower bound on r(C, X) is obtained as a function of H(C) as This lower bound is termed as LIR.
The result in Th. III.1 shows that the image entropy, as defined in (2), is a lower bound to the IR image feature. Note that the quantity in (2) is consummately referred to as image entropy and is a common measure of variability of an image in computer vision.
Theorem III.2. A lower bound on m(C, X 1 ; X 2 ) is obtained as a function of I(C, X 1 ; X 2 ) as This lower bound is termed as LMIR.
Note that both the IR and the MIR are positive defined. The sample mutual information in (6), as the sample entropy in (5), is a commonly-used measure in computer vision to quantify the similarity among two images.
Generally, customary cameras, the resolution of daily captured images, and color mapping technologies satisfy the condition in Lem. III.3.
Corollary III.1. The following inequalities hold for t ∈ {1, 2} and where r(C, X t ) is the IR of the channels C for the first and second images.
Cor. III.1 shows how the IR feature is upper bounded by a function of the joint histogram of the two images.
The results in Th. III.1, Th. III.2, Lem. III.3 and Cor. III.1 are presented here to highlight some useful properties of the IR and the MIR image feature. In particular, these results highlight the IR and the MIR can be approximated.

IV. NUMERICAL EXPERIMENTS
In this section we present three sets of numerical evaluations to validate the use of IR and the MIR as image features. The IR feature is used for feature points count in a single image, while the MIR is used for feature matching across two images.
Experiment Setup: For the numerical evaluations, a computer system with Processor Intel(R) Core(TM) i5-6200U CPU @ 2.30GHz, 2.40GHz, 8.00 GB RAM is used. MATLAB software is used for the experiments, in particular the MAT-LAB computer vision toolbox implementations of the ORB, SURF(64D), and KAZE is used for extracting the image feature points, and matchFeatures function used for matching them.
Datasets: We evaluate the IR and the MIR image feature performance over two conventional image datasets: the University of Oxford's Affine Covariant Regions [12] and INRIA copydays [13]. In the first dataset, there exist six type of images with six images for each type: for evaluating specific purposes, Fig. 2 provides some illustrative examples. The second dataset is comprised of 157 random images from different scenes, places, and creatures.
Overview: ORB, SURF, and KAZE algorithms are applied to the datasets to obtain the per-frame(channel) features and matched features of two consecutive frames(channels), respectively. Simulations are performed over all three channels of an image, i.e. C ∈ {R, G, B}. For brevity, only section the results of Red channel are reported here. The performance in the remaining two channels substantially confirm our conclusions. Evaluations are repeated for multiple images in the same category in the dataset.
Image Feature distance: The KAZE, SURF, and ORB extract general local image features. Generally speaking, a local image feature is a neighborhood, containing some pixels of the same intensity level, which identifies salient points such as edges, lines, corners, textures, and so forth. The images in the datasets have 256-level pixels: to avoid identifying the same image feature multiple times, we consider two features to be valid when they have minimum distance greater than a chosen threshold d. This is equivalent of reducing the histogram levels by merging contiguous intensity levels. For instance, sub-sampling the intensity levels to 128 values coincides to a minimum feature distance of d = 2. Note that, in human perception, the people's eyes are sensitive to the different feature distances, according to the individual visual psychology [15]. In the remainder of the section, results are listed for five values of the feature distance: d ∈ {1, 8}.
IR feature evaluation: We evaluate the IR image measure by obtaining the IR feature count with varying levels of image brightness. To do so, we consider a varying brightness level K, multiply the intensity over each channel by the intensity level K and obtain the IR feature count for the chosen level of K. This experiment is performed on both datasets: a part of the results is presented in Table I. The results are also presented in Fig. 3   MIR feature evaluation: For the MIR feature, we compare the matched points in two successive frames to within a desired feature distance d as compared to KAZE, SURF, and ORB. To verify the predictive value of the lower bound in Th. III.2, we also evaluate this lower bound, a part of results is reported in Table II  IR based optimization method: This experiment which is devised to analyze the IR applicability, shows that the maximum count of the extracted features is not necessarily for K = 1, it is showed in Fig.3b. This means that there is a Fig. 2: Oxford's Affine Covariant Regions: from left to right, bark (512 × 765) for rotation, bikes (700 × 1000) for blur scene, graffiti (640 × 800) for orientation and field of view variation, Leuven (600 × 900) for intensity variation, trees (700 × 1000) for blur scene, and UBC (640 × 800) for JPEG compression.
(a) The IR feature count and its lower bound(LIR) as a function of the brightness level K. This shows the dependency between feature distance and the peak of feature counts curve.
(b) The KAZE, SURF, and ORB feature count as a function of the brightness level K.   m ← Compute the input channel(X(C)) mean 4: For K ∈ [0.9, 255 m ], step = 0.1: End If 10: End For

11:
Use K Optimizer × X(C) instead of X(C)

V. DISCUSSION
The image channel histogram, on a first approximation, is invariant to orientation and rotation, it is also somewhat robust to variation in intensity, quality and field of view. For this reason, the IR and the MIR image features also possess these characteristics, thus making these image features particularly attractive for practical applications. In the concept of scale variation, it is worth noting that the count of feature points is depended on the scale of image. In this study, the devised manners are used for consecutive frames with the same size, but for frames with different sizes a helpful way is upsampling or down-sampling method to make them the same in size. The IR and the MIR image features, are fundamentally linked to the image entropy and image mutual information which are fundamental measures of variability and dependence among images, respectively. It is worth nothing that in image processing, the entropy and the mutual information are global, rather than local features. Accordingly, the lower bounds in Th. III.1 and Th. III.2 are rather useful lower bounds to the the IR and the MIR performance. As such, these bounds can be used to predict this performance before calculation, which can be rather intensive. In Fig.3, it is shown that feature extraction algorithms follow the same curve as the IR feature. This, intuitively, expresses that with varying K, the features in an image change and the IR appropriately captures this variation. The curves related to d = 8 reveal this fact that the best coefficient K is not one as the algorithms show in Fig.3b. Based on the reported results in Table I, it is clear that the other feature matching in the literature, i.e. KAZE, SURF, and ORB, can extract an amount of features lower than the IR and LIR features. Since the goal in many computer vision tasks uses features as a starting point, having a larger amount of starting feature points can potentially yield a better result. For this reason, we conclude that the large number of the IR feature can potentially enable better performance, although this claim requires further investigation. Feature matching is indeed among these higher-level computer vision tasks. Let us next discuss the result as reported in Table II. Only the ORB algorithm and for d ≥ 8, matches more features than the LMIR feature and is close to the IR. In almost other distances the MIR yield more matches that the KAZE and ORB. Also note how closely the LMIR predicted the matches obtained through the MIR features. By comparing the Table II and Table  I, we note that the count of matching MIR feature is much lower than the number of total IR features. The effectiveness of the IR feature is shown computationally in Table III, the count of extracted features through the KAZE, SURF , and ORB algorithms are maximized by finding the corresponding K Optimizer based on the IR feature in a flash of second.
In conclusion, we argue that the results presented here show that the IR and the MIR features have the potential improving the preformance of a number of computer vision. Future research direction will focus in determining the computational complexity of relevant algorithms operating on larger dataset. Although present algorithms, such as the KAZE, SURF, and ORB, cannot produce as many features as the IR and the MIR features, further research is necessary to better determine the potential of these novel features.

VI. CONCLUSION
Image feature extraction and matching is a prerequisite to high-level computer vision tasks. For this reason, obtaining a large set of features and matches leads to more effective algorithms. In this paper, two new image features are proposed: the Information Ratio (IR) and the Mutual Information Ratio (MIR) features. The IR feature is used to produce a large count of local image feature points. The MIR feature is used to match local feature across two (or more) consecutive frames. We show that the sample entropy and the sample mutual information are lower bound on IR and MIR, respectively. Therefore, the optimization methods which need more image information, i.e. exactly image feature points, can use IR and MIR instead of the entropy and mutual information as objective functions. One method to maximize the detectable features based on the IR feature is devised which it outperforms significantly applied feature extraction algorithms. We also compare the IR and the MIR features with other image features proposed in the literature: the ORB, KAZE, and SURF features. More precisely, we compare the IR with these features in two tasks: total features per image and feature matching across frames in two datasets. These numerical evaluations are only a partial characterization of the potential of these two measures. Further research is necessary to more thoroughly characterize the task better suited for the IR and the MIR image features.