{'fraction_non_alphanumeric': 0.03696145124716553, 'fraction_numerical': 0.014159737969261779, 'mean_word_length': 4.888872403560831, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 0, 'https://': 8, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 3, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
General Purpose (GenP) Bioimage Ensemble of Handcrafted and Learned Features with Data Augmentation

Bioimage classification plays a crucial role in many biological problems. In this work, we present a new General Purpose (GenP) ensemble that boosts performance by combining local features, dense sampling features, and deep learning approaches. First, we introduce three new methods for data augmentation based on PCA/DCT; second, we show that different data augmentation approaches can boost the performance of an ensemble of CNNs; and, finally, we propose a set of handcrafted/learned descriptors that are highly generalizable. Each handcrafted descriptor is used to train a different Support Vector Machine (SVM), and the different SVMs are combined with the ensemble of CNNs. Our method is evaluated on a diverse set of bioimage classification problems. Results demonstrate that the proposed GenP bioimage ensemble obtains state-of-the-art performance without any ad-hoc dataset tuning of parameters (thus avoiding the risk of overfitting/overtraining).


Background and Driving Forces
Biomedical research is increasingly dependent on computer vision and machine learning in the discovery of new knowledge and methods of diagnosis. Storing, retrieving, and analyzing large biological images has become critical, in part because of the enormous amounts of data generated by recent advances in imaging technologies. Automated image analysis has become an indispensable tool for handling the throve of data being collected for a variety of applications such as cell phenotype recognition, subcellular localization, and histopathological classification [1][2][3][4].
Because computer vision and image classification rely on methods for extracting highly discriminative feature sets, a significant area of research in these domains has focused on generating ever better methods for extracting robust descriptors. Until recently, however, most bioimage research has concentrated on the problem of segmentation [5] with little attention devoted to investigating the discriminative power of texture descriptors-even though it has been shown that extracting highly discriminative texture descriptors can circumvent the problem of segmentation [5][6][7]. Some whole image methods of note that have been proposed in the literature include [8][9][10], and some popular descriptors used in automatic bioimage classification include traditional Gabor filters [11] and Haralick's famous texture features [12].
More recent descriptors applied to automatic bioimage analysis include such powerful handcrafted descriptors as the scale-invariant feature transform (SIFT) and local binary patterns (LBP) [13][14][15]. Handcrafted descriptors are designed by researchers to extract specific image characteristics and is typically accomplished as follows: characteristic regions of an image are located by a key point detector, and these regions are described by a vector of measurements (the descriptor) that depends on the specific image characteristics under consideration. The extracted set of descriptors is then used to train a classifier, such as the Support Vector Machine (SVM) [16].
In contrast to handcrafted descriptors are learned descriptors, which are automatically learned by a classifier system. Learned descriptors have only recently been explored in bioimage classification. In [17], for instance, an automatic feature discovery method was proposed using class-specific dictionaries for the diagnosis of ovarian carcinomas, and [18] handcrafted and learned descriptors were combined to discriminate irregularities in brain cells; this system joined an unsupervised feature learning method with learned linear combinations of Riesz wavelets.
Within the last few years, some innovative learned approaches have been proposed that exploit deep learners [19], esp. the Convolutional Neural Network (CNN). It appears that deep learners analyze input images via the different layers in the architecture by evaluating sets of features learned directly from observations of the training images [20]. Some layers are even thought to preprocess images using a pyramidal approach [21]. When deep neural networks, such as CNN, are trained on a set of images for a specific classification problem, features extracted by the shallowest layers (close to the classification layer) strongly depend on the training set, while features taken from the first layers resemble Gabor filters or color blobs that are transferable to many other classification problems [22]. This discovery has been exploited by bioimage researchers who have used CNN or ensembles of CNNs as feature extractors in this domain [23,24]; the resulting learned features are then treated like SIFT and LBP and become the input to traditional classifiers, such as SVM.
It is well known that when it comes to deep architecture, large image datasets are needed. The requirement of large datasets is difficult to meet in some fields: this is certainly the case in bioimaging where images are expensive to acquire and patient availability is limited. In such contexts, data augmentation is a popular method for increasing images in small datasets: indeed, such techniques have been used extensively in the analysis of medical and biological images [25]. Data augmentation also prevents overfitting CNNs and is known to improve performance. A good example of this approach in bioimage classification can be found in [26], where the researchers were able to accurately detect breast cancer in a set of histology images containing less than 100 images per class by combining pre-trained deep network architectures with multiple augmentation techniques.
The most common methods of image data augmentation involve reflection, translation, and rotation [27][28][29][30], as these augmentations generate different representations of the same sample. Different representations of a given image can also be constructed by altering the contrast, saturation, and brightness [27,29,30] of images. Yet another common technique is PCA jittering, which accentuates the most relevant features of an image by adding to it some of its principal components multiplied by a small number [27,29]. Most deep learning frameworks make use of a limited set of basic image transforms. Recently, however, libraries of fast augmentation methods have been developed, such as Albumentations [31], that provide large sets of image transforms with easy-to-use wrappers around other augmentation libraries.
Specific problem-dependent augmentation methods can also be applied to expand small datasets. For example, [32], the authors replicated speckle noise, a common artifact in synthetic aperture radar images, by applying random pointwise multiplications to images, and [33], different stretchings of the human body were reproduced by creating elastic deformations of breast cancer images. Operations like elastic transforms and grid distortions are useful in medical imaging, where non-rigid structures that have shape variations are quite common [31].
In [34] an ensemble of networks for histopathological biopsy was recently proposed where the authors successfully employed multiple data augmentation techniques to improve their accuracy. Their augmentation protocol included horizontal and vertical flips, contrast enhancement, zoom, shear, and rotations. In [35] color data augmentation for the segmentation of histological images was also proposed that combined the color structure of an image with the appearance of another one.
Another method for enlarging small datasets uses Generative Adversarial Networks (GANs) to synthesize new images that are different from those contained in the original dataset [36][37][38][39]. GANs are based on an adversarial game between two neural networks: a generator network that produces synthetic samples given a perturbation and a discriminator network that distinguishes between the generator's synthesized image and the true image. Because GANs generate new images on a separately trained network, they produce, unlike data augmentation techniques, a unique yet relevant set of new images. In [40] a new GAN augmentation was proposed the output of which was smooth thanks to a spatial constraint. The authors in [40] also managed to create synthetic annotations for microscopy segmentation.

Research Goals and Contributions
In this work, we present a new General Purpose (GenP) bioimage classification method that combines both handcrafted and learned descriptors. Ideally, GenP image classification systems can handle a broad range of different image classification tasks within a specific domain (such as medical imagery) with little (if any) parameter tuning. A GenP system should also perform competitively against other systems that are optimized for task-specific image classification problems.
As illustrated in Figure 1, our latest bioimage GenP system combines handcrafted features and deep learning methods to obtain a high degree of generalizability across a range of bioimage datasets. As in [41], the system proposed here takes a representative set of robust handcrafted descriptors and individually trains them on separate SVMs, with the set of SVMs combined by sum rule. Similarly, we also generate a high-performing ensemble of CNNs that is then combined with the SVM ensemble for a final decision.
Unlike [41], however, the focus here is not on the combination of handcrafted features and deep learning methods but rather on the implementation of powerful data augmentation approaches to strengthen the ensemble of CNNs. In this paper, different CNN topologies are trained using two different learning rates {0.001, 0.0001} [42], four different batch sizes {10, 30, 50, 70}, and multiple combinations of different data augmentation approaches, with two of them being proposed here for the first time. The new data augmentation approaches proposed here are based on two wellknown feature transforms: the Discrete Cosine Transform (DCT) and Principal Component Analysis (PCA). Both methods are based on projecting the original image onto the DCT/PCA subspace, which perturbates the retro-projection from the subspace to the original space. Data augmentation via PCA has proposed in [43] and [44]. In [43], PCA is applied by perturbing the training images using extracted eigenvalues. In [44], the first principal component of each sample is multiplied by a random value drawn from a uniform distribution, then the images are back projected to the original space. In this work, we propose three different methods based on PCA.
It should also be noted that in [45] an ensemble of classifiers was proposed where the same network architecture was also repeated several times, with each repetition having different activation functions in the activation layers. In other words, each activation layer of each network was randomly substituted with a random activation selected from a pool. Figure 1. Schematic of the proposed ensemble system. The data augmentation step is performed only in the training step. In the test step, we use the fine-tuned CNNs.
To prove that our system is highly generalizable, we evaluate our method on a diverse set of bioimage classification problems, each represented by a benchmark dataset. Some of these datasets are publicly available in the IICBU 2008 database, and each bioimage task represents a typical subcellular, cellular, and tissue level classification problem. Results show that the proposed GenP bioimage ensemble obtains state-of-the-art performance without any ad hoc datasetdependent tuning of parameters.

Organization of this Work
This paper is organized as follows. The set of handcrafted methods evaluated in this work are listed in section 2. The CNN architectures used to build the ensemble of deep learners are presented in section 3. The new augmentations methods and stochastic ensembles are detailed in sections 3.1 and 3.2, respectively. Our experiments are described in section 4 along with a discussion of experimental results, and the study concludes in section 5 with suggestions for further research.

Handcrafted Methods
As noted above, our GenP bioimage ensemble combines both handcrafted and learned descriptors with some traditional and novel augmentation methods. The handcrafted descriptors considered in this work are described in Table  1, where we also provide the parameter settings used for each descriptor. In subsections 2.1-2.11, a detailed discussion of all these descriptors (except for MOR which is described in Table 1) is provided. Color images are processed by evaluating descriptors on the three channels separately and training an SVM for each one; the SVMs are then combined by sum rule as noted in Figure 1. The hyperparameters of the SVM are chosen using the training data. where the image is subdivided into four quadrants. [51] COL A simple and compact set of soft color descriptors such as the mean and standard deviation of the color space (RGB, HSV, and CIE Lab). [52] AHP Adaptive Hybrid Pattern combines i) a Hybrid Texture Model (HTD) composed of local primitive features and a global spatial structure, and ii) an adaptive quantization algorithm (AQA) to improve noise robustness. We fixed quantization level = 5; we used 2 (radius, neighboring points) configurations: (1,8) and (2,16). [53] FBSIF Extension of the canonical Binarized Statistical Image Features (BSIF) by varying the parameters of filter size (SIZE_BSIF, size ∈ {3, 5, 7, 9, 11}) and a threshold (th) for binarizing (FULL_BSIF, th ∈ {-9, -6, -3, 0, 3, 6, 9}). [54] LET A simple but effective representation, LET encodes the joint information within an image across feature and scale spaces. We use the default values available in the MATLAB toolbox. [55] CLM CodebookLess Model. We use the ensemble called CLoVo_3 in [41] based on e-SFT, PCA for dimensionality reduction, and one-vs-all SVM for the training phase. [56]

ETAS
We utilized Threshold Adjacency Statistics from a novel perspective to enhance discrimination power and efficiency. Seven threshold ranges were used to produce seven distinct feature spaces, which were then used to train a single SVM. The source code for this method is located here: https://drive.google.com/file/d/0B7IyGPObWbSqRTRMcXI2bG5CZWs/view. [57] MOR Morphological descriptor is a set of measures (aspect ratio, number of objects, area, perimeter, eccentricity, etc.) extracted from a segmented version of the image as enumerated in [58]. [58]

Local Ternary Pattern (LTP)
LTP [47] extends LBP [46], which can be expressed as where is the pixel under consideration (the central point), and qp is a set of points surrounding . This neighborhood is bounded by the radius R surrounding and the total number of pixels P included in the neighborhood. The variable = − , or the difference between the intensity levels of qp and qc. The function s(x) produces the binary encoding: Each digit of LBP is either 0 or 1, with codes ranging in [0, 2 P -1]. LBP descriptors are the histograms of these binary numbers.
The LTP extension of s(x) in [47] provides a ternary encoding using a threshold τ around zero: The resulting ternary codes are much larger than LBP. For this reason, LTP histograms are split into binary subhistograms that are concatenated: each LTP code is divided into a positive and negative binary pattern according to the sign of its components.

Completed Local Binary Pattern (CLBP)
CLBP [49] is so named because this descriptor represents a local region using the region's center pixel (CLBP-C) and the Local Difference Sign-Magnitude Transform (LMSMT). This transform calculates two components: the difference signs, called CLBP-Sign (CLBP_S), and the difference magnitudes, called CLBP-Magnitude (CLBP_M). CLBP is the histogram formed by the three binary representations of CLBP_C, CLBP_S, and CLBP_M.
More formally, given a central pixel along with its P neighbors and letting = − , then can be decomposed into the two LMSMT components, CLBP_S and CLBP_M, as follows: where Sp represents the sign of dp, and mp the magnitude of dp. As can be observed, LDSMT transforms the vector CLBP_S can now be defined similarly to LBP as in Eq. (1): where c is the mean value of m.

Rotation Invariant Co-occurrence among adjacent Local Binary Pattern (RIC)
The main idea of RIC [50] is simple. LBP fails to preserve structural information in the binary patterns. To retain this information, RIC represents using LBP pairs. Letting I be an image and = ( , ) a position vector in I, LBP can also be expressed as where P is the number of neighbor pixels and Δ is the displacement vector from the center pixel to the neighboring pixels given by Δ = ( cos�θ � , sin (θ )), where θ = 2 ⁄ and ( ) is the scaled parameter of LBP in Eq.(2).
The co-occurrence among LBPs over an image can be converted into LBP pairs at a given pixel thus: where Δ = ( cos θ , sinθ) is a displacement vector between an LBP pair, with an interval between an LBP pair, and θ = 0, 4 ⁄ , 2 ⁄ , 3 4 ⁄ . Rotation invariance can be incorporated by assigning a rotation invariant label to each LBP pair equal in rotation.

Gaussians of Local Descriptors (GOLD)
GOLD [51] improves the canonical Bag of Words (BoW), which extracts local features in order to create a codebook from which a global image representation can be produced. GOLD is a four-step process: 1. Dense SIFT descriptors are extracted on a regular grid of the input image.
2. The image is decomposed into regions by a multilevel recursive image decomposition and softly assigned to regions according to a local weighting. 3. Each region is represented as a multivariate Gaussian distribution of the extracted local descriptors via a local mean and covariance inference. 4. The covariance matrix is projected on the tangent space and concatenated to the mean to obtain the final region descriptor. Descriptors are vectorized by projecting the covariance matrix on a Euclidean space and concatenating the mean vector.

Color Descriptors (COL)
To produce the COL [52] descriptors,The following soft descriptors of the color space (RGB, HSV, and CIE Lab) are computed in terms of the mean, standard deviation, and k-th moment, defined as • k-th moment: where n is the number of pixels in the image, , ∈ [0,1] is the intensity of the i-th pixel in the c-th color channel, and is the average intensity value of the c-th color channel.

Adaptive Hybrid Pattern (AHP)
AHP [53] uses a Hybrid Texture Model (HTD) to overcome problems with LBP's sensitivity to noise. HTD is composed of local primitive features and global spatial structures. HTD is then combined with an adaptive quantization algorithm (AQA) to improve the noise robustness of the angular space quantization since the vector quantization thresholds are adjusted based on the content of the local patch.
Let X be an input random variable for quantizing, then the variation interval of X is A. Now define partition = where is the probability density function describing the input variable X.
The quantization thresholds for are set as where erf (•) is the error function, and is the standard deviation of the mean value of the whole image texture.
Letting be the standard deviation of and given Eq. (8), the quantization thresholds for are where L and G indicate local and global, respectively.
The length of the histogram can be compressed by splitting the global pattern and the local pattern into many binary patterns.
The algorithm for computing AHP features is a five-step process: 1. Compute the thresholds for adaptive quantization using Eq. (12) and (13). 2. For each local patch with radius R and neighborhood P, compute the local patterns and global patterns as follows: 6. Map each binary pattern into rotation invariant uniform (riu2) or uniform (u2) patterns; 7. Compute the statistical histogram of _ and _ accumulated over the whole texture image; 8. Output the joint combination of the histograms from step 4 as the final AHP texture descriptor.

Full Binarized Statistical Image Features (FBSIF)
FBSIF [54] is an extension of the Binarized Statistical Image Features (BSIF). With a set (n) of linear filters, BSIF assigns an n-bit label to each pixel in a given image. These assignments project local image patches (size of pixels) onto a subspace.
The n-bit label is represented in binary form as where is a × 2 matrix containing the compilation of the vector notations of the filters and X is the 2 × 1 vector notation of the × neighborhood. If the vector is defined as containing the pixels of the × neighborhood, and wi is the i-th row of W, then the i-th digit of is a function of the i-th linear filter wi expressed as with each bit of the BSIF code defined as The set of filters wi is produced by maximizing the statistical independence of the filter responses on a set of patches in an image by applying Independent Component Analysis (ICA). FBSIF is an extension of BSIF. FBSIF alters the filter size (SIZE_BSIF, size ∈ {3, 5, 7, 9, 11}) and a threshold th used for binarizing (FULL_BSIF, th ∈ {-9, -6, -3, 0, 3, 6, 9}). This process produces thirty-five (size, th) combinations, each of which trains a separate SVM. The set of SVMs is then combined via the sum rule.

Locally Encoded Transform feature histogram (LET)
LET [55] is a histogram that encodes the joint information within an image across feature and scale spaces. Computing LET descriptors is a three-step process: 1. Transform features that characterize local texture structures and their correlation are constructed (this is accomplished by applying linear and non-linear operators on the responses of directional Gaussian derivative filters in scale space). 2. Scalar quantization is performed with binary or multilevel thresholding used to quantize the transform features into texture codes. 3. The discrete texture codes are aggregated into a histogram representation.

Codebookless Model (CLM)
CLM [56] is a variant of the Bag of Feature (BoF) [59]. CLM substitutes the dictionary in BoF with a single Gaussian.
N local features { ∈ ℝ ×1 , = 1, … , } are extracted from a given image on a dense grid using the maximum likelihood method, and the features are represented with a gaussian model as: where det(·) is the matrix determinant, = 1 � ∑ =1 is the man vector, and Multiscale SIFT descriptors [60] are then extracted. This is accomplished with the standard BoF pipeline having a cell size 2 , = {1,2, … , } and with a single scale pixel-wise covariance descriptor [61] using dense sampling (steplength = 2). The dense covariance descriptors are calculated using a raw feature vector of length seventeen. This vector provides information about the intensity of the image and four kinds of first-order and second-order gradients. Matrix logarithm is evaluated on the covariance descriptors (LogCov) and vectorized. In this work, we use the VLFeat library [62] to calculate the SIFT features.
Concatenated with the SIFT and LogCov descriptors is more information about the gradient, scale, entropy, color, and location of the image. The width and height of the image are also resized to a minimum of sixty-four pixels to ensure sufficient data for calculating the covariance matrices and Gaussian models. It is also necessary to add ε=10 -3 to avoid zeros in the diagonal of the covariance matrices. The image is further divided into regular regions (such as 1 × 1, 2 × 2, 3 × 3, and 4 × 4 ) using a spatial pyramid strategy. A Gaussian model is computed on each region and concatenated to represent the whole image.

Extended Threshold Adjacency Statistics (ETAS)
ETAS [57] utilizes Threshold Adjacency Statistics and seven threshold ranges to generate seven feature sets that are fed into seven separate SVMs. A given input image is binarized using the following seven threshold values : where is the average intensity of the image, and is the limit or cut-off value for calculating the average. The seven resulting binary images are then used to extract nine statistical features.

Deep Learner Ensembles and Augmentation
CNNs are a class of deep feed-forward regularized multilayer perceptrons that are scale, shift, and distortion invariant and composed of interconnected neurons arranged in three dimensions (width, height, and depth). Every layer in a CNN transforms a 3D input volume into a 3D output volume of neuronal activations. The basic CNN is composed of five layers: convolutional (CONV), activation (ACT), pooling (POOL), Fully-Connected (FC), and classification (CLASS). As noted in the Introduction, CNNs automatically extract features during the training process. In general, CONV layers are responsible for feature extraction and are processed sequentially. The ACT layer is used for thresholding each input element using, e.g., the tanh or ReLu function. The POOL layer takes the output of the ACT layer and performs a downsampling according to two parameters: pool size (the rectangular size considered for downsampling) and stride value (the step size for processing the rectangular window). One or more FC layer are commonly placed at the end of the network, just before the final CLASS layer.
In this work, we evaluate ensembles composed of the following classic CNN architectures pretrained on the ImageNet database: • In our experiments, each CNN is fine-tuned on each of the tested datasets. Fine-tuning a CNN is a procedure that essentially resumes the training process of a given pre-trained network so that it learns a new classification problem. The net is therefore initialized using pre-trained weights obtained on the large ImageNet dataset and further trained using the training set of the target problem. All the networks are modified by changing the last FC and CLASS layers to fit the number of classes of the target problem, without freezing the weights of the previous layers.
A CNN that fails to converge is excluded from the ensemble. We also exclude CNNs trained with large batch sizes or that result in a "GPU out of memory" error.

Augmentation Methods
It was noted in the introduction that Data Augmentation (DA) is not only a highly effective way to enlarge a small training set but also a method for enhancing performance and reducing overfitting during training.
Our DA workflow can be described as follows. At the beginning of each epoch, we randomly transform each image in a given dataset with some basic preprocessing methods, such as rotation and reflection. Following this random preprocessing stage, four different data augmentation protocols (App1-4) are applied. These augmentations are performed on the fly on every batch. Hence, they do not require more memory, but they do increase the training time.
The four DA protocols used in this study are the following: 1. App1: The image is reflected in the left-right direction with 50% probability. 2. App2: The image is randomly reflected in both the left-right and the top-bottom directions. In addition, App2 linearly scales the image along both axes by two different factors that are randomly sampled from the uniform distribution in [1,2] In addition to the four different DA protocols, we applied two new approaches presented for the first time here that are based on two common feature transforms: DCT and PCA. Both these transforms are based on the projection of the original image onto the DCT/PCA subspace and perturb the backprojection from the subspace to the original space.
Our two new DA approaches are labeled as follows: 1. App5: DA approach based on PCA; 2. App6: DA approach based on DCT.
PCA [67] is a popular method for image compression, and it is often used as an unsupervised dimensionality reduction method. Computationally cheaper to compute than PCA, DCT maps feature vectors into a smaller number of uncorrelated directions calculated to preserve the global Euclidean structure. Similar to DCT, PCA also extracts an orthogonal projection matrix by maximizing the variance of the projected vectors. DCT provides a good compromise between information packing and computational complexity [68]. Computational complexity is reduced because DCT is not data-dependent, while PCA is based on the eigenvalue decomposition of the data covariance matrix. DCT components are also small in magnitude since most salient information exists in the coefficients with low frequencies. However, discarding the transform coefficients corresponding to the highest frequencies from the representation produces small errors in image reconstruction.
Once the PCA and DCT coefficients of the decompositions are calculated, we propose three different methods for generating new images. In the first method, every component of the feature vector obtained with PCA/DCT is randomly set to zero with a given probability. Then, the inverse of the transform is performed on the new feature vector, and a new image is generated. In the second method, some of the features extracted in random points selected using a gaussian distribution are reset. After that, the inverse of the transform is performed. In the third method, five random images in the dataset are selected that have the same label as a given image. We then perform a feature transform on all six images to obtain their feature vectors via the PCA/DCT transform. At this point, we randomly exchange some of the features of the original image with some of the corresponding features of the five randomly selected images. We then perform the inverse of the transform to generate the new image, which is a mixture of the six images, and label it the same as the others the augmented image was generated from. We do this for each image of the training set adding the new images to it.
Below is the pseudocode for the new DA methods. For the sake of brevity, we report the pseudocode for DCT only. The methods based on PCA are the same, except that the PCA space is built using the training data instead of the DTC space. The images generated by PCA and DCT are also reflected in the left-right direction with 50% probability for further augmentation. Notice that for each image we have three channels (RGB). The formula for calculating DCT used in the three algorithms presented in above is the following:

Stochastic approach
The stochastic approach creates an ensemble of networks starting from a given architecture and a pool of different activation functions. We create a new random network by randomly substituting all the activations in the original architecture with a new one among those in the pool. Note that a different activation is selected in every layer. This algorithm can be repeated multiple times and yield a different network each time. Hence, these networks can be grouped together to create an ensemble. In our experiments, we use ResNet50 as our backbone architecture and a large pool of activations. The functions that we include in the pool are ReLU, leaky ReLU, ELU, PReLU, S-Shaped ReLU, Adaptive Piecewise Linear Unit, MeLU, GaLU, Symmetric MeLU, Symmetric GaLU, flexible MeLU, MeLU+GaLU, Tanh+ReLU, Splash, and Parametric Deformable Linear Unit [69].

Datasets
Several datasets that include very different image types were selected to test our system and demonstrate the generalizability of our GenP bioimage system. In order to ease the comparison of other systems with the one proposed here, the datasets used in our experiments are all publicly available: • CH: the CHINESE HAMSTER OVARY CELLS [70] dataset of 327 fluorescent microscopy images, size 512×382, that are divided into five classes. This dataset is located at http://ome.grc.nia.nih.gov/iicbu2008/hela/index.html#cho; • HE: the 2D HELA dataset [70] of 862 images, size 512×382, of HeLa cells acquired by fluorescence microscope and divided into ten classes. This dataset is located at http://ome.grc.nia.nih.gov/iicbu2008/hela/index.html; • LO: the LOCATE ENDOGENOUS [71] dataset of 502 images, size 768×512, of mouse sub-cellular images showing endogenous proteins or specific organelle features. The images are unevenly divided into ten classes. This dataset is located at http://locate.imb.uq.edu.au/downloads.shtml; • TR: the LOCATE TRANSFECTED dataset of 553 mouse sub-cellular images, size 768×512, showing fluorescence-tagged or epitope-tagged proteins transiently expressed in specific organelles [71]. The images are unevenly divided into 11 classes. This dataset is located at http://locate.imb.uq.edu.au/downloads.shtml; • RN: the FLY CELL dataset [71] of 200 images, size 1024×1024, of fly cells acquired by fluorescence microscopy and divided into ten classes. This dataset is located at http://ome.grc.nia.nih.gov/iicbu2008/rnai/index.html; • MA: Muscle aging [72] dataset of images, size 1600×1200, of C. elegans muscles at four ages. This dataset is located at https://ome.grc.nia.nih.gov/iicbu2008; • TB: Terminal bulb aging [72] dataset of images, size 300×300, of C. elegans terminal bulb at seven ages (hence, seven classes). This dataset is located at https://ome.grc.nia.nih.gov/iicbu2008; • LY: Lymphoma [72] dataset of malignant lymphoma images, size 1388×1040, of three subtypes. This dataset is located at https://ome.grc.nia.nih.gov/iicbu2008; • LG: Liver gender [72] dataset of images, size 1388×1040, of liver tissue sections from 6-month male and female mice on a caloric restriction diet (the classes are the two genders). This dataset is located at https://ome.grc.nia.nih.gov/iicbu2008; • LA: Liver aging [72] dataset of images, size 1388×1040, of liver tissue sections from female mice on an adlibitum diet of 4 ages. This dataset is located at https://ome.grc.nia.nih.gov/iicbu2008; • CO: Collection of images, size 150×150, of textures in histological images of human colorectal cancer [73]. This dataset is located at https://zenodo.org/record/53169#.WaXjW8hJaUm.
Unless specified otherwise in the description of the datasets above, the protocol used in our experiments was the five-fold cross-validation method. To avoid overfitting, the same set of descriptor parameters were used for all descriptors across all tested datasets. The following experiments were statistically validated using the Wilcoxon signedrank test [74].

Experiments
In Table 2, we report the performance obtained by some of the handcrafted features and the following ensembles: In the last row of Table 2, we report the performance of the handcrafted ensembles tested in Nanni et al. [41], labeled PREVIOUS. In the column labeled Avg, we report the average accuracy obtained by a given descriptor/ensemble across the entire set of tested datasets.
Examining Table 2, we find that FBSIF and MLPQ obtained the best performances among the tested individual descriptors. There is no statistical difference between these two methods, however. Both outperform all the other individually tested methods with a p-value of 0.1. The best performing ensembles are FUS1 and FUS2. They outperform the other ensembles, as well as the best performing individual descriptors FBSIF and MLPQ, with a p-value of 0.1.
In Table 3, we compare the different approaches for data augmentation, reporting, for the sake of space, the performances using ResNet50 and DenseNet. The method labeled ENS is the sum rule among the CNNs trained using the six data augmentation approaches. The performance reported in Table 3 is the sum rule of each method trained with the two different learning rates ({0.001, 0.0001}) and different batch sizes ({10, 30, 50, 70}), as reported in the previous section. The method named E_ReLu is an ensemble of 15 CNNs all trained with a learning rate of 0.001 and a batch size of 30. E_ReLu represents a baseline for comparing the ensembles built considering different data augmentation approaches. SA is the performance of a single CNN trained with a learning rate of 0.001 and a batch size of 30 (which, on average, is the optimum set of hyperparameters). Table 2. Performance accuracy of local-based approaches and their fusion across the considered datasets Clearly, the best approach is given by ENS, which outperforms all the other approaches with a p-value of 0.1. Among the stand-alone methods, the best performance is obtained by the DCT-based method. These results demonstrate the value in using feature transforms for augmenting datasets and improving the performance of CNN.
Finally, in Table 4 we compare the performance of some of our ensembles with several state-of-the-art approaches reported in the literature. The following ensembles are reported in Table 4: • HAND: the method named FUS1 in Table 2 When we combine two methods by sum rule, their scores are normalized before the fusion to mean 0 and standard deviation 1. HAND + DEEP is the best ensemble proposed here: it outperforms all the other approaches reported in the experimental section with a p-value of 0.1. Clearly, the proposed ensemble outperforms the ensembles in [41] and [75]. It should be noted that we tried to improve the performance of HAND + DEEP by adding Stoc to it, but the performance was very similar.
Unlike the other state-of-the-art methods, the MATLAB source code for reproducing results is freely available. Given that all the descriptors can be calculated in parallel by exploiting the modern multicore CPUs (for handcrafted features) and GPUs (for deep learning features), all the descriptors can be extracted in a reasonable amount of time for all applications where real-time computation is not important (which is the case for many medical image classification problems).  Inés et al. [83] 98.00 99.00 ---------73.00 95.00 ---100. 99.00 ---

Conclusion
In this paper, we propose a GenP bioimage ensemble that combines handcrafted and learned descriptors. An ensemble of deep learning methods is built using different criteria (different batch sizes, learning rates, topologies, and methods of data augmentation). We propose three new methods for data augmentation based on feature transforms (principal component analysis and discrete cosine transform) that boost the performance of Convolutional Neural Networks (CNNs). Each handcrafted descriptor is used to train a different Support Vector Machine (SVM), and the different SVMs are combined with the ensemble of CNNs. The experimental section shows that a boost in performance is obtained by combining local features, dense sampling features, and deep learning approaches. The discriminative power and generalizability of our best performing bioimage system are verified on a wide range of publicly available bioimage benchmark datasets, each representing a different classification task.
The main contributions of the proposed paper are the following: • Three new approaches for data augmentation based on PCA/DCT are presented; • The power of using different data augmentation techniques for building an ensemble of CNNs is demonstrated; • A set of handcrafted/learned descriptors is proposed that is not only highly generalizable but also capable of achieving state-of-the-art performance on a large group of diverste image datasets.
In the future, we plan to investigate methods for combining this system with other approaches. We also plan to investigate novel methods for training CNNs on smaller training sets and for reducing the dimensions of deeper CNN layers.
To reproduce the experiments reported in this paper, the MATLAB code for all the descriptors and data augmentation methods is available at https://github.com/LorisNanni. Author Contributions: Author Contributions: L.N. conceived the presented idea., G.M., S.G. and M.P. performed the experiments. S.B., G.M., L.N. wrote the manuscript. S.B. and M.P. provided some resources.
Funding: This research received no external funding.