{'fraction_non_alphanumeric': 0.02880710659898477, 'fraction_numerical': 0.013071065989847716, 'mean_word_length': 4.94082171127026, 'pattern_counts': {'":': 0, '<': 2, '<?xml version=': 0, '>': 0, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 1, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Cell Segmentation in 3D Confocal Images using Supervoxel Merge-Forests with CNN-based Hypothesis Selection

Automated segmentation approaches are crucial to quantitatively analyze large-scale 3D microscopy images. Particularly in deep tissue regions, automatic methods still fail to provide error-free segmentations. To improve the segmentation quality throughout imaged samples, we present a new supervoxel-based 3D segmentation approach that outperforms current methods and reduces the manual correction effort. The algorithm consists of gentle preprocessing and a conservative super-voxel generation method followed by supervoxel agglomeration based on local signal properties and a postprocessing step to fix under-segmentation errors using a Convolutional Neural Network. We validate the functionality of the algorithm on manually labeled 3D confocal images of the plant Arabidopis thaliana and compare the results to a state-of-the-art meristem segmentation algorithm.


INTRODUCTION
The shoot apical meristem present in flowering plants is comprised of a network of stem cells responsible for all above ground development of the plant. These tightly arranged cells partition the space into approximately convex polyhedra, resembling a Voronoi diagram, and forming a dome-like structure. To support scientists in their quest to understand the mechanisms that lead to cell development in the meristem and consequently plant growth, it is helpful to automatically quantify the geometry and topology of the network. With this information one can then, e.g., compute the spatial-temporal distribution of cell size from images obtained after lab experiments, use measurements to assist in the creation and validation of mathematical models of development and cell-cell communication, and carry on computational simulations on more faithful representations of the existing network. The work we present here is a step towards the goal of automatically quantifying the meristem of the Arabidopsis thaliana plant, with possible applications in similar images.
Several automated approaches have been presented in the past [5,10,11,13] to quantitatively analyze potentially thousands of cells. Most existing methods preprocess the images to clean and boost the signal on cell plasma membranes, e.g., using mathematical morphology [5,7,13], Hessianbased edge enhancement [9,13] or anisotropic filtering [10], followed by the actual segmentation using a 3D watershed algorithm [5,7,9], a combinatorial fusion of 2D segmentations [13] or variational methods [4,10]. Finally, postprocessing heuristics are often used to suppress spurious detections based on volume constraints. While existing approaches work approximately well in the epidermis of the meristem, a region with better signal-to-noise ratio and comparable cell volumes, we found that most methods have particular difficulties in deeper layers of the meristem (Fig. 1A). Large variations of the fluorescent signal affect the performance of global, intensity-based parameters like h-minima that are frequently used for initialization of the 3D watershed and may cause leakage across weak boundaries or may erroneously split cells due to being set too sensitively. Hessian-based plane enhancement frequently fails to properly reconstruct the signal of en-face membranes and may even remove faint edges, resulting in under-segmentation errors in the axial direction.
To overcome these limitations and to improve the quality of automatic segmentation also in deeper tissue layers, we largely skip sophisticated preprocessing steps that poten-tially introduce additional artifacts. Instead, we try to obtain an initial over-segmentation that partitions the image using a set of supervoxels that does not span across cell boundaries. We then agglomerate neighboring supervoxels based on prior knowledge to form a set of merge-trees (e.g., based on edge intensity, object volume and shape). As a final postprocessing step, under-segmentation errors that were caused by erroneous fusion of supervoxels are resolved using a Convolutional Neural Network (CNN). The algorithm does not rely on a nucleus channel to provide seeds for the segmentation and solely uses features present in the membrane images. Moreover, a key advantage of the algorithm is the need of few parameters making it easy to adjust and apply to new image data. We validated the performance of the proposed algorithm on manually annotated 3D confocal microscopy images of the Arabidopis thaliana shoot apical meristem and compare the results to existing cell shape segmentation algorithms [5,9].

SUPERVOXEL-BASED 3D SEGMENTATION
Supervoxel-based segmentation approaches have been successfully applied in various computer vision domains (including segmentation of 2D and 3D microscopy images) and form the basis of the method presented in this contribution [6,8,13,16]. The approaches consist of mainly three steps: (1) a preprocessing stage to improve the image signal and to enhance the desired object boundaries, (2) a supervoxel generation step that partitions the enhanced image into meaningful parts and (3) a supervoxel merging phase that tries to agglomerate existing supervoxels to the desired objects of interest. An important precondition that has to be met by the supervoxel algorithm is that identified regions should not span over multiple objects as these under-segmentation errors cannot be corrected in subsequent merging steps.

Preprocessing and Supervoxel Generation
Starting with the raw input image (Fig. 1B), we use a 3D Gaussian filter to reduce the high-frequency noise in the image (Fig. 1C). Subsequently, we apply an iterative morphological closing to the Gaussian-smoothed image, in order to close membrane gaps [13,15] (Fig. 1D). The radius for the iterative closing should be selected such that the maximum radius is smaller than the radius of the smallest objects that need to be resolved. Throughout the experiments performed for this paper, we used a maximum closing radius of R cl max = 3. For supervoxel generation we apply a morphological watershed algorithm that is initialized with all local minima present in the preprocessed image (Fig. 1E) [2]. By skipping the frequently used h-minima transform in the first place, we strongly reduce under-segmentation errors and the need to manually optimize this parameter locally, as done in [18]. However, using all local minima potentially generates several split segments per cell that need to be combined to complete cells in a subsequent supervoxel merging phase using a region adjacency graph.

Merge-Forest Generation of 3D Supervoxels
The following step is the generation of a merge-forest, i.e., adjacent supervoxels are iteratively merged on the basis of volume-and intensity-based edge features to a set of mergetrees. Each node of a merge-tree corresponds to a potential segmentation hypothesis. In contrast to existing methods that precompute a single merge-tree for the entire image [3,6,8], we terminate the merging phase based on a specimen-dependent maximum volume constraint. This allows us to pick the best hypotheses using a CNN-based postprocessing step that only needs to be trained on small image snippets that maximally cover a few cells rather than all levels of detail of the entire image.

Minimum Volume Condition
The first merge feature uses the minimum volume V min to determine which of the supervoxels is smaller than the smallest cell and needs to be merged. This specimen-dependent parameter can be approximated, e.g., by measuring the smallest radius R min of any object within the data set. By approximating the cells as spherical objects, we calculate V min = 4 3 · π · R 3 min . If the segmented objects substantially violate the sphericity assumption, V min can be replaced by measurements of manual segmentations or textbook knowledge. To identify which of the supervoxels should be merged, we linearly map all values between zero and V min to the interval [0, 1]: If the volume of the smaller merging candidate is below V min , the supervoxels i and j need to be merged. To ensure that edges with smaller intensities are collapsed first [6], we multiply f Vmin (i, j) with the average intensity µ cmn ij of the shared boundary between the two segments i and j if f Vmin (i, j) < 1. If the volume of segment i or j is larger than V min , f Vmin evaluates to 1 and does not trigger further merges.

Bright Homogeneous Boundary Condition
The second merge feature is based on the assumption that each cell should be surrounded by a bright boundary with a darker interior where no fluorescent marker is expressed ( Fig. 2A). We compare the intensity difference of the mean intensity on the shared boundary µ cmn ij to the mean intensity of the entire cell including the boundaries µ total ij versus the intensity on the boundary after fusion of the segments µ bdry ij : A value of f bdry (i, j) smaller than 1 indicates that the intensity difference of the considered intersection to the intensity of the entire cell including the boundary is smaller than the intensity of the boundary of the fused cell. Thus, the shared boundary is likely to be a part of the cell interior and may be removed. Instead of using absolute intensity values, the measure is based on relative intensity differences to avoid that region merges in low-intensity regions are more likely than region merges in high-contrast regions.

Edge Feature-based Merge-Forest Generation
In order to merge supervoxels using multiple features, all features need to be normalized such that none of the features suppresses the others due to a larger value range. All employed edge features lie in the interval [0, 1], with 1 indicating that no merge should be performed and a value closer to zero indicating that a merge is likely (Fig. 2B). All features are then combined to a feature vector, which is divided by the square root of the number of features for normalization: As long as f sort (i, j) < 1, at least one of the constraints for a valid segment boundary is violated. For each pair of adjacent supervoxels i and j, we compute f sort (i, j) and sort the merge queue in ascending order based on this feature. The algorithm sequentially processes the merge queue, by merging the two supervoxels connected by the edge with the lowest merge score (Fig. 2C). After a merge has been performed, all edges previously connected to one of the two merged supervoxels are updated and inserted to the merge queue at the appropriate location. Using optimized data structures that keep track of the connectivity, intensity properties and merge hierarchy of all supervoxels, merge features can be recomputed recursively. This provides a fast way of updating edge features after merging without further iterations over all voxels in subsequent steps. Additionally, each performed merge is memorized and stored as a merge-tree, yielding a merge-forest when merging supervoxels is no longer valid. To prevent the merge-trees from growing too large, edges are skipped if the combination of the two neighboring supervoxels would violate the maximum volume criterion.

CNN-based Selection of Merge-Tree Nodes
As a final post-processing step, we use a Convolutional Neural Network to classify all hypotheses of all merge-trees and to resolve under-segmentations that are potentially introduced by the region merging step (Fig. 2D). The classification is based on 3D image patches of size 32 × 32 × 32 voxels that contain either a single correct cell, a cell segment (over-segmentation) or multiple connected cells (undersegmentation). The data for training and testing the network was extracted from the top layers of 20 data sets of the shoot apical meristem in A. thaliana (Fig. 2E) [18]. We adapt TensorFlow's Deep MNIST network to work with 3D input images using two convolution layers (5×5×5), each followed by a max pooling layer (2×2×2, stride 2). These layers yield tensors with 32 and 64 features, respectively. Finally, we use a fully connected layer with 1024 output nodes with a dropout probability of 0.5 that is connected to a 3-class softmax layer. Training was performed using the ADAM optimizer on the cross-entropy loss function implemented in Google's Tensor-Flow API and executed on an NVIDIA Titan Xp. To use the identified class probabilities to correct under-segmentation errors, all merge-trees are revisited in a top-down fashion.
If the under-segmentation probability is larger than one of the other classes, the current segmentation hypothesis (i.e., the root node of the current merge-tree) is discarded and the hypotheses of the daughter nodes are checked instead. This process is repeated recursively until one of the probabilities of the other classes exceeds the under-segmentation probability or upon reaching a leaf node of the merge-tree (Fig. 2F).

SEGMENTATION OF THE SHOOT APICAL MERISTEM IN A. THALIANA
We first applied the new pipeline to 124 3D confocal image stacks of A. thaliana with manually corrected L1 and L2 layers [18]. On average, the proposed algorithm achieved a precision of 0.991 ± 0.008, a recall of 0.973 ± 0.012 and an F-score of 0.982 ± 0.008 for the L1 layer and a precision of 0.960 ± 0.033, a recall of 0.932 ± 0.023 and an Fscore of 0.945 ± 0.023 for the L2 layer. Due to the limited batch-processing capabilities of the current implementation, the CNN-based postprocessing was not yet enabled for these experiments and we expect that the recall will further improve upon adding this under-segmentation correction step.
To investigate the segmentation quality in deeper layers as well, we densely labeled a 128 × 128 × 200 image region using SEGMENT3D, a new interactive and collaborative 3D segmentation correction tool [12]. Fig. 3 and Tab. 1 compare the segmentation performance of ACME [9], MARS [5] and three intermediate steps of the proposed algorithm both qualitatively and quantitatively. The final pipeline including the CNN-based correction module reaches the highest F-Score of 0.943 and successfully corrected under-segmentation errors that were introduced by the merging step, yielding an im-

CONCLUSION
In this contribution, we present a new supervoxel-based approach to 3D cell shape reconstruction comprising an initial over-segmentation, an iterative supervoxel merging step and a CNN-based under-segmentation correction. The algorithm requires only two parameters, the minimum and maximum volume of cells. Even in deeper tissue layers that exhibit varying signal intensities and weak en-face membranes, the approach proved to reliably reconstruct 3D cell shapes for the most part. We validate the algorithm on manually labeled 3D confocal microscopy images of A. thaliana and quantify the improvements compared to state-of-the-art algorithms for cell shape reconstruction [5,9]. An initial version of the pipeline is implemented in the open-source software XPIWIT [1].
To improve the generalization capabilities of the CNN and to ideally make it work across different data sets and across specimens, we plan to train the network on both manually annotated and synthetic 3D microscopy data that represent and mimic various kinds of imaging circumstances [14,17]. In the current implementation, the preprocessing, the region merging and the over-segmentation correction modules are implemented in C++, MATLAB and Python. In future releases, we plan to streamline the pipeline, such that the entire approach can be directly performed in a single XPIWIT pipeline [1]. Furthermore, we plan to replace ITK's serial 3D watershed implementation by a parallel watershed implementation, to eradicate the current bottleneck of the pipeline and to make the algorithm applicable to terabyte-scale 3D+t data sets.