{'fraction_non_alphanumeric': 0.038113334078462055, 'fraction_numerical': 0.012071085279982116, 'mean_word_length': 5.025589225589226, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 0, 'https://': 5, 'lorem ipsum': 0, 'www.': 1, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
AnnotatorJ: an ImageJ plugin to ease hand annotation of cellular compartments

AnnotatorJ combines single-cell identification with deep learning (DL) and manual annotation. Cellular analysis quality depends on accurate and reliable detection and segmentation of cells so that the subsequent steps of analyses, for example, expression measurements, may be carried out precisely and without bias. DL has recently become a popular way of segmenting cells, performing unimaginably better than conventional methods. However, such DL applications may be trained on a large amount of annotated data to be able to match the highest expectations. High-quality annotations are unfortunately expensive as they require field experts to create them, and often cannot be shared outside the lab due to medical regulations. We propose AnnotatorJ, an ImageJ plugin for the semiautomatic annotation of cells (or generally, objects of interest) on (not only) microscopy images in 2D that helps find the true contour of individual objects by applying U-Net–based presegmentation. The manual labor of hand annotating cells can be significantly accelerated by using our tool. Thus, it enables users to create such datasets that could potentially increase the accuracy of state-of-the-art solutions, DL or otherwise, when used as training data.


Test data
We tested the efficiency in terms of both annotation time and accuracy on two test image sets: nucleus and cytoplasm. A sample of the test sets is displayed on Figures 3B-C and S1B-C; images are collected from publicly available datasets and our collaborators (see sources in (Hollandi et al. 2020)). We collected images from various modalities including different microscopy imaging (brightfield, fluorescent), sample origin (cell lines, histological tissue sections), staining technique (label-free, immunohistochemical or fluorescent markers for specific cellular compartments) that show the target object (nucleus or cytoplasm) in a heterogeneous environment so that we can test our plugin in general cases.

Evaluation
We used SEM (standard error of the mean) in our evaluation of annotation time and accuracy which is formulated as where the sample is denoted as , ( ) marks the standard deviation of and is the number of elements in .
We also tested annotation on a cytoplasm test image set containing more complex regions e.g. overlapping objects (see Figure S1). Our cytoplasm model aided annotation, however, due to its semantic nature it could not always separate touching objects properly hence the increased annotation time in certain cases.
Supplementary Figure S1. Annotation times on cytoplasm images. AnnotatorJ was tested on sample microscopy images (both fluorescent and brightfield, as well as cell culture and tissue section images), annotation time was measured on a per-object (cytoplasm) level. Bars represent the mean annotation times on the test image set, error bars show SEM (standard error of the mean). Orange corresponds to Contour assist mode and blue to only allowing the Automatic adding option. A) Cytoplasm annotation times, B) example histopathology test images, C) example cell culture images. Images shown on B-C) are 256x256 crops of original images. Some images are courtesy of Kerstin Elisabeth Dörner, Andreas Mund, Viktor Honti and Hella Bolck.
As any user-defined U-Net model can be easily imported in AnnotatorJ via DL4J, we tested general applicability on the Cityscapes dataset (Cordts et al., 2016): we trained a custom model on the car object class. Expert annotators were not needed for testing on these everyday objects. Figure S2 shows annotation times and example images. Acceleration in annotation time is also significant (similarly to the nucleus test set); contour suggestion helps the most in complex-shaped regions such as outside mirrors and bumpers.
Supplementary Figure S2. Annotation times on car images. AnnotatorJ was tested on sample images of the Cityscapes dataset (Cordts et al., 2016), annotation time was measured on a per-object (car) level. Bars represent the mean annotation times on the test image set, error bars show SEM (standard error of the mean). Orange corresponds to Contour assist mode and blue to only allowing the Automatic adding option. A) Car annotation times, B) example images as 1024x1024 crops of original images outlined by a randomly selected annotator.
Additionally, we tested AnnotatorJ using a model trained on electron microscopy (EM) data from the ISBI 2012 challenge (Arganda-Carreras et al., 2015) (fetched from https://github.com/zhixuhao/unet, which repository was also used to train our U-Net models in Keras) to help annotate neuronal cells (see Fig. S3). The complex shapes these structures had also emphasized the efficiency of our contour suggestion.
Supplementary Figure S3. Annotation times on electron microscopy (EM) images. AnnotatorJ was tested on sample images of the ISBI 2012 challenge (Arganda-Carreras et al., 2015), annotation time was measured on a per-object (cell) level. Bars represent the mean annotation times on the test image set, error bars show SEM (standard error of the mean). Orange corresponds to Contour assist mode and blue to only allowing the Automatic adding option. A) EM annotation times, B) example images as 256x256 crops of original images outlined by a randomly selected expert.

Active contours (AC)
Active contours (Kass et al., 1988) is a generally well applicable image processing technique to fit an initial contour to the boundaries of an object by evolving a socalled snake contour driven by 2 energy functions as follows. (4) is the combination of the internal (5) and external (6) energy functions, and is minimized during the iterative evolution process. The internal energy is responsible for extending and smoothing the contour while the external energy restrictively drives it towards the edges of the image which are enhanced by the LoG (Laplacian of Gaussian) filter-convolved image (data term). , and are the regularization parameters.

Gradient vector flow (GVF)
GVF was first published in (Xu and Prince 1997) as a method to define a vector field based on the image gradient, that is the derivatives of the image, as enhanced by edges. It is attracted to bright regions (ideally edges) on the image.

Additional annotation tools
Since open source or local software is of primary interest in medical research (see in Introduction and Comparison to other tools and software packages), we extend our previous list of compared tools here.
We briefly discussed supervise.ly (https://supervise.ly/) above, a service-based webhosted solution for data labelling with DL model support. Training of various DL models including Mask R-CNN, U-Net, YOLO and others is possible. It supports annotation as bounding box, semantic-and polygonal drawing, editing points in polygons and class labelling. However, its free community edition intended for research purposes provides limited functionality.

FastAnnotationTool (https://github.com/christopher5106/FastAnnotationTool) is a lightweight, open source bounding box annotation tool with editing option in C++.
Make Sense (https://www.makesense.ai/) is another open source labelling tool that also adds DL support for annotation suggestion (using pre-trained models) and labelling. Point, polygon and bounding box annotation is possible, and classes can be assigned. Implemented in TypeScript, it offers both a web-based and a local (https://github.com/SkalskiP/make-sense) version.
ilastik (Sommer et al., 2011;Berg et al., 2019) is a pixel-based classification software that can be trained via brush strokes on the image. It can export semantic segmentation masks and their probability maps as well. However, these are returned by machine learning algorithms; users can annotate regions (pixel-based) with a brush or objects (instances) by single clicks to train them. Suite2p (Pachitariu et al., 2017) was designed for a specific task: neuronal twophoton microscopy image analysis. It has a pipeline that can be modified in each step so that it fits the user's data, automatic detection and classification of ROIs (cells) can be manually curated.
LabKit (Arzt 2017) is an ImageJ plugin for machine learning-supported labelling that similarly to the Trainable Weka Segmentation (ImageJ) plugin (Arganda-Carreras et al., 2017) and ilastik (Sommer et al., 2011;Berg et al., 2019) uses user-defined regions marked by brush strokes to train a machine learning algorithm and automatically segment image pixels to the defined classes. Both plugins offer several machine learning algorithms implemented in the Weka framework (Frank et al., 2016). Trainable Weka Segmentation can save semantic segmentation masks or their probability maps (like ilastik) while LabKit can also save manually drawn objects as multi-labelled .tiff files (like AnnotatorJExporter).
We collected the sources of annotation tools in Table S1.