{'fraction_non_alphanumeric': 0.033398525650669475, 'fraction_numerical': 0.01160566528401642, 'mean_word_length': 4.824258355238453, 'pattern_counts': {'":': 0, '<': 2, '<?xml version=': 0, '>': 3, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Single-shot self-supervised particle tracking

Particle tracking is a fundamental task in digital microscopy. Recently, machine-learning approaches have made great strides in overcoming the limitations of more classical approaches. The training of state-of-the-art machine-learning methods almost universally relies on either vast amounts of labeled experimental data or the ability to numerically simulate realistic datasets. However, the data produced by experiments are often challenging to label and cannot be easily reproduced numerically. Here, we propose a novel deep-learning method, named LodeSTAR (Low-shot deep Symmetric Tracking And Regression), that learns to tracks objects with sub-pixel accuracy from a single unlabeled experimental image. This is made possible by exploiting the inherent roto-translational symmetries of the data. We demonstrate that LodeSTAR outperforms traditional methods in terms of accuracy. Furthermore, we analyze challenging experimental data containing densely packed cells or noisy backgrounds. We also exploit additional symmetries to extend the measurable particle properties to the particle's vertical position by propagating the signal in Fourier space and its polarizability by scaling the signal strength. Thanks to the ability to train deep-learning models with a single unlabeled image, LodeSTAR can accelerate the development of high-quality microscopic analysis pipelines for engineering, biology, and medicine.

of super-resolution fluorescence [9], and to quantitatively characterize sub-wavelength particles [10]. The most widely used deep-learning methods use supervised learning, where a neural network is trained to solve a particular problem using large amounts of high-quality training data consisting of input data and corresponding expected results (ground truth). Obtaining these datasets represents the effective bottleneck in the application of deep learning to particle tracking [2]. While public datasets are available, they are often inadequate to represent the idiosyncrasies of any specific experimental sample and setup. Thus, acquiring the needed experimental datasets in house is often the only viable option, but it comes with its own burdens in terms of time and effort. As a consequence, most deep-learning methods for particle tracking rely on synthetic data [7,8,9,10]. However, accurate synthetic replication of experimental data is very challenging, even for relatively simple transmission microscopes [2].
Even if a sufficiently large dataset can be collected, determining the corresponding ground truth with sufficient accuracy can be even more challenging. Human-derived annotation of experimental data is highly labor-intensive and prone to inconsistencies [11], especially when dealing with high-noise data or when sub-pixel precision is required. Automatically-generated annotations can be employed by mimicking an existing method, but this relies on a prior ability to algorithmically analyze the data -and the resuting deep-learning method is unlikely to outperform the original method. Finally, numerically-generated data naturally come with the exact ground truth used to generate them, but their applicability is limited to experimental setups that can be recreated numerically, which is only possible in some cases.
Here, we tackle these issues by developing a novel deep-learning approach, named LodeSTAR (Low-shot deep Symmetric Tracking And Regression), that exploits the inherent symmetries of particle tracking to enable training on extremely small datasets (down to a single particle image) without ground truth. We achieve this by merging two recent advances in deep learning. First, we employ geometric deep learning to dramatically reduce the amount of required training data by exploiting the symmetries inherent in particle tracking, e.g., the fact that a roto-translation of the input particle image relates to an equal roto-translation of the expected result. Second, we employ self-distillation to part with the need to know the ground truth by generating a pseudo-result, e.g., from a prediction of the position of an object in an image, we can generate a ground truth for a translated view of the same object by considering the translational symmetry of the problem. We demonstrate that a single training image is sufficient for training LodeSTAR to outperform standard methods in terms of accuracy, while simultaneously providing robust tracking in complex experimental conditions such as densely-packed or noisy images. Furthermore, beyond in-plane particle tracking, we demonstrate that it is possible to exploit additional symmetries to measure other particle properties for which no widespread standard methods are available, e.g., the particle vertical position exploiting the propagation symmetry of the image in Fourier space and the particle polarizability by exploiting the scaling symmetry of the signal strength of the image.

Geometric self-distillation
LodeSTAR is designed to exploit the symmetries inherent in particle tracking. For example, even if we do not know the absolute position of a particle, we can say for sure that, if we translate the particle image by a certain amount, its position gets also translated by the same amount -and similarly for rotations and reflections. This is known as an equivariance. In fact, for a particle whose image has a well-defined center of symmetry, such as that shown in Figure 1: LodeSTAR single-shot training and performance. a Example image of a single particle used to train the neural network (N × M pixels, C color channels). b Two copies of the original image transformed by translations and rotations. c The transformed images are fed to a convolutional neural network. d The neural network outputs two tensors (feature maps), each with N/2 × M/2 pixels: One (top) is a vector field where each pixel represents the distance from the pixel itself to the particle (top left, blue arrows), which is then transformed so that each pixel represents the distance of the particle from the center of the image (top right, blue markers). The other tensor (bottom) is a weight map (normalized to sum to one) corresponding to the contribution of each element in the top feature map to the final prediction. e These two tensors are multiplied together and summed to obtain a single prediction of the position of the particle for each transformed image. f The predicted positions are then converted back to the original image by applying the inverse translations and rotations. The neural network is trained to minimize the distance between these predictions. g-k LodeSTAR performance on 64 px × 64 px image containing different simulated particle shapes: g point particle, h sphere, i annulus, j ellipse, and k crescent. Even though LodeSTAR is trained on a single image for each case (found in the corresponding inset), its root mean square error (RMSE, blue circles) shows that it outperforms traditional methods based on the centroid [12] (gray triangles) or radial symmetry [13] (gray diamonds), especially at low signal-to-noise ratios (SNRs). Interestingly, even in the crescent case k, where there is no well-defined particle center, LodeSTAR is able track it to within a tenth of pixel. Figure 1a, a prediction of the position of the particle that is equivariant to the Euclidean group (translations, rotations, and reflections) can be shown to necessarily locate the center of the particle (see Methods, 'Theory of Geometric Self-Distillation").
LodeSTAR trains a neural network to achieve equivariance, i.e., an exact correspondence between the transformation applied to its input image (e.g., a translation by a certain amount) and the effect this has on the output prediction (e.g., a translation of the predicted particle position by the same amount). We start with an image of a single particle (Figure 1a). By applying a transformation from the Euclidean group to this image, we create several transformed images (e.g., the two images in Figure 1b). A neural network (Figure 1c) predicts the position of the particle within each image (Figure 1d, which will be more thoroughly explained in the next paragraph), yielding two predictions of the x and y coordinates of the particle, one for each transformation (Figure 1e). To compare these predictions, we back-transform the predicted coordinates to determine their corresponding positions in the original image ( Figure 1f). Then, if the neural network is equivariant to the applied transformations, the two predictions should perfectly overlap, while any deviation between these two prediction can be used to train the neural network (i.e., to adjust the neural network by updating internal weights to minimize the distance between the two predictions).
We now turn our attention to the details of the neural network employed by LodeSTAR. While most architectures are compatible with LodeSTAR, it is convenient to use a completely translation-equivariant neural network (e.g., a fully convolutional neural network). This presents a two-fold advantage. First, having inherent translation-equivariance, the neural network does not need to learn the equivariance during training, which significantly reduces the required complexity of the model (while rotation-equivariance and reflection-equivariance still need to be learned). Second, as will become clear in the following sections, this will permit us to use the network also to track multiple particles while still training on a single particle image.
As schematically shown in Figure 1c (details are described in the Methods, "Neural network architecture"), we use a fully convolutional neural network, with two convolutional layers, one max-pooling layer, and seven additional convolutional layers, the last of which outputs three channels, ∆x, ∆y, and ρ. The outputs of this neural network are further analyzed, as shown Figure 1d (this analysis also needs to be translation-equivariant). The vectors (∆x, ∆y) estimate the distance from each pixel to the particle (blue arrows in upper left panel of Figure 1d; far away from the particle, the length of these vectors goes to zero because of the limited receptive field of the convolutional neural network, indicating that it does not see a particle in its input data). By adding to each of these vectors the respective pixel position, we retrieve a map of predictions of the particle position relative to the upper left corner of the image (blue markers in the upper right panel of Figure 1d). The ρ channel provides a weight map (normalized to sum to one) corresponding to the probability of finding the center of the particle near each pixel (bottom panel in Figure 1d). The final prediction of the particle position is obtained by an average of the estimated particle positions weighted by the weight map.
LodeSTAR manages to train this network using a single image of the particle. Moreover, thanks to the small size of the neural network, it can be trained fully from scratch in an order of 10 4 mini-batches, which takes a few minutes, even without a graphics processing unit. Finally, LodeSTAR can be trained with small batch-sizes, and as such needs less than a gigabyte of runtime memory (see details in the Methods, "Neural network training").

Tracking a single particle
We start by considering the performance of LodeSTAR on the simplest case: a point particle (e.g., the image of a single molecule obtained from a fluorescence microscope). We simulate 10 4 images of point particles with signal-to-noise ratio (SNR) between 2 and 20 using the Python library DeepTrack 2.1 [2]. We use a single one of these images (inset in Figure 1g, SNR = 10) to train LodeSTAR, using 5000 mini-batches of 8 samples. LodeSTAR achieves a sub-pixel root mean square error (RMSE) for all SNRs and a RMSE < 0.1 px for SNR > 5 (blue circles in Figure 1g). Strikingly, LodeSTAR performs well also far from the SNR at which it is trained.
In order to assess the performance of LodeSTAR, we compare it with two of the most standard particle tracking methods, i.e., the centroid method [12] (gray triangles in Figure 1g) and the radial-center method [13] (gray diamonds in Figure 1g). While the radial center method achieves sub-pixel accuracy over the whole range of SNRs, it is consistently outperformed by LodeSTAR. The radial-center method approaches LodeSTAR's performance for high SNRs, but is not competitive at lower SNRs, even underperforming the centroid method. Overall, these results are consistent with the literature, where deep-learning-based methods have been shown to outperform traditional methods especially at low SNRs [2,7,8].
In contrast to traditional methods that are optimized for certain sets of particle shapes, LodeSTAR can track particles of arbitrary shapes. We show some examples in Figures 1h-k, where we present the results for a sphere (Figure 1h), an annulus (Figure 1i), an ellipse (Figure 1j), and a crescent ( Figure 1k). Overall, LodeSTAR achieves a sub 0.1 px error for the majority of the SNR-range in all cases, outperforming the standard methods by up to an order of magnitude. While we can expect LodeSTAR to perform well on symmetric objects, such as the sphere (Figure 1h) and the annulus (Figure 1i), it is an important confirmation of the generality of LodeSTAR that it also works for non-symmetric particles, such as the ellipse (Figure 1j), which only has two axes of symmetry instead of full radial symmetry, and the crescent (Figure 1k), which has only one axis of symmetry. The latter case is particularly interesting, because the standard methods perform rather poorly (RMSE > 1 px), while LodeSTAR retains a sub 0.1 px error, on-par with the symmetric cases (since the center of a crescent moon is not uniquely defined, we accept any position along the axis of symmetry to be a valid prediction as long as it is consistent across all images). Video 1 demonstrates LodeSTAR tracking the particles at various orientations and noise levels.
We turn now our attention to the discriminative power of LodeSTAR, i.e., its ability to learn about the specific shape used in its training. Such discriminative power is important for heterogeneous samples or just to minimize false positives. In order to do this, we consider the distribution of predicted positions of the particle in view preceding the pooling operation, i.e., the feature map ( Figure 1d). Specifically, we measure the self-consistency of the model by calculating the weighted variance of the feature map. We expect that the neural network is highly self-consistent when evaluated on images similar to the training data, while less so when evaluated on something distinctly different. As can be seen in Table 1, the variance is several orders of magnitude lower when the model analyzes an image drawn from the distribution on which it has been trained. This demonstrates that LodeSTAR acts differently when presented data similar to the training distribution compared to data from another distribution, which clearly shows that LodeSTAR has acquired discriminative power.

Tracking multiple particles
Thanks to the translation-equivariant design of the neural network, LodeSTAR trained on a single particle image as described in the previous section can immediately be used to track multiple particles without any additional training. In fact, since the receptive field of the convolutional network is limited, additional particles in view (such as in Figure 2a) are analyzed largely independently by the neural network. Taking advantage of these observations, we can bestow LodeSTAR with the capability to detect multiple particles, simply by removing the  Table 1: LodeSTAR displays discriminative power. Each model trained by LodeSTAR is significantly more self-consistent (lower average weighted variance) when presented data from its training distribution (bold font), compared to when it is evaluated on different data. This demonstrates that LodeSTAR has acquired some discriminative power.
weighted global pooling layer (i.e., the final multiplication in Figure 1e) and operating directly on the feature-maps themselves, i.e., on the predicted particle position map ( Figure 2b) and on the weight map ( Figure 2c). By multiplying a measure of the local density of particle positions by the weight map ( Figure 2d), we obtain a detection map whose local maxima represent the positions of the detected particles. After detection, the exact location of the particle is determined by a weighted average of the local region around the detection, analogous to the single-particle case, finally yielding a successful tracking of the image ( Figure 2e). See also details in Methods, "Particle detection criteria".

Validation with experimental data
We now apply LodeSTAR to various experimental images with multiple particles, which present different challenges (Figures 2f-j). We highlight that in all cases we train LodeSTAR on a single image (shown in the insets in the lower left corners in Figures 2f-j) and then apply it to the whole image. In all cases, the objects are dividing over time, resulting in a large range of densities and morphologies that LodeSTAR needs to handle. Videos visualizing the tracked images can be found for each case in the supplementary material (Videos 2-6). First (Figure 2f), we consider a dense sample. We analyze an experimental time-series of images of densely-packed mouse stem-cells. Even though the cells are very densely packed (in fact, they often touch each other), LodeSTAR manages to accurately identify their positions (red markers in Figure 2f), achieving a true positive rate of 99.4% and a false discovery rate of 0.9%. Strikingly, the ability to separate closely packed cells is completely emergent, since the neural network is never shown images with multiple cells during its training.
Then (Figure 2g), we consider an experimental scenario where the cells vary highly both in morphology and in intensity. We track the nuclei of human hepatocarcinoma-derived cells, showing the identified cell positions as red circles. Despite being a highly dynamic dataset with a very large variability between cells, LodeSTAR achieves a true positive rate of 92.8% and a false discovery rate of 2%. Furthermore, it should be noted that most missed detections stem from not immediately detecting cell division events present in the data.
In Figure 2h, we consider the doubly-challenging case of densely packed cells that vary in morphology and intensity. We analyze a time-series of pancreatic stem cells, showing the identified cell positions as red dots. Despite the cells being highly packed and varying significantly over instance and time, LodeSTAR achieves a 92.4% true positive rate and a 8% false discovery rate. However, it should be noted that a large portion of the false discoveries originate from errors in the ground truth provided by the Cell Tracking Challenge [11].
Finally, we consider some more complex objects with significant internal structure, some particles to be analyzed by LodeSTAR (LodeSTAR is still trained on a single particle image, as described in Figure 1). b LodeSTAR returns clustered predictions of particle positions and c a weight map representing the likelihood of finding a particle near each pixel. d An estimation of the local density of particle detections is multiplied by the weight map to obtain a detection map, whose local maxima are considered particle detections (orange markers in e). f -i Examples of applications of LodeSTAR to experimental data that present different challenges. In all cases, LodeSTAR is trained on the single crop shown in the respective inset and then applied to the whole time-series. See also the corresponding Videos 2-6. f LodeSTAR finds the positions of mouse hematopoietic stem cells (red markers), achieving a 99.3% true positive rate and a 0.9% false discovery rate despite the dense sample (data from [11]). g LodeSTAR identifies human hepatocarcinoma-derived cells (red circles), achieving a 92.8% true positive rate and a 2.8% false discovery rate, despite the high variability between cells (data from [11]). h LodeSTAR tracks pancreatic stem cells (red markers), achieving a 92.4% true positive rate and a 8% false discovery rate, despite the densely packed sample and the high variability between cells (data from [11]). i LodeSTAR tracks the plankton Noctiluca scintillans. In this case, LodeSTAR tracks the optically dense area of the tentacle attachment point (red circles). j Interestingly, if the data is downsampled by a factor of 3 (so that the training image is 50 px × 50 px instead of 150 px × 150 px) before training and evaluation, the model finds the cell as a whole.
Noctiluca scintillans, large (400 µm to 1500 µm) single-celled dinoflagellate planktons (Figures 2i-j, see also Methods, "Plankton preparation and imaging"). LodeSTAR works by identifying some characteristic feature of the object to be tracked. If this object is symmetric, the tracked feature is its center. Since the planktons have a complex internal structure, LodeSTAR does not necessarily find their geometrical center, but it nonetheless consistently identifies some specific feature of their internal structure. Furthermore, we can expect this feature to depend on the details of the neural network and its training. In fact, when LodeSTAR is trained on the inset of Figure 2i (150 px × 150 px), it learns to consistently identify the region where the tentacle attach and organelles aggregate. In contrast, when we downsample the images by a factor of 4 so that the training is made on the 50 px×50 px figure shown in the inset of Figure 2j, LodeSTAR consistently finds the plankton as a whole. In this way, it is possible to tune the scale of the tracking performed by LodeSTAR.

Exploiting additional symmetries
LodeSTAR can exploit additional symmetries to extend the range of particle properties that it can measure. Off-axis holography is a prime example of an imaging modality ripe with additional symmetries [14]. Unlike ordinary brightfield microscopy where only the intensity of the incoming wave is imaged, off-axis holography permits access to the entire complex electromagnetic field. Consequently, it is possible to infer quantitative information about the imaged objects by measuring and manipulating the Fourier representation of the image. As we will see in the following sections, when analyzing holographic microscopy images, we can exploit the Fourier propagation symmetry to detect the axial position and the signal strength scaling symmetry to determine the particle polarizability. Implementation-wise, additional symmetries are encoded as additional channels of the intermediate feature-map, and are trained using their own set of equivariances in the same manner as for the in-plane position we have discussed in detail until now.

3D tracking exploiting Fourier propagation symmetry
A natural extension of two-dimensional tracking is to incorporate the third axial dimension, normal to the imaging plane. This can give crucial insight into the full volume dispersion of objects, as well as provide more data to calculate statistical measures about the objects' motility, such as their diffusion. An example of (the imaginary part) of an holographic image a Imaginary part of the hologram of a 228 nm radius polystyrene particle numerically Fourier-propagated to different axial distances from the focal plane. b LodeSTAR exploits this Fourier propagation symmetry to learn how to track particles in three dimensions (orange positions). c The vertical position estimated by LodeSTAR agrees well with the expected position acquired using a traditional approach described in [10]. See also Video 7.
d The distributions of the in-plane xy-diffusion (blue histogram) and the axial z-diffusion (orange histogram) of the particles show a strong peak at at the expected diffusion (dashed black line), and agree well with the expected theoretical distribution obtained by calculating the diffusion constant of 10 4 synthetic traces (solid black line).
of a particle (a 228-nm-radius polystyrene sphere) is shown by the top slice (z = 0 µm) in Figure 3a. The holographic image can be propagated to different planes (i.e., different axial positions from the focal plane) by employing Fourier transforms, as shown by the other slices in Figure 3a. This provides an equivariance that LodeSTAR can learn, similar to the equivariances in the plane. By training on the image in the top slice of Figure 3a, LodeSTAR learns to track the polystyrene spheres in 3D space, as shown in Figure 3b, where the measured vertical position is visualized as the distance above the image. Note that the refractive indices of the medium (n medium ) and the immersion oil (n oil ) change the apparent axial position of the particle [15]. We correct this by multiplying the measured axial position by a factor n oil /n medium = 1.128 [15]. As a verification, we compare the predicted vertical position with a traditional focusing approach where the vertical position is found by iteratively refocusing a region near a detection until a focusing criterion is met [10]. As shown in Figure 3c, the two methods are in very good agreement. Moreover, we calculate the standard deviation of the error of the two methods based on the covariance estimate proposed in Ref. [16]: LodeSTAR outperforms the classical approach significantly, reaching σ z = 105 nm compared to σ z = 231 nm.
As a second verification, independent of the traditional approach, we compare the predicted diffusion constants of each particle trace, calculated either from its in-plane or axial movement. Since the diffusion constant is a statistical measure of the thermal motion of the particle, the diffusion of individual particles in different directions may not agree, while the ensemble of all particles should. As such, we compare the distribution of measured diffusion constants in Figure 3d for in-plane movement and axial motion. We find that both in-plane diffusion and axial diffusion are distributed similarly, and that both show a peak at the expected diffusion of D SE = 0.97 µm 2 s −1 . Moreover, we find that the distributions closely agree with the expected distribution of diffusion constants obtained by simulating 10 4 traces.

Particle polarizability exploiting signal strength symmetry
The holographic image of a particle carries information not only about the particle position, but also about its morphology and composition. For example, the real part of the polarizability of an object (which henceforth will be referred to as just the polarizability for convenience) is proportional to the integrated phase acquired by the light passing through the particle, which is particularly relevant for biological objects, where refractive index and density are strongly correlated. In other words, for biological materials, it is possible to directly translate the polarizability of a particle to its mass [17]. The integrated phase, in turn, increases roughly proportional to the amplitude of the scattered light for small non-absorbing particles. Thus, there is an equivariance between the polarizability and the scale of the signal that LodeSTAR can learn. Figure 4a shows the real and imaginary parts of a simulated holographic image of a particle with radius 228 nm and refractive index 1.5. By multiplying these images by a factor (Figure 4b), we can alter the signal scale and, therefore, the polarizability of the imaged particle. LodeSTAR is trained to estimate the logarithmic difference between the scale factors, in addition to the particle position. We remark that this equivariance does not constrain the absolute scale of the polarizability, which thus needs to be calibrated against some observation of known polarizability.
In Figure 4c, we evaluate the trained LodeSTAR on simulated particles, varying their radius and their refractive index. The mean absolute percentage error remains below 10% for the majority of the considered range, only increasing for very low signal observations where noise would corrupt most of the signal. This is comparable to the findings of other methods, and yields accurate determinations by averaging over several observations of the same particle [18,10]. In Figure 4d, we validate LodeSTAR's ability to measure the polarizability of particles in experimental data. We consider a bi-dispersed sample of 150 nm radius and 228 nm radius polystyrene particles imaged through a holographic microscope. LodeSTAR is trained on one observation from the 228 nm population and is subsequently used to predict the polarizability of all particles in the sample. By calibrating against the 228 nm population, LodeSTAR successfully identifies the 150 nm population with high accuracy.
Having verified that LodeSTAR can reliably determine the polarizability of particles, we consider a biological sample with 225 nm radius green-fluorescent polystyrene beads that were incubated with human neuroblastoma cell from the SH-SY5Y cells-line (Figure 4e, see Methods, "Human neuroblastoma cell sample preparation"). The particles are simultaneously imaged with sample-position-modulated holography and fluorescence (see Methods, "Holographic imaging"). This allows us to classify detections as polystyrene particles (orange markers, which are fluorescent) or biological aggregates (blue markers, which are not fluorescent). As can be seen in the zoomed-in region in Figure 4e, the signal is extremely low compared to the background. As such, one can expect spurious detections. We filter these out by only considering observations that could be linked over time for at least 40 frames, disregarding observations that arose from random noise. We find that the majority of detections are co-located with the cells. Further, studying the time-series (Video 8) the particles move in unison with the cells, supporting the premise that some of the particles have been taken up by the cells. See also details in Methods, "Measuring particle polarizability".
Finally, we study the distribution of polarizability for the particles and the biological matter in Figure 4e. We find a much broader distribution for the polystyrene particles than the bi-dispersed case, which is expected if some of the particles are measured inside of the cells, since the intracellular medium has a higher refractive index than the surrounding medium (1.38 compared to 1.33 [19]) and consequently yields slightly lower polarizability of the particle. Moreover, particles inside the cells are likely to be coated with biological material, further broadening the distribution. The peak of the distribution aligns well with the expected polarizability of PS inside the cell, given a cytoplasmic refractive index of 1.38 [19]. However, a second peak near the polarizability of PS in water suggests that a significant portion of the particles have not entered the cells, which agrees with other cell-particle uptake experiments [20].
The distribution of polarizability of biological particles is significantly narrower and peaks at just under 0.01 µm 3 . It should be noted that the detection of objects with polarizability lower than 0.006 µm 3 is not reliable due to the low signal, which means that the lower end of the distribution should be analyzed with some caution. Regardless, the two distributions are clearly distinct, indicating that they represent two separate physical properties of the sample.

Discussion and conclusions
We have developed a method, named LodeSTAR, that exploits the inherent symmetries of a problem to enable label-free training of neural networks using tiny datasets. We have demonstrated this capability by training neural networks to track objects in a broad range of simulated and experimental scenarios. Moreover, we have shown that LodeSTAR can quantitatively measure the objects in terms of their axial position and their polarizability. The software together with the source code and all the examples in this work are made publicly available through the DeepTrack 2.1 GitHub repository [21].
Compared to traditional approaches, we are able to achieve a better tracking performance in terms of sub-pixel accuracy, while generalizing to more arbitrary morphologies. Moreover, unlike established methods for object tracking using deep learning, which commonly require hundreds if not thousands of annotated images for training [2], LodeSTAR successfully tracks difficult experimental data using just a single datapoint for training. On top of this, LodeSTAR can be trained on an ordinary computer with no hardware acceleration (e.g., graphics processing units) within a few minutes.
As a novel approach to an established problem, we see several future lines of inquiry. Foremost, we expect the existence of many additional symmetries not considered in this work, particularly symmetries in the experimental design (such as the symmetries that allowed us the measure the axial position and polarizability of nanoparticles). We also expect that incorporating techniques from other areas of deep learning to be fruitful; for example, using active learning to optimize performance from low amounts of human supervision. Regarding the development of the technique itself, we consider the development of techniques to further improve the specificity of the model, such as leveraging negative labels to indicate what not to track. Finally, LodeSTAR may be used for segmentation, similarly to the results of achieved with the DINO (self-distillation with no labels) algorithm [22], leveraging the fact that only regions within the object are informative to its position.
The comparably low barrier of entry permits rapid creation of a custom tracking solution, requiring little-to-no expertise from the user. Additionally, by side-stepping the need for synthetic data, LodeSTAR opens up the possibility to train high-quality models to analyze data that is difficult to reproduce synthetically, without relying on fallible human annotation.

Theory of geometric self-distillation
To ensure that the training converges to the physically correct solution (e.g., identifying the centroid of the particle), the symmetry group used to train LodeSTAR needs to have a set of properties. We will describe them here.
Let X be the set of possible input images, Y be the set of possible outputs, f and h be functions h : X → Y , and G be a group of transformations acting on both X and Y . (In particle tracking, X is all possible images of the particle, Y is all possible particle positions R 2 , f is the ground-truth function that returns the particle centroid, h is the neural network that need to be trained to return the particle centriod, and G is the Euclidean group consisting of translations, rotations, and reflections. ) We can show that f is necessarily identical to h for ∀x ∈ X, if the following set of assumptions hold: (1) f (gx) = gf (x) and h(gx) = gh(x) ∀g ∈ G, x ∈ X -i.e., both f and h are equivariant to G. (2) For any x 1 and x 2 in in X, there exists a g ∈ G such that gx 1 = x 2 (known as transitivity in group action theory). (3) There exists two distinct g 1 , g 2 ∈ G such that g 1 x = g 2 x for some x ∈ X, i.e., G is not free. (4) There exist an x ∈ X and a g ∈ G that fixes x for which f (x) is the only fixed point. (In particle tracking: (1) asserts that the centroid of the particle (f ) and the neural network (h) are equivariant to the Euclidean group. f is true by definition, h is true by training. (2) asserts that the image of a particle in one position and orientation should be transformable to any other position and orientation using a combination of Euclidean transformations. (3) asserts that there exists some non-trivial transformation that leaves the image of the particle unchanged. For example, rotating a square 90 degrees does not change it. (4) asserts that there exists a transformation g, such that gx = x and gy = y hold if and only if y is the centroid of the particle whose image is x.) Since per (1) h is equivariant, h(gx) = gh(x). Further, due to (3), gx = x, and consequently h(gx) = h(x). Together, we find that gh(x) = h(x). Finally, due to (4), there exists a g and x such that g y = y is true only if y = f (x ). Consequently, g h(x) = h(x) can only be true if h(x ) = f (x ). Now, consider some arbitrary g ∈ G. We find that f (gx ) = gf (x ) = gh(x ) = h(gx ), where the first and third equality signs are given by equivariance. As such, if f (x ) = h(x ) for some x , then f (x) = h(x) for all x reachable by acting g on x. Due to (2), all x ∈ X are reachable from x through actions from G. Consequently, f (x) = h(x) for all x ∈ X.
Assumptions (1) and (2) ensure that the network will consistently track the particle, while (3) and (4) ensure that exactly the centroid is tracked. As such, even if (3) does not hold (as is the case for asymmetrical particles), the particle will still be tracked consistently.

Neural network architecture
The neural network consists of three 3 × 3 × 32 convolutional layers with ReLU activation, followed by a 2 × 2 max-pooling layer, followed by eight 3 × 3 × 32 convolutional layers with ReLU activation, and finally by a single 1×1×3 convolutional layer with no activation. Splitting the output channels into ∆x, ∆y and ρ respectively, LodeSTAR first calculates where N is the size of the input along the first dimension, and M is the size of the tensor along the second dimension, and k is a scale factor relating the size of the input to the output (k = 2 for the proposed architecture). During runtime, the weights are normalized as where S(·) is the sigmoid function used to constrain individual elements between 0 and 1, which makes choosing a detection threshold for multi-particle tracking easier.

Neural network training
During training, the weights are normalized differently, namely: where is some small value (10 −6 ), D[·, 0.01] is a dropout with a dropout-rate of 1 %. The dropout avoids the solutions where a single element is large and the rest are small, increasing the robustness of the network. The epsilon adds non-linearity during training, forcing the network to utilize the entire span of the sigmoid function. Further, the network strives to minimize the weight of non-informative regions. The only way to achieve this is to increase the normalization factor W i,j , which is maximized by predicting large values at informative regions. During evaluation, the weight is not normalized, in order to make it independent of the number of objects in view.
We employ two loss functions during training. The first is a loss between the untransformed predictions and their mean. The second is an internal consistency loss, which, for the case of the particle centriod prediction, is calculated as In the case where additional features are predicted (e.g., z-position and polarizability), the corresponding loss term is calculated in the same way as L x i,j and L y i,j . This auxiliary loss ensures that the prediction is internally consistent, and is what causes the clustering of the prediction. Note that if the model predicts ρ = 1 for one position and 0 everywhere else, this metric would be 0. As such, it encourages a tight spatial weight distribution, which is useful for separating closely packed objects. However, the dropout in the weight during training ensures that it does not collapse to a single pixel. Without this loss, LodeSTAR would occasionally ignore the feature maps, and instead tune the weights such that only the average is correct.
In all cases, LodeSTAR was trained using the Adam optimizer [23], with a learning rate of 0.001. Unless otherwise stated, the model was trained using 5000 mini-batches of 8 samples. The exception is for the 3D tracking described in Figure 3, where the model is trained on 15000 mini-batches.

Particle detection criteria
Particles are detected by finding local maxima in a score map using the function h maxima of the Python module skimage. The score map is based on two separate metrics. The first is the weight map w. The second is a clustering metric calculated as where P is the feature map produced by the network. This can be likened to convolving the feature map with a variance kernel. The two metrics are combined geometrically as w α c β , where α and β can be tuned to optimize performance. In the cases presented in this paper, we consistently use β = 1 − α. For the cases in the chapter "Validation with experimental data", we use, in order, alpha = 0.2, alpha = 1, alpha = 1, alpha = 1. For the remaining cases, we use alpha = 0.1 Local maxima in this product are taken as observation if they are larger than some threshold, which in turn is chosen as a quantile of all scores. The quantile can be chosen a priori based on the density of the experimental data, but it can commonly be taken as the 99 th percentile.

Plankton preparation and imaging
Noctiluca scintillans (SWE2020) was isolated from the Swedish west coast in November 2020. The culture was maintained in 16 • C, 26 psu, and 12:12 h light:dark cycles. The culture flasks were shaded by a screen to limit growth of the food organisms, Dunaliella tertiolecta. Noctiluca cultures were fed ad libitum 2-4 times per month. The planktons are imaged with an inline holographic microscope, illuminated with a LED source (Thorlabs M625L3) of center wavelength 632 nm (details in [24]). The images are recorded with a CMOS sensor (Thorlabs DCC1645C) placed at a distance of ≈ 1.5 mm from the sample well, at 10 frames per second and with an exposure time of 8 ms.
Human neuroblastoma cell sample preparation SH-SY5Y cells were grown in cell culture media (CCM) containing a 1:1 mixture of minimal essential medium (MEM) and nutrient mixture F-12 Ham supplemented with 10% heatinactivated fetal bovine serum, 1% MEM nonessential amino acids, and 2 mM l-glutamine. The cells were detached (trypsin-EDTA 0.25%, 5 min) and passaged twice a week. The cells were tested and verified mycoplasma-free. Cells were plated 1 day prior to experiments in glassbottomed culture dishes (MatTek; 25000 cells/14 mm glass region) for microscopy. Cells were washed 1x with serum-free CCM before exposure to 225 nm green fluorescent polystyrene particles diluted in serum-containing CCM with the addition of 1% Penicillin-Streptomycin. After a 4 hour incubation at 37 • C 5% CO2, the cells were washed twice for 2 minutes with serumfree CCM, followed by addition of serum-containing CCM supplemented with 1% Penicillin-Streptomycin and 30 mM HEPES to buffer the medium. The cells were imaged at 37 • C using a OKOlab stage top incubator.

Holographic imaging
The used monodisperse particles are and 0.15 µm (modal radius) polystyrene (Invitrogen) and 0.23 µm (modal radius, NIST-certified standard deviation ±6.8 nm) polystyrene (Polysciences) (sizes verified using nanoparticle tracking analysis performed by NanoSight). Samples were imaged under flow in straight hydrophilized channels with a height of 20 µm and a width of 800 µm in chips made from Topas (COC, ChipShop). The images were captured using a offaxis holographic microscope [10], using a 633 nm HeNe laser (Thorlabs) and a Olympus 40× 1.3 NA oil objective. The interference pattern was collected using a CCD camera (AlliedVision, ProSilica GX1920), at a frame-rate of 30 frames per second, and an exposure time of 2 ns to 4 ms. For the intracellular data, a 40×, 0.95 NA Objective (Nikon, CFI Plan Apokromat) objective was used instead. The particles consisted of 225-nm-radius green-fluorescent polystyrene spheres (PS-FluoGreen, microparticles Gmbh). The particles were diluted 5000 times in cell media from the stock solution concentration of 2.5 wt% to a concentration of 5.25 µg ml −1 . Further, to improve the data aquisition quality, the stage was oscillated at 1.1 µm roughly every 4 seconds and offset pairs of frames were subtracted from each other to mitigate noise from reflections. The sample was imaged at 3 frames per second.
The fluorescence arm was illuminated using a 465 nm LED excitation (CoolLED), and imaged using a ORCA-Flash 4.0 V2.0 CMOS camera (Hamamatsu). We filtered the signal using a 491 nm dichromatic mirror (Chroma Technology Corporation), and separated the fluorescence channel from the holography channel using a long-pass dichroic mirror (605 nm), as well as a 525 ± 39 nm bandpass emission filter from Thorlabs.

Measuring particle polarizability
Particles were detected in each frame using a neural network trained by LodeSTAR using a single observation. The observations were subsequently traced over time by linear sum assignment, with a distance threshold of 1 µm. The polarizability was averaged over measurements from each detection in a trace. The LodeSTAR-measured polarizability was calibrated against a known population as α = 3V n 2 p −n 2 m n 2 p +2n 2 m , based on the Clausius-Mossotti relation. Further, for the intracellular data, each detection was compared to a fluorescence channel. If the detection was within 400 nm of a fluorescence detection, it was considered a polystyrene particle. If > 95% of detections in a trace were linked to a fluorescence detection, the trace was considered a trace of a polystyrene particle. If < 5% of detections in a trace were linked to a fluorescence detection, the trace was considered a trace of an intracellular particle.

Supplementary Videos Video 1
LodeSTAR successfully tracks particles of various shapes, orientations and noise.

Video 2
Mouse hematopoietic stem cells are tracked by LodeSTAR as they divide. The score map used for detection is also shown.

Video 3
Human hepatocarcinoma-derived cells are tracked by LodeSTAR. The score map used for detection is also shown.

Video 4
Pancreatic stem cells are tracked by LodeSTAR as they divide. The score map used for detection is also shown.

Video 5
Noctiluca scintillans are tracked by LodeSTAR. LodeSTAR detects the optically dense area near the ventral groove where the nucleus is located. The score map used for detection is also shown.

Video 6
Noctiluca scintillans are tracked by LodeSTAR. LodeSTAR detects the plankton as a whole. The score map used for detection is also shown.

Video 7
Polystyrene particles under flow are tracked by LodeSTAR in 3D.

Video 8
SH-SY5Y cells with polysyterene particles inside and around the cell. LodeSTAR detects both the polystyrene (orange markers) and intracellular particles (blue markers).