{'fraction_non_alphanumeric': 0.03291284091218701, 'fraction_numerical': 0.013050704302100646, 'mean_word_length': 5.257074667575861, 'pattern_counts': {'":': 0, '<': 1, '<?xml version=': 0, '>': 6, 'https://': 1, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Automated Deep Aberration Detection from Chromosome Karyotype Images

Chromosome analysis is essential for diagnosing genetic disorders. For hematologic malignancies, identification of somatic clonal aberrations by karyotype analysis remains the standard of care. However, karyotyping is costly and time-consuming because of the largely manual process and the expertise required in identifying and annotating aberrations. Efforts to automate karyotype analysis to date fell short in aberration detection. Using a training set of ~10k patient specimens and ~50k karyograms from over 5 years from the Fred Hutchinson Cancer Center, we created a labeled set of images representing individual chromosomes. These individual chromosomes were used to train and assess deep learning models for classifying the 24 human chromosomes and identifying chromosomal aberrations. The top-accuracy models utilized the recently introduced Topological Vision Transformers (TopViTs) with 2-level-block-Toeplitz masking, to incorporate structural inductive bias. TopViT outperformed CNN (Inception) models with>99.3% accuracy for chromosome identification, and exhibited accuracies>99% for aberration detection in most aberrations. Notably, we were able to show high-quality performance even in"few shot"learning scenarios. Incorporating the definition of clonality substantially improved both precision and recall (sensitivity). When applied to"zero shot"scenarios, the model captured aberrations without training, with perfect precision at>50% recall. Together these results show that modern deep learning models can approach expert-level performance for chromosome aberration detection. To our knowledge, this is the first study demonstrating the downstream effectiveness of TopViTs. These results open up exciting opportunities for not only expediting patient results but providing a scalable technology for early screening of low-abundance chromosomal lesions.


Introduction
Precision medicine for hematological malignancies typically hinge upon the karyotype results, in which large genomic signatures are identified by arresting, staining, and imaging chromosomes in metaphase. Karyotyping assesses the whole genome in an unbiased, albeit low resolution manner from single metaphase cells, and is the gold standard for clinical testing of chromosome abnormalities, including numerical changes, like gains and losses of whole chromosomes, and structural aberrations, such as translocations, inversions, deletions, insertions duplications and amplifications. Its versatility has made karyotyping vital for diagnosis and management decisions not only for somatic, clonal aberrations in cancer, but also for inherited genetic disorders with constitutional abnormalities.
However, the current karyotype analysis process is labor-intensive, taking a skilled technologist many hours of hands-on analysis work per specimen. Real-world time constraints have made it standard practice to analyze only 20 metaphase cells per specimen even though hundreds more are typically available. Even when limiting the analysis in this way, cost and turn-around time for diagnosis remain an issue. Since expert analysis relies largely on visual pattern recognition, and all samples from the same species share an expected baseline, this task is ideally suited for deep-learning.
Since the first application of artificial intelligence in the form of convolutional neural networks (CNNs) 1,2,3 for human chromosome classification in 2004 4 , the accuracy has improved significantly 5,6,7,8 . Most recently, Vajen et al. applied CNNs successfully to classify normal R-banded chromosomes with an accuracy of 98.8% 5 . Aberration detection has remained a challenge as clonal chromosome abnormalities in hematological malignancies are highly complex, diverse, and sometimes rare. Together with the relative data scarcity challenge in biomedical data (compared to natural image datasets), it is not a given that the latest advances in CNNs translate to biomedical applications. For example, inspired by the success of Transformers for natural language processing (NLP), the Vision Transformer (ViT) model was the first proposal that efficiently scaled Transformers for imaging applications 9,10 . On medium to large dataset sizes, ViTs outperformed CNN-based approaches 9,10 . However, small data regimes have remained challenging compared to CNNs, in part because the lack of relative pixel location in the canonical ViT attention mechanism makes local image substructure harder to learn. This paper utilizes modern Transformer-based deep learning approaches to provide high-accuracy for both chromosome identification and aberration detection, starting from individual chromosome images. In particular, we utilize a Transformer-encoder with masked attention to learn correlations between pixels in the "normal" image, then fine-tune the learned model for anomaly detection. The masked attention effectively modulates the attention layer by a learnable function of the L1-distance between pairs of patches in the unflattened image. We test this approach on a diverse set of chromosome aberrations (an intra-chromosomal unbalanced abnormality: del(5q); intra-chromosomal balanced rearrangements: inv(3) and inv (16), and inter-chromosomal translocations: t(9;22), t(9;11), and t (11:19)) commonly seen in for the TopViT model for chromosomes from the test set. Each point is colored by its ground truth label with predictions that disagree with the training label enlarged 10-fold and marked with an X. D) Accuracy (left y-axis) and percentage of remaining data (right y-axis) after removing high entropy predictions as a function of entropy cutoff. As an example, removing predictions with higher than 3.6x10 -06 entropy increases the accuracy to 99.9 by removing 5% of the data.

Results
An overview of our selected modeling approach is shown in Figure 1. Individual chromosome images were curated from 42,049 karyograms (~70% normal, split into an 80% training, 10% validation, and 10% test sets) prepared from 9,207 bone marrow and peripheral blood specimens (94.5% and 5.5% respectively) of 4,711 patients with hematological malignancies. These chromosome images were first partitioned into "patches" and flattened into a linear vector before being passed into a Transformer network for the pretraining task of predicting individual chromosomes. Multiple Transformer-based variants were tested (Figure 2A), primarily differing in the nature of the self-attention encoder blocks. The first variant utilized the standard ViT attention as described previously 10 . The second variant used a memory and time efficient Performer-based attention 11,12 . The final ViT variant, TopViT, uses a recently published topological masking strategy to encode distances between patches in the original unflattened image, thereby preserving the 2D-topology of the original chromosome image 13 . The highest accuracy model for chromosome identification was then fine-tuned to produce individual models for each aberration-specific, "normal"/"abnormal" classification task.

Chromosome Identification
The pretraining task for normal chromosome identification contained roughly equal counts for all chromosomes (with the exceptions of X and Y). Figure 2A shows performance of the three ViT variants described in Figure 1 and a CNN-based (InceptionV3 14 ) model. In each case, a variety of hyper-parameters were explored with the models ranked by their best performance during training on a validation set (Figure 2A, individual points). All ViT-based strategies substantially outperformed the Inception-baseline, and only a small reduction in performance was observed when transitioning from standard ViT to the more efficient Performer-based models (Figure 2A, blue and orange). The TopViT model (Figure 2A, green) yielded far more performant models (a nearly 50% reduction in errors on average compared to the Inception baseline) and was selected for all subsequent analyses. This model successfully classified normal chromosomes with an accuracy of 99.3%.
We next examined the distribution and nature of errors for the optimal parameter TopViT model. Figure 2B shows the confusion matrix of predicted vs. original labels. Most discordant predictions were between chromosomes of similarly-sized chromosomes and balanced (roughly symmetric around the diagonal in Figure 2B). While some of the more frequently swapped predictions are consistent with those one might expect of a human practitioner due to similarities in appearance (such as chr4/chr5 and chr21/chr22), others (such as chr16/chr19 and chr20/chr22) seem to be specific to the model. To better understand the relationship between discordant and concordant predictions, we created a UMAP embedding of the last layer of the TopViT model for all test examples ( Figure 2C). Clustering of certain discordancies suggested the same underlying mechanism for systematic swaps seen in Figure 2B. Generally, chromosomes were fairly well separated and discordancies were enriched at the periphery (or outside) of chromosome clusters, suggesting that the model considers these less canonical representations of the predicted chromosome. To explore this, we computed the entropy of the 24-class probability vector for all predictions ( Figure 2D). Concordant predictions had much lower entropy than the discordant predictions indicating that the model had significantly higher uncertainty for discordancies. When introducing an entropy-based filter (< .00001) we were able to substantially improve the calling accuracy to 99.9% while only dropping 3% of the data (Supplemental Figure 1).
We then performed expert, manual re-examination of all discordancies (both low and high-entropy). Most (72%) of low-entropy discordancies were actually mislabels in the training set, while most (90%) of high-entropy discordancies were due to low-quality data (cross-overs, folded or bent chromosomes, poor banding, poor chromosome morphology, and cellular debris (Supplemental Figure 2). Dropping all low-quality chromosomes and correcting mislabels in the test set led to a revised accuracy of 99.95%.

Chromosomal Aberration Detection
We next employed the method for the more challenging task of aberration detection. To evaluate the utility of the model in this revised context, we first employed two of the most common, medically relevant aberrations in our cohort: del(5q) and t(9;22) ( Table 1). Deletion of the long arm of chromosome 5 [del(5q)] is associated with MDS and other myeloid neoplasms. To start, we used the optimal model architecture (TopViT) with randomly initialized weights and retrained a two-class (normal/abnormal) model for both aberrations ( Figure 3B). While we observed reasonable performance for del(5q) and the derivative chromosome 22 of t(9;22), i.e. the Philadelphia (Ph)-chromosome, the model exhibited weaker performance for the derivative 9 chromosome from t(9;22) [der(9)] ( Figure 3B, rightmost green points in each subplot). Next, we reran the experiment but initialized the model with weights from the chromosome identification model and then fine-tuned the model for aberration detection (as described in Figure 1A and Methods). In all cases performance improved, with der(9) AUC's becoming comparable to that of del(5q) and der(22) ( Figure 3B, rightmost purple points in each subplot). To better understand the intuition for this, we visualized embeddings of healthy and abnormal chromosomes 5, 9, and 22 before fine-tuning ( Figure 3A). As expected, while der(9) was more mixed, del(5q) exhibited some separation prior to fine-tuning, indicating that the latent embedding space captured features sensitive to the missing portion of chromosome 5.
To model low-frequency aberrations, we downsampled the number of abnormal examples for these three aberrant chromosome sets ( Figure 3B). Performance of all points was evaluated on the same held-out test set as that used for the complete training set. Fine-tuned models consistently maintained high AUC's even with 10s of examples, while models initialized with random weights required hundreds of training examples (or more). Notably, the Ph-chromosome [der(22)] aberration set maintained robust fine-tuning performance with only three examples, with performance comparable to the randomly initialized model on the full dataset, Figure 3B.
Given the strong performance in the downsampling task, we examined other infrequent aberrations from our patient cohort. In addition to having sufficient examples (at least 10 patients) to evaluate performance, the selected aberrations were critical for diagnosis and prognosis and represented diverse rearrangements, prevalence, and difficulty levels for microscopic assessment (Table 1) [15][16][17] . Figure 4, shows performance across the four additional aberrations tested. Given the limited sample sizes, the points here represent the average performance across a 10-fold cross-validation (split by specimen --all cells collected from a patient hospital visit). While individual performance was often lower than for the more frequent del(5q) and t(9;22) aberrations, ROC AUC was high across all aberrations and median and mean AUC PR exceeded 0.8 and 0.7, respectively, across all cases ( Figure 4A,B).
We extended the analysis to simultaneously test for their partner chromosomes (i.e., the other member of the pair involved in a translocation) as well as aggregating within cells (purple) and across all cells (blue/green) from a patient specimen ( Figure 4C and Supplemental Figure 3). We employed simple aggregation techniques (See Methods) with either the max probability score across examples or the second best probability score to better model clinical practice of requiring two aberrant cells to define a clone 18 . Substantial improvement was seen in both cases, with the resulting specimen level predictions showing zero false positives with full or near full recall ( Figure 4C). We then applied this same strategy back to the original del(5q) and t(9;22) patient cohorts ( Figure 4C). As before, a notable improvement was observed in both aberrations. In particular, when accounting for both chromosomes in the t(9;22) translocation we were again able to achieve 100% precision and recall ( Figure 4C, right plot, green line).
As a final step we attempted to identify aberrant chromosomes de novo. For each chromosome we calculated the euclidean distance between its embedding vector and all embeddings vectors from normal chromosomes in the training set, and then selected the distance to the nearest neighbor in the training set. PR-curves generated from these distances ( Figure 4D, Supplemental Figure 4) had performances comparable to fine-tuning across all aberrations ( Figure 4C, Supplemental Figure 3). In particular, del(5q) and t(9;22) returned perfect accuracy, while two of the other abnormalities (inv(3) and t (11;19)) showed 100% precision with >90% recall. Each boxplot corresponds to a distinct cross-validation sets for each chromosome involved in an aberration and is colored by aberration. Individual points for folds and averages (black diamond) are overlaid on each boxplot. (C) Precision-recall curves for t(9;11), t (11;19), del(5q), and t(9;22), at the individual chromosome image level (orange) or aggregated at the cell`(purple) or specimen levels. For specimen level >= 1 abnormal (blue) the single highest probability abnormal chromosome was used, for specimen level >= 2 abnormals (green) the second highest probability abnormal chromosome was used. Similarly, (D) shows precision-recall for de novo aberration detection based on distance to N-nearest point (here 50th) for t(9;11), t (11;19), del(5q), and t(9;22), respectively. ** One patient with t(16;16) was not included in the analysis data set for inv (16).
*** These translocations were each seen in fewer than 10 patients and therefore not formally analyzed in this study. † "Balanced" refers to aberration being apparently balanced during microscopic assessment † † Depending on concurrent abnormalities § Also known as APL § § Chromosomes

Discussion
Here, we present a novel deep learning model utilizing Topological transformers that yield high-quality models for chromosome aberration detection. This work not only provides highly accurate models that further improves the accuracy for chromosome identification (with 99.3% accuracy) but also demonstrates a practical approach for detection of chromosomal aberrations with accuracies between 97.8% and 99.4%. These quantitative accuracy improvements ( Figure  2) are substantial but only tell part of the story. Qualitatively, the models handled low-quality input; i.e., metaphase chromosomes at 400-band level or less from bone marrow and leukemic blood specimens. Of the 9207 total specimens used in this study, 94.5% were bone marrow aspirate and/or bone marrow biopsy specimens while 5.5% were peripheral blood from patients with hematological malignancies. In contrast, previously published studies used phytohemaglutinnin-stimulated peripheral blood lymphocytes, amniotic fluid samples, or normal cells from bone marrow specimens, which generally yield chromosomes with better morphology and higher band resolution. We are encouraged that our models were able to handle more challenging samples with variable sample quality. Moreover, expert clinicians evaluating the most performant, TopViT, models reported a substantially better experience. While the InceptionV3, ViT and XViT-P models missed a number of "easy" cases, discordancies in the TopViT model were primarily due to low-quality input or mislabeling in the original training set -to the extent that identifying if the model (or training set) was in error became increasingly challenging suggesting near expert-level performance.
The set of abnormalities tested represented not only a diversity of chromosomal rearrangements and prevalence but varying levels of difficulty. For example, while most cytogenetic technologists have no problem recognizing a del(5q), those without specific training in cancer cytogenetics can easily miss an inv (16). Interestingly, the models showed a similar trend to humans (Figure 4, Supplemental Figure 3). To our knowledge, this is the first deep-learning tool that is capable of detecting any chromosome abnormalities. Additionally, this large, pre-training task proved immensely powerful. In addition to substantially outperforming equivalent models without pre-training on normal chromosomes, these models required only a small number of training examples of abnormal chromosomes (as low as 10) to make reasonable predictions (Figure 3). Moreover, when aggregating across multiple cells from the same specimen, mimicking the clinical practice in a diagnostic cytogenetics laboratory in that an abnormality is defined as clonal in nature when seen in at least 2 cells, the methods achieved perfect accuracy in multiple aberrations even in low-data regimes with only 39 abnormal examples of t(11;19) ( Figure 4C). Perhaps most surprisingly, embeddings learned by the pre-training task capture sufficient chromosome features to enable de novo aberration detection near the performance of the fine-tuned models (Supplemental Figure 4).
While we demonstrated substantial improvement of chromosome classification and aberration detection using machine learning, there are limitations to this study. An obvious next step is to apply this process on a larger corpus of aberrations. This process is non-trivial, as many aberrations are infrequent or ambiguous, making statistically rigorous per-aberration validation challenging. Additionally, prospective evaluation of patient specimens to ensure generalization is an important step. Compute and memory requirements of TopViT may also be a concern. We attempt to mitigate this in two ways: 1) Utilizing a fine-tuning strategy that enables shorter training times and smaller training datasets, and 2) Introducing more efficient, lower memory performer (xViT-P) models (Figure 2A). While moving to the xViT-P model (accessible via our github repository at https://github.com/google-research/scenic/tree/main/scenic/projects/tasseo) can lead to a degradation in performance, this trade-off may be worthwhile depending on the setting. For example, a combination of approaches may enable "real-time", interactive inference.
All current efforts on automation of karyotype analysis using machine learning remain highly dependent on karyograms, which require manual preparation 4-8 , as does this study. Although recent improvement shortened the time required for this step with commercial software 5 , further development is needed to enable direct application on metaphase. The ability to continuously add new aberrations, coupled with metaphase segmentation, these pipelines would allow ultra-fast karyotype analysis.
This study provides functionality clinicians can immediately utilize and a framework to potentially broaden karyotype usage. As one example, karyotype analysis is required for the selection of appropriate therapy for patients with AML, meaning that with the current labor-intensive approach, patients with an explosive disease often need to wait a week or more for cytogenetic results before they can start therapy. The potential rapidity of the approach described here would be of major clinical benefit. Further, because of the labor involved, currently karyotyping is generally limited to 20 cells per sample. The ability to analyze hundreds of metaphases per specimen would increase the sensitivity of the assay and reveal further details about the clonal architecture of these diseases. Specifically, rare abnormalities at a lower, subclonal, level are important driving forces for cancer recurrence after initial chemotherapy-induced remission 20 . These data, in turn, should lead to improved risk-stratification, better treatment selection, and the development of novel methods for following the course of treatment.
As we enter into the new era of genetic manipulation of hematopoietic stem and progenitor cells as treatment for a wide variety of conditions, one concern has been the potential impact of these approaches on the genetic integrity of the target cell. Several cases of leukemia have already been documented in patients receiving gene-modified stem cells. The ability to rapidly and efficiently screen large numbers of cells for karyotypic abnormalities could become an important method to improve the safety and efficacy of these new treatments. Moving forward, we believe this work can usher in a broad set of new use cases for karyotyping in the clinic.

Methods
Selection and construction of patient cohort and dataset Dataset Initial datasets were assembled using results collected from five years of clinical testing performed by the Seattle Cancer Care Alliance (SCCA) Cytogenetics Laboratory, now part of the Fred Hutchinson Cancer Center. As part of this routine testing, metaphase images captured by an automated imaging system were processed into karyograms by experienced cytogenetic technologists. These images were curated into a set consisting of 42,049 completed karyograms (approximately 70% normal and 30% abnormal) collected from 9,207 specimens by 4,711 patients. A corresponding set of diagnoses and karyotype nomenclature was obtained from the Laboratory Information Management System (LIMS) used for reporting the patient's clinical results. Each chromosome image was isolated, centered on a white background and cropped to 199x99 pixels. This study was approved by the Institutional Review Board (IRB) of the Fred Hutchinson Cancer Center.
For the chromosome identification pre-training task, patient specimens were first separated into normal and aberrant subsets, and only normal patient specimens were retained for pre-training; normal patients were those with non-aberrant karyotypes. The normal chromosome examples were then grouped by patient specimen and split into 10 folds such that all chromosomes/slides/karyograms from a single specimen were collocated in a single fold. Of these 10 folds, 8 of the folds were used for pre-training models on the chromosome identification task with 1 of the held out folds used for training validation and stopping criteria metrics. The chromosomes from the 2 folds that were not incorporated into the pre-training set (i.e., the validation fold and unused test fold for the chromosome identification task), which in this setup are all normal chromosomes, were added to the corresponding aberration training sets as healthy exemplars (i.e., negative examples relative to each aberration identification task). For example, the normal chromosome 9s from the held out pre-training folds were added to the t(9;11) aberration training set for normal vs aberrant chr9 identification and likewise for the remaining aberration datasets.
For each aberration dataset, patient specimens were included if the specimen karyotype included the aberration of interest; i.e., additional aberrations may be present in the karyotype beyond the aberration of interest as opposed to selecting patients that have only the specified aberration exclusively. By construction, patient specimens in the aberration datasets are disjoint from the chromosome identification pre-training dataset. Additional normal chromosomes from the held-out chromosome identification folds were included in the aberration training set and by construction, the patient samples in the chromosome identification pre-training task are disjoint from both the aberrant and normal patient specimens included in each aberration dataset. The combined aberrant+normal patient specimens were then split into 10 folds with 8 folds being used for training each aberration model (either fine-tuning or from-scratch) with a held-out fold used for model validation and training stopping criteria.
For the chromosome identification pre-task, models were trained for 500 epochs with a batch size of 512 examples (368,106 examples per epoch). The model with the highest accuracy on the validation set was chosen for chromosome identification task and later was fine-tuned for all aberration detection tasks. For consistency, we used the same model architecture for aberration detection tasks even in the cases that we did not use pre-training (details of the model is available in Supplemental Table 2). For training aberration models (fine-tuned or from-scratch), models were trained for 2k steps with a batch size of 512 examples independent of the epoch size, which varied among the aberration datasets. All the evaluations on aberration models are performed on the final checkpoints after the aforementioned number of steps.

Aberration annotation
To build a gold standard dataset of chromosomal aberrations to use for training each of the disease identification tasks we did the following: identify patients belonging to each disease cohort via existing diagnostician-assigned karyotype; extract chromosomes corresponding to the disease variant from an assembled karyogram; e.g., extract all derivative chr9 and chr22 for the t(9;22) cases; present each chromosome set (e.g., all chr9s) to a trained karyotyping diagnostician and have the expert label each of the chromosomes in the set according to a pre agreed upon disease classification rubric (e.g., "normal" vs "abnormal"). Upon receiving the individually labeled chromosomes, there were instances of disagreement among the expert reviewers and these were each adjudicated via group discussion to reach label consensus.

InceptionV3
We train an off-the-shelf Inception (V3) model for the baseline chromosome identification task.
We ran error mode studies to check the misclassifications of our trained models. They led us to discover that the majority of cases we got wrong were when chromosomes overlapped in the metaphase image or when the global image quality of the metaphase image was poor. To remedy such scenarios, we utilized two parallel InceptionV3 14 networks, one each for singleton chromosomes as well as the corresponding metaphase image to generate feature vectors for the singleton chromosome and its metaphase image. We use the feature vectors of the metaphase image for affine scaling of the singleton chromosome feature vector. The idea was to move/scale the feature vector of the singleton chromosome in a way such that it is classified correctly.
All our networks were trained with a learning rate of 5e-3 using the SGD optimizer with momentum. We used a batch size of 64. The feature vector sizes of the metaphase and singleton chromosome InceptionV3 branches were 4096 and 2048 respectively. The metaphase feature branch was split into two 2048 vector embeddings, one each for the scale and bias terms.

From ViT, through XViT to TopViT
For implementing image-based Transformers, we utilized the class of Vision Transformer Models (ViT) 10 and their scalable ViT-Performer 12 variants (from XViT architectures) 10 , using Scenic library, as well as the new class of Topological Transformers (TopViTs, 13 ) using github.com/google-research/google-research/tree/master/topological_transformer/ 21 TopViT-Library. We used 8x8 TPUv3 for the training and for the evaluation.
In order to find the best model, we ran detailed ablation studies over different hyper-parameters and several Transformers' architectures. ViT The initial transformer model was based on the first ViT implementation by Dosovitskiy, et al 10 . Briefly, this model divides an image into fixed-size non-overlapping patches. Each of the patches is then transformed into a latent embedding space and the linearized sequence of feature vectors obtained in this way is then fed to a standard Transformer encoder where the embeddings of patches are contextualized with respect to their relationships, measured by the explicitly materialized attention matrix. Running time and memory consumption are quadratic in the number of patches, therefore changing the patch size drastically affects the overall compute cost.
Performer (from XViT) Performers' architectures 12 address quadratic space and time complexity of the attention module in regular Transformers' architectures by introducing an implicit attention computation model. In that model, attention is never explicitly materialized, but its low-rank decomposition is applied to emulate attention-matrix action on the lower-rank value tensor. Different low-rank decompositions lead to attention matrices defined by different kernels. The most notable Performers' variants include: Performer-ReLU, Performer-Square and Performer-softmax. The first two apply kernels defined as dot-products of pointwise-transformed queries/keys, where the transformation is defined by the ReLU or Square mapping. Those are the fastest Performers since they do not require random projections to construct kernel features. Performer-softmax is capable of unbiasedly emulating the attention matrix with randomized kernel features (leveraging random feature map mechanism) and thus it requires additional random projections.

Topological Transformer (TopViT)
Our best-accuracy models for chromosome identification as well as aberration detection are TopViTs.
The general concept of the topological masking for Transformers, leading to TopViTs, was recently introduced 13 . The idea is to modulate the core component of Transformers: regular attention matrix modeling relationships between different input tokens (in our case: different patches of the input image) via the masking matrix M with entries encoding learnable functions of the distances between the tokens in the underlying graph. Critically, the distance that is used is defined on the original graph of the input rather than its flattened representation (as it would be the case if the so-called regular relative positional encoding mechanism is applied). This is very important since distances introduced on the flattened representations lead to arbitrarily large distortions. For the vision input data that means that consecutive patches in the image might translate to distant tokens in the flattened sequence and distant patches in the image might translate to neighboring tokens in the flattened sequence. This is what topological masking avoids. The M-induced attention matrix A modulation is given as A M, where ⊙ ⊙ stands for the Hadamard (element-wise) product.
Our TopViTs apply L1-distance on the 2D-grid underlying canvas of the image with nodes of the grid corresponding to different patches and edges encoding patch-neighborship relationships. As explained in 13 , such a mechanism leads to the so-called 2-level block Toeplitz masking matrices. The attention kernel is inherited from the Performer-ReLU which turned out to be the most effective among various Performer-models for our tasks. We applied different learnable masks for different layers as well as different heads within a layer. The particular 2-level block Toeplitz structure of the mask combined with the decomposable (low-rank) Performer-ReLU attention makes TopViT computationally-compatible with regular Performers -the result of the attention module can be computed in time log-linear in the number of patches (Performers' time complexity is linear) rather than quadratic (as it is the case for the regular Transformers). Interestingly, this does not incur any approximation error since deterministic kernel features are applied.
The mathematical description of our core attention module for the input query/key/value tensors Q / K / V is given below: where: , stand for the ith/jth row of Q/K respectively (ith/jth query/key row-vector), is the all-ones 1 vector of length L (where L stands the number of all the patches), is a learnable function and stands for the -distance between the ith and jth token in the 2-dimensional grid 1 ( , ) 1 encoding the canvas of the image. Here, as already explained above, symbol denotes the ⊙ Hadamard (element-wise) product of matrices. Note that since the number of different 1 -distances is finite, is effectively a learnable vector. Note also that mask M is a 2-level block Toeplitz matrix under an appropriate ordering of all the tokens (for instance induced by the natural ordering of the 1-dimensional slices of the 2-dimensional canvas) 13 .

Downsampling
To evaluate the performances of our models under data scarcity, we downsampled the number of abnormal examples in the training datasets of del(5q) and t(9;22) then trained the aberration detection models on these sets. To generate the downsampled sets, we fixed the normal examples and randomly picked a subset of abnormal samples from the full training sets. We repeated the process five times for each aberration and number of abnormal examples to increase the reliability of the results.

Cross-validation aggregation
The dataset consists of chromosome images of each patient , such that for each patient we get a set of images . For 10-fold cross validation we partition the data in 10 = { 1 , 2 , ...} parts such that the parts are mutually exclusive and, for each fold, no sample observed in a test set contains images from that same sample in its training set. For each fold the model provides as output the prediction probability, , of an image being aberrant. For Figures 4A and 4B  ( ) we calculate the ROC AUC and ROC PR on the test set for each of the 10-folds.
To further analyze the performance across different data points we take a union of the probabilities on the test set across the folds. This gives us the probability of aberration across the entire dataset. Note that since each datapoint only appears as a test once, they are assigned probability by one model. This helps us in calculating the precision recall curve across the entire dataset. This is reflected in the "chromosome level curve" in Figures 4C, 4D, 4E and 4F. We then examined predictions at the cell (all chromosomes from a single cell), and specimen (all chromosomes from a single patient visit) levels. In both cases we first employed a simple aggregation strategy in which we assume if any of the chromosomes in the set are aberrant that we can yield an abnormal label: . ( ) = arg max { ( 1 ), ( 2 ), ... } Additionally, for samples, we employed the cytogenetic definition of a clone -an aberration is considered clonal when observed in at least two cells. For this hypothesis we calculate the score of the patient as the prediction probability of the second highest prediction for the given patient sample,: , which we refer to as "specimen level 2 ( ) = 2( (   1   ), (   2   ), ... }) [2] ≥ abnormal" in the figures.