{'fraction_non_alphanumeric': 0.026199621624720518, 'fraction_numerical': 0.01278449807945881, 'mean_word_length': 5.3548269581056465, 'pattern_counts': {'":': 0, '<': 8, '<?xml version=': 0, '>': 3, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Assessing the necessity of confirmatory testing for exome sequencing results in a clinical molecular diagnostic laboratory

Purpose Sanger sequencing is currently considered the gold standard methodology for clinical molecular diagnostic testing. However, next generation sequencing (NGS) has already emerged as a much more efficient means to identify genetic variants within gene panels, the exome, or the genome. We sought to assess the accuracy of NGS variant identification in our clinical genomics laboratory with the goal of establishing a quality score threshold for confirmatory Sanger-based testing. Methods Confirmation data for reported results from 144 sequential clinical exome sequencing cases (94 unique variants) and an additional set of 16 variants from comparable research samples were analyzed. Results 103 of 110 total SNVs analyzed had a quality score ≥Q500, 103 (100%) of which were confirmed by Sanger sequencing. Of the remaining 7 variants with quality scores <Q500, 6 were confirmed by Sanger sequencing (85%). Conclusions For single nucleotide variants, we predict we will be able to reduce our Sanger confirmation workload going forward by 70–80%. This serves as a proof of principle that as long as sufficient validation and quality control measures are implemented, the volume of Sanger confirmation can be reduced, alleviating a significant amount of the labor and cost burden on clinical laboratories wishing to utilize NGS technology. However, Sanger confirmation of low quality single nucleotide variants and all indels (insertions or deletions less than 10 bp) remains necessary at this time in our laboratory.

However, as read depth increases and additional samples are tested using a consistent experimental protocol and analytical pipeline, more information is available to interrogate the validity of a given variant call. In addition to the count of reference and non-reference ("variant") nucleotides observed at a given position, valuable data amasses. These data include: mapping quality (MQ), strand origin, base call quality, position of the variant within a sequence read, haplotype information, and cross-sample comparisons. The commonly used genotype calling pipeline employing the Genome Analysis Toolkit ("GATK") [1,17] implements a Bayesian genotype likelihood model (based on known polymorphic loci such as dbSNP variants) and variant quality score recalibration (VQSR) to estimate posterior probabilities for each variant call (with hapmap_3.3.b37.sites and 1000G_omni2.5.b37.sites for training resources).
While technically these final quality scores ("Qscores", or "QN" where N is a value greater than zero) are reported as Log-scaled probabilities, comparison across experiment types is not advisable due to the large degree of variability of data volume, data quality, and options between NGS analytical pipelines. In this study, Qscores are considered to be relative measures, and are compared only between clinical exome sequencing (CES) datasets from the end-to-end analytically validated procedures established in the UCLA Clinical Genomics Center, which is part of the UCLA Molecular Diagnostics Laboratories (both CLIA-and CAP-accredited).
For variants with high quality scores (>Q10,000) and high coverage (>100x), the amount of information supporting the genotype call is overwhelming. For such variants, failure to replicate the finding by Sanger sequencing is highly indicative of human error (such as a sample swap). Thus, for high-quality NGS variants, Sanger confirmation serves almost exclusively as a sample quality control (QC) measure. Therefore, it is the goal of this study to establish a conservative internal quality score cutoff, above which Sanger confirmation of CES-identified variants will no longer be a necessary quality control (QC) measure in our laboratory.
genomic DNA was isolated from whole blood collected in a lavender-top tube (K 2 EDTA or K 3 EDTA) using a QIAcube (QIAGEN). For all of the clinical samples, exome sequencing was performed using the Agilent SureSelect Human All Exon 50mb for exome capture and Illumina HiSeq2000 for sequencing as 50bp paired-end runs using V3 chemistry. For the non-clinical samples, Agilent SureSelect Human All Exon 50mb XT kit (V2) was used for exome capture and Illumina HiSeq2000 for sequencing as 100bp paired-end runs using V3 chemistry.
Data analysis was performed using the analytical pipeline implemented and validated for clinical exome sequencing in the UCLA Clinical Genomics Center. All sequence reads were aligned to the human reference genome (Human GRCh37/hg19) using Novoalign. PCR duplicates were marked by Picard, and GATK was used to realign indels, recalibrate the quality scores, call, filter, recalibrate and evaluate the variants. All variants called across the protein-coding regions and flanking junctions were annotated using Variant Annotator X (VAX), an in-house MySQL database using data from the publicly available Ensembl Variant Effect Predictor [18]. A detailed description of the bioinformatic methods used to analyze these data is presented as Supplemental Materials 1.
Several steps are taken to reduce the probability of sample swap errors in our laboratory including: 1) assays are performed by appropriately licensed technologists with experience in next generation sequencing workflows; 2) at least two unique identifiers are used to label all reaction vessels and worksheets at all pre-analytical stages; 3) samples are alternated by gender. In addition, when related individuals are tested as part of a trio, Mendelian errors are analyzed by counting the number of inconsistent genotypes. For instance, from internal experience, the proband should not have more than five de novo amino acid altering rare variants, and approximately half of the heterozygous variants present in the proband should be inherited from the mother and the other half from the father. Also -when available -prior genetic testing results (such as variants described in clinical reports from individual gene assessments or regions of homozygosity from chromosomal microarray analyses) are cross referenced with the CES data as well. Some additional steps that laboratories could employ to reduce sample errors include running samples in duplicate, running the CES assay in parallel with a SNP array or genotyping identity panel for concordance analysis (if not previously performed as mentioned earlier), or spiking the blood sample with a unique plasmid during extraction and confirming the plasmid sequence occurs in the final result.

Variant Selection
All clinically reported variants, both clinically significant findings and variants of uncertain significance (VUS), were selected for confirmation. In total, 110 unique SNVs were selected for Sanger confirmation (Table 1) and a subset of these (16 SNVs) were randomly selected for assessment from a pool of variants with quality scores <Q2000. These additional variants had not been clinically reported, as they are not in genes which are known to cause any clinical condition.
All SNVs selected, regardless of report status, are predicted to be non-synonymous and are rare (with an average minor allele frequency <1% in the Exome Variant Server [19]).

Sanger Sequencing
PCR primers were designed for each target locus using the web-based Primer3Plus software [20]. Targets were amplified using PCR and subjected to agarose gel electrophoresis for size analysis of resulting amplicons. If no amplicon was observed, multiple amplicons were observed, or an amplicon of improper size was observed, a second independent set of PCR primers was designed and tested in a similar fashion.
Unique, properly sized amplicons were purified using standard techniques. BigDye Terminator DNA sequencing reactions were then performed on eluted amplicons and sequenced by automated capillary gel electrophoresis (ABI 3730, Life Technologies Corp., Carlsbad, CA). An ABMG board-certified clinical molecular geneticist manually analyzed the resulting sequence traces using Sequence Scanner (ABI).

Cost Analysis
The reagent cost per validation is estimated to be $20 (USD) per variant on average. The personnel cost for designing PCR primers, running the assay, analyzing the data, and interpreting the results is estimated to be $120. Overhead (including facilities, maintenance, instrument costs, and other considerations) contribute approximately $100 per test. Combined, the estimated cost of performing Sanger confirmation of a single SNV is thus approximately $240. These values were calculated based on standard clinical molecular genetics practices and average licensed medical technologist salaries in the UCLA Molecular Diagnostics Laboratories.

Results
Exome sequencing results were confirmed for 103/103 (100%) of SNVs with quality scores ≥Q500 ( Table 1). The coverage depth for these variants ranged from 5x-250x with a mean of 116x. The correlation between quality score and coverage depth is positive and statistically significant (R=0.56, P<0.0001) (Figure 1). Of the 7 SNVs with quality scores <Q500, only one was not corroborated by the Sanger sequencing data (Table 1, Figure 2).
From the first 144 signed out reports, the average number of reported variants per report is approximately 1 (range: 0-5 variants). With an estimated cost of $240 USD per confirmation and a sample volume of 40 reports per month, the total cost to the laboratory performing the test is $9,600 per month ($115,200 per year). Furthermore, the number of clinically relevant (non-incidental) variants reported per case is not expected to decline over time. Instead, as more disease-gene associations are made, we expect the number of cases with at least one potentially causal variant to increase. Thus the cost of Sanger confirmation will scale at least linearly with this increased sample volume. Notably, turn-around time for reports requiring variant confirmation were delayed at least one week on average compared to reports with no reported variants.

Discussion
For each UCLA Clinical Exome Sequencing test, the decision to report a variant begins with interpretation by a group of diverse experts at a Genomic Data Board. This interpretation considers the molecular genetic evidence (such as the effect a DNA change is predicted to have on its corresponding protein product) as it relates directly to the primary clinical concern(s) noted by the ordering physician. At present, incidental findings are not reported. If the board decides a variant is worthy of reporting, the laboratory then considers the technical validity of the finding. Prior to May 2013, Sanger sequencing was used as an alternate methodology for validation of each reported variant. Since that time, only indels and SNVs with quality scores <Q500 are validated by Sanger sequencing.
As it has been considered the "gold standard" for over two decades, using capillary-based Sanger sequencing for confirmation of all NGS results is a safe choice. However, taken out of context, this is highly unusual; technical confirmation of results from a validated assay using an alternate methodology prior to reporting is not often employed for other types of molecular testing. Additionally, there are several specific reasons to suspect that Sanger confirmation of all clinically relevant SNVs detected by NGS is an unnecessarily conservative approach with significant drawbacks.
First, NGS can be sampled to generate dozens or hundreds of independent reads across a locus whereas increased sampling of Sanger sequencing requires technical replicates. Although a Sanger sequencing peak does represent a large number of individual DNA molecules, these are clonal and arise from an unknown number of original template molecules. At heterozygous positions sequenced bidirectionally, the minimum number of original template molecules required to produce a signal is only four: forward reference, forward alternate, reverse reference, and reverse alternate. While it is likely that a larger number of template molecules are typically amplified, it is not possible to assess or confirm this number due to the clonal nature of PCR amplification. While the error rate for a single base is relatively higher in NGS than Sanger sequencing, high read depth ("coverage") of a locus can overcome this issue.
Additionally, PCR-based amplification is susceptible to allele dropout due to cryptic variation within primer binding sites, whereas the target enrichment techniques used in exome sequencing are not. Additionally, some genomic intervals are extremely difficult to amplify, and may not yield high quality Sanger results despite multiple attempts. Being unable to report a clinically significant variant due to a failure of the Sanger technology introduces a challenging obstacle if the NGS assay is analytically validated.
NGS variant identification is not without error. However, above a certain hypothetical quality threshold, the probability of observing a false positive NGS result is lower than the false negative rate of Sanger sequencing (which itself is not perfect). This means that for variants meeting this threshold, performing Sanger sequencing is non-informative beyond sample QC, as the vast majority of results will be concordant and the remaining negative results will not be interpretable. Thus, such high-quality next generation sequencing results, when routinely obtained using a method validated by a clinical laboratory, should be considered an equally defensible "gold standard." The difficulty then is in determining a high-confidence quality threshold. Coverage depth is a useful guide, but probabilistic genotyping algorithms such as those implemented within the GATK [17] provide highly informative quality scores. Because quality scores are assayspecific and relative, it is not possible to calculate an a priori threshold value. Rather, based on a sample of 110 SNV confirmation tests, we have established a conservative in-house quality score threshold of Q500 (approximately 40x coverage) for the Clinical Exome Sequencing test in our laboratory, above which all 103 single nucleotide variants detected were confirmed by Sanger sequencing.
Manual inspection of variant calls using a visualization tool such as the Integrative Genomics Browser (IGV) [21,22] by a genomics expert is a potential alternative to our quality score threshold approach. While our experiences generally support this as a valid potential solution, we do not have sufficient data to broadly assess the efficacy of this approach.
Small insertions and deletions ("indels" defined here as <10bp) are also detected by Clinical Exome Sequencing and reported if clinically significant. At this time, we do not have sufficient data to propose a quality score threshold for confirmation of indels and will thus continue to Sanger-confirm all such reported variants.
The perceived benefits of performing Sanger confirmation on all NGS-detected SNVs lies in quality control and risk avoidance. This must be weighed against increased test cost, delayed turn-around time, and the potentially paralyzing failure to confirm a very high quality variant of clinical significance. While current professional practice guidelines recommend confirmatory testing of all clinical NGS results, they also allow for laboratories to reduce the amount of confirmatory testing performed as long as suitable validation studies have been completed [23]. Follow-up testing of identified variants in additional family members for carrier or pre-symptomatic status by Sanger sequencing is performed in our laboratory upon request for an additional fee. However, in practice, this has been a rare occurrence; for the majority of our exome sequencing cases, the original proband is the only family member tested, which also argues against the need to have a pair of Sanger sequencing primers available in the lab for every variant detected.
All genetic tests introduce uncertainty. At the genomic level, it is the exception, not the rule, when a causal relationship between a genetic variant and a clinical condition can be made absolutely. Thus, when counseling for Clinical Exome Sequencing results, the slight probability of a high quality variant being an analytical false positive is typically a minor consideration compared to the uncertainty of genotype-phenotype relationships. This argues against devoting large amounts of resources to confirmatory testing for variants of high confidence, especially when the testing laboratory is conservative in the ascertainment and reporting of "causative" variants, as ours is.
Stemming from these theoretical and practical considerations, and based on data resulting from the confirmation of 110 SNVs, our group has decided to discontinue routine Sanger confirmation of reported Clinical Exome Sequencing results with quality scores >Q500 (SNVs only). However, other laboratories wishing to follow this paradigm must establish their own quality thresholds for each assay and provide empiric evidence to support those decisions.

Figure 1. Correlation between Quality Scores and Depth of Coverage
Individual quality scores are plotted against read depth for 110 SNV loci tested. Quality score threshold of Q500 is marked by a dashed grey vertical line. The correlation is positive and significant (Pearson Correlation Significance Test, P<10 −13 ).  Table 1 Variant information.