{'fraction_non_alphanumeric': 0.043791956835435096, 'fraction_numerical': 0.009034443817052512, 'mean_word_length': 4.145255003227889, 'pattern_counts': {'":': 0, '<': 3, '<?xml version=': 0, '>': 0, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 1, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Viral Gene Compression: Complexity and Verification

The smallest known biological organisms are, by far, the viruses. One of the unique adaptations that many viruses have aquired is the compression of the genes in their genomes. In this paper we study a formalized model of gene compression in viruses. Specifically, we define a set of constraints that describe viral gene compression strategies and investigate the properties of these constraints from the point of view of genomes as languages. We pay special attention to the finite case (representing real viral genomes) and describe a metric for measuring the level of compression in a real viral genome. An efficient algorithm for establishing this metric is given along with applications to real genomes including automated classification of viruses and prediction of horizontal gene transfer between host and virus.


Introduction
In contrast to the lengthy, often redundant, genomes of higher organisms, the genomes of viruses are extremely efficient in the encoding of their genes. Where mammalian genomes, for example, possess lengthy introns which code for no genes at all, any given segment of a viral genome may be a coding region for several genes. In addition to prefix and suffix overlap of viral genes, some genes may also be encoded in a retrograde fashion (that is, the gene would be read in a direction opposite to other genes). These systems provide evidence that viruses have evolved a special type of information compression technique. Studying this natural compression system in a rigorous setting could yield insight into the structure of viral genomes and may contribute to a basis for classifying such structures.
In this paper, we will specifically consider the types of compression seen in two small double-stranded DNA virus families, Papillomavirus and Polyomavirus, and single-stranded RNA viruses from the Bornavirus, Coronavirus and, to a lesser extent, the Filovirus and Retrovirus families.
The importance of this genetic compression becomes obvious when considering the structure of viruses. Viruses generally consist of two principal components: a protein capsid, and genetic material inside the capsid. The capsid serves as protection for the genetic material and also as a mechanism for inserting the genetic material into a host cell. The genetic material may consist of single-or double-stranded DNA or RNA and, in some rare cases, a mixture of the former possibly also including proteins.
The need for compression stems from the fact that the size of the capsid limits the amount of room for genetic information inside the virus. In the case of Polyomaviruses, the genome is constrained to be approximately 5kbp (5,000 basepairs) of DNA (compare to the human genome of size 3,150,000 kb), yet still manages to encode 6 distinct genes.
We exposit here a formal model of the viral compression techniques in terms of constraints on languages. For example, we would say that a language satisfies the "viral overlapping compression" property if some prefix of some word in the language is also a suffix of some other word in the language. We can likewise define constraints for other viral compression techniques, including retrograde encodings. We will focus here on deterministic modeling of the gene-level mechanics, in contrast to the probabilistic analysis of [8], which addresses gene compression from the point of view of evolutionary pressures and constraints on entire genomes.
The organization of the paper is as follows: In section 2 we consider basic notation and prerequisites. In section 3 we define formal versions of the basic viral compression techniques and investigate relationships and dependencies between them. We consider also the question of for which families of languages it is possible to decide these properties. Section 4 focuses on the finitary case of the problem as this is the most interesting from the point of view of applied viral genetics. We present efficient algorithms to decide each of the properties for real viral genomes. Section 5 contains our conclusions and a discussion of practical applications.

Notation and Prerequisites
For a general introduction to virology, we refer the reader to [3] and [10]; for formal language theory preliminaries, we refer to [9]. Let Σ be a finite alphabet. We denote, by Σ * and Σ + , the sets of words and non-empty words, respectively, over Σ and the empty word by λ. A language L is any subset of Σ * . For a word w ∈ Σ * , we denote the length of w by |w| and the reversal of w by w R . Let N be the set of positive integers. Furthermore, for k ∈ N, define Σ ≥k = {w ∈ Σ * | |w| ≥ k}.
A full trio is a language family closed under homomorphism, inverse homomorphism and intersection with regular sets. A full trio is also referred to as a cone. It is known that every full trio is closed under arbitrary a-transductions 1 and hence arbitrary gsm mappings. We refer to [1,4] for the theory of AFL's.
For a binary relation ⊆ Σ * × Σ * and a language L ⊆ Σ * , we define We will consider the following well-known relations. Let w, v ∈ Σ * .
Also, for each of the relations above, we prepend the word "proper", which will be denoted by < p , < s , < i where we enforce that x, y ∈ Σ + above.
For example,

Viral Properties
Before formally stating the definitions of the viral properties, we will define the following sets which will be used for the conditions.
Let L ⊆ Σ * be a language, and let n ∈ N such that 1 ≤ n ≤ 6 and let k ∈ N. Then we define the following sets: So, for example, Z(i, L, k) consists of all words w ∈ L such that there exists a word u of length at least k, a non-empty word x and a word v whereby xu is in L and w = uv which is also in L.
We now define the properties that we will study.
We also call condition W (1, k, l) the l-weak k-prefix overlapping property, condition W (2, k, l) the l-weak k-suffix overlapping property, condition W (3, k, l) the l-weak k-overlapping property, condition W (4, k, l) the l-weak k-double-sided overlapping property, condition W (5, k, l) the l-weak k-retrograde overlapping property and condition W (6, k, l) the l-weak k-concatenated retrograde overlapping property.
For example, a language L satisfies W (1, k, l) if and only if there exists l distinct words w ∈ L whereby w = uv for some u, v, x ∈ Σ * , with u of length at least k,x non-empty and xu ∈ L.
We also define a strong version of the properties above.
We also refer to each of these properties by replacing the prefix "l-weak" of each condition above with "strong". 2 We now consider the relationships of these properties to each other. The following is immediate from the definitions.
Then the following are true: Also, we note the following: Then the following are true: Proof. The first three statements are straightforward. For the fourth statement, Combining Lemma 1, 2, we obtain: Then the following statements are true: We see, however that if L 1 = {abc, aa} and We also define the following sets which we will use for a characterization.
This leads naturally to some decision problems. One would like to provide algorithms to test whether languages (or genomes) satisfy these properties. Namely, can we decide whether a given language satisfies one of the properties, depending on the language family that the given language is in? For each weak condition, this amounts to deciding whether |Z(i, L, k)| ≥ l and for each strong condition, it amounts to deciding whether Z(i, L, k) = L. Proposition 3. Let L 1 , L 2 be language families effectively closed under intersection and the full trio operations with L 1 being effectively semilinear and L 2 having a decidable equality problem. Then the following are true: 1. For each k, l ∈ N and i, 1 ≤ i ≤ 4, it is decidable whether L ∈ L 1 satisfies W (i, k, l) and it is decidable whether L ∈ L 2 satisfies V (i, k). 2. If L 1 , L 2 are also effectively closed under reversal, then it is decidable whether L ∈ L 1 satisfies W (5, k, l) and it is decidable whether L ∈ L 2 satisfies V (5, k). 3. If L 1 , L 2 are also effectively closed under reversal and +, then it is decidable whether L ∈ L 1 satisfies W (6, k, l) and it is decidable whether L ∈ L 2 satisfies V (6, k).

Proof. It is easy to construct a-transducers which output
Also, every intersectionclosed full trio is closed under union and concatenation since L 1 $Σ * ∩ Σ * $L 2 is in L 1 and L 2 , there is an a-transducer which outputs L 1 ∪ L 2 and there is a homomorphism which outputs L 1 L 2 . Thus, Z(1, L, k), Z(2, L, k), Z(3, L, k), Z(4, L, k) are in L 1 and L 2 . Additionally, if L 1 , L 2 are closed under reversal, then Z(5, L, k) is in L 1 and L 2 and if L 1 , L 2 are closed under reversal and +, then Z(6, L, k) is in L 1 and L 2 . Since L 1 is effectively semilinear, we can decide if L ∈ L 1 is infinite [5] and if it is not, then we can effectively find the length of the longest word in L. Then, we can test membership of every word of length less than or equal to that length to determine whether |Z(i, L, k)| ≥ l (emptiness is always decidable for semilinear sets, and since L 1 is closed under intersection with regular languages, we can decide whether w ∈ L by testing whether L∩{w} = ∅). Also, by the decidability of equality for L 2 , the proposition follows.
We denote by NCM the family of languages defined by one-way nondeterministic, reversal-bounded multicounter machines. It is known that NCM is an intersection and reversal closed full trio effectively closed under semilinearity [7]. Also, it is known that the family of regular languages is closed under all of the operations above and has a decidable equality problem.

Corollary 1.
For each L ∈ NCM, each i, 1 ≤ i ≤ 5 and each k, l ∈ N, it is decidable whether L satisfies W (i, k, l). In addition, for each L ∈ REG, each i, 1 ≤ i ≤ 6 and each k, l ∈ N, it is decidable whether L satisfies W (i, k, l) and V (i, k).

Computational Verification of Viral Properties
Ideally, one would like to apply the formal definitions given here to real viral genomes as a method for classifying viruses based on gene compression. In this section we will consider fast algorithms to do exactly this, and their complexity. Since all real viral genomes are finite, we will restrict ourselves to dealing with finite input languages here. We will describe algorithms which will verify each of the viral properties for a given input viral genome. A viral genome is a finite language in which the words are the genes of the virus.
For a finite language L ⊆ Σ + , we let s L be the sum of the lengths of every word of L (the length of the genome).
We recall a well-known and important result from [2]. A partial deterministic finite automaton is a deterministic finite automaton in which each state need not have a transition on every letter. The smallest partial DFA for a given regular language is the partial DFA that recognizes the language and has the smallest number of states. In [2], it is demonstrated that, for each word w ∈ Σ * , the smallest partial DFA accepting For each algorithm in this section, we assume that we have some encoding of L as input, whereby there is only one copy of each word given. We have discussed above how to perform the method suffix dfa. It is easy to pass in the reversal of a language to suffix dfa, in time linear in s L . Then, in line 5 of Algorithm 1, we can check to see if the intersection is empty by keeping a counter starting at k and running w through the transition function of M , decreasing the counter at each step. Then, when the counter reaches 0, we test every state we hit on input w to see whether it is a final state. If it is, we increase l 1 and set v 1 indicating that w ∈ Z(1, L, k). Also, in line 8, we are testing whether w R ∈ Z(1, L R , k). Indeed, by Lemma 2(3), w R ∈ Z(1, L R , k) if and only if w ∈ Z(2, L, k). Thus, if this is true, we increase l 2 and set v 2 to true. In addition, w ∈ Z (3, L, k) if and only if w ∈ Z(1, L, k) ∪ Z(2, L, k) and so we increase l 3 if and only if either v 1 or v 2 is true, and we reset each to false. In this way, when the method completes, l 1 , l 2 and l 3 will be the maximum such that L satisfies W (1, k, l 1 ), W (2, k, l 2 ) and W (3, k, l 3 ), respectively. Furthermore, this method runs in time O(s L ) time. In addition, it is well-known that we can test whether a word w is in the language generated by an NFA in time O(|Q||w|) (see [6]). Thus, to find the largest integer l 6 such that L satisfies W (6, k, l), we construct the NFA from L and decide membership of w R for each w ∈ L. This takes time O(|w 1 ||Q| + · · · + |w m ||Q|) = O(|Q|s L ). Thus, one can decide whether a finite language L satisfies W (6, k, l) in time O(s 2 L ). Finally, the strong properties can also be verified straightforwardly using the algorithms presented above. Indeed, they are just a special case where l = |L|. We summarize the preceding thusly: Proposition 4. Let i satisfy 1 ≤ i ≤ 5 and let Σ be some fixed alphabet. Then given a finite language L ⊆ Σ + as input without duplicates and k ∈ N, we can both find the largest integer l such that L satisfies W (i, k, l) and we can decide whether L satisfies V (i, k) in time O(s L ). Furthermore, we can both find the largest l whereby L satisfies W (6, k, l) and we can decide whether L satisfies V (6, k) in time O(s 2 L ).

Conclusions and Discussion
We have presented here a formalization of the process of gene compression that occurs in many viral genomes. We have shown dependencies and relationships between these properties and demonstrated that, in general, most of the weak versions of the properties can be decided for languages defined by nondeterministic finite automata augmented with reversal-bounded counters while the strong versions can be decided for regular languages. Most significantly, we have given algorithms which can efficiently decide these properties for real viral genomes and provide information which is immediately useful to virologists. These algorithms give us the ability to study the relative amount of gene compression between related viruses in a quantifiable way. It may be possible to infer evolutionary relationships between viruses using this information. The fact that genes overlap one another provides a very serious constraint for viral genome evolution. It is known that viruses occasionally aquire genes horizontally (that is, a gene from an infected host becomes part of the virus's own genome). Clearly, only those genes which meet very specific constraints (e.g. those that are "compressible" relative to the virus's genome) will be able to be incorporated into the virus. Using the algorithms presented here and real viral genome data, we can find target genes in the host organism which, due to their structure, have the greatest probability of being incorporated into the viral genome.
Finally, the formal properties here also present a framework for automated classification of a virus given only its genome. The family of Coronaviruses, for example, has a very regular genomic structure: a single strand of +-sense RNA of length 27-30kb. The beginning of this RNA strand always encodes a viral polymerase (often as part of a polyprotein) and the remainder encodes a series of "nested" genes. Each of these nested genes is a proper suffix of the previous gene.
This structure can obviously be formally encoded using the properties given here. Similar compression regularities can be found in other viral genomes and encoded using our properties. Classification of a new virus is then simply a matter of verifying compliance to our properties and then checking to see if this matches any known structures.
By formalizing this ancient form of data compression, we have provided tools which will allow for further insight in the molecular evolution of viruses and assist in the automated classification of new viruses by reference to only their genomes.