#set page(numbering: "1", number-align: center)

#set math.equation(numbering: it => {[6.#it]})
#counter(math.equation).update(0)
#import "../notation.typ": vectorn, uvectorn, dvectorn, udvectorn, matrixn

= Skywarp: An Attention-Based model for the Detection of Gravitational-Wave Compact Binary Coalescences <skywarp-sec>

Convolutional Neural Networks (CNNs), though effective, have not been the stars of the machine-learning world for a few years now @architecture_review. Just as AlexNet @image_classification paved the way for the era of CNNs, and the surge to prominence of artificial neural networks as problem-solving solutions in the image domain, a similar step change occurred when the confluence of several different technologies led to the development of Large Language Models (LLMs) @LLM_ref, most notably of which in recent years has been the Generative Pretrained Transformer (GPT) @LLM_ref @attention_is_all_you_need of ChatGPT fame @chatgpt_ref, though it should be noted that this was far from the first LLM @LLM_ref. 

Although CNNs were extremely successful at solving previously intractable problems in the image and audio domain @cnn_review @deep_learning_review, there were still many challenges remaining within Natural Language Processing (NLP), the area of study relating to the analysis of the text domain @LLM_ref. Text sequences differ from audio time series, in that rather than vectors of continuous numerical values, they consist of sequences of discrete tokens that encode externally defined values. There had been some work to attempt these problems with CNNs @text_classification and Recurrent Neural Networks (RNNs) @lstm_text_generation, but it was only through the application of multiple different insights including positional encoding @attention_is_all_you_need, attention mechanisms @attention_1, and skip connections @skip_connections, that the start of the avalanche of progress we have since seen in this area, began. The groundbreaking paper by Vaswani _et al._, @attention_is_all_you_need first introduced the transformer architecture to the world.

Following the success of transformers in NLP, there has been much research into the application of attention-based models to other domains, including image and audio processing. Notably, AlphaFold @alpha_fold, a project by Google Deepmind, successfully solved the protein folding problem. Transformers have proven effective, scalable solutions due to their highly parallelizable nature @LLM_ref @transformer_review, and new attention-based architectures such as generative diffusion models have seen great success in text-to-image generation problems @diffusion_review, as is seen in products such as Stable Diffusion @stable_diffusion, Midjourney @midjourney, and Dall-E @dalle_3. Gravitational-wave astronomy has also seen the application of a large number of machine-learning approaches @gw_machine_learning_review; however, there exists a considerable delay between the advent of new techniques and their application to gravitational wave data. Since this research was originally carried out, the gap has narrowed, nonetheless, it remains a relatively unexplored area that may prove fruitful in the future. In this chapter, we introduce Skywarp, an attention-based model for the detection of gravitational waves produced by Compact Binary Coalescences (CBCs).

The distinguishing feature of the transformer architecture is the use of attention @attention_is_all_you_need. Attention is a technique that can determine the information value of data elements in a series, crucially including the relative information derived contextually from the value of all other data elements within that series @attention_review. As opposed to convolutional layers, which learn feature-extracting filters that are convolved with the input data @conv_review, attention layers learn a weighting for each data element, identifying the information interactions between elements. One key advantage of this method is that attention is computed globally, whereas convolutional layers only use a local context dictated by their receptive field. This makes attention an ideal candidate for analyzing time-series data, wherein important contextual information can be located at a significant temporal distance from any given datum. RNNs share many of these properties @reccurant_neural_networks @rnn_review; however, they make use of an internal state, which means they are much harder to parallelize and therefore scale, and their knowledge of data outside of their receptive field is sometimes uni-directional, as opposed to transformers, which are bi-directional (although it should be noted that bi-directional LSTMs do exist @bidirectional_lstm). In addition, it should be noted that self-attention layers are more general than convolutional layers, as it is proven that a self-attention layer can generalize any convolution @generality_of_attention. This is not to say that more generality is necessarily good. As we have demonstrated in @perceptron-results, we are unable to train dense networks, the most general of all, to identify gravitational-wave signals at a useful false alarm rate.

This section is organized into the following structure. First, we will give a brief technical overview of the concepts and structure of the transformer architecture in @attention-method, including a discussion of tokenization, embedding, and attention, and how these elements are assembled into transformers proper. Then we will review the small amount of relevant literature that has not already been discussed in @skywarp_review. Next, we will provide details on the models and training procedures we used to train Skywarp in @skywarp-method. We will show the validation results from our trained models in @skywarp-results, before finally in @skywarp-discussion, we discuss the importance of these results and investigate how the method employed by the transformer models differs from that used by CNNs by comparing their respective attention and convolution maps.

== Attend Closely <attention-method>

The transformer is a deep learning model first described in Vaswani _et al._ @attention_is_all_you_need. This paper was an NLP paper, demonstrating a deep learning sequence-to-sequence model that could ingest text data and predict the token that was most likely to appear next in the sequence. By recursing inferences of the model, new sentences could be generated based on previous inputs. In some sense, our time series data is already closer to the required input of a deep learning model, however, it is easiest to explain attention by using NLP as an example. Therefore whilst describing attention we will use the text domain as example data before replacing the relevant vectors with gravitational-wave equivalents.

=== Tokenisation and Embedding

There are several steps required in order to condition text data for consumption by deep learning models. Artificial neural networks work solely with numerical data therefore text must somehow be converted into numbers before it is ingested by the model. The most obvious way to do this would be to use a preexisting character format such as ASCII @ASCII or Unicode @Unicode. However, if we were to do this, the numbers would relate very little even to the characters they represented, let alone the words. This would make these inputs very difficult for the model to analyze. Therefore, in order to make the task easier, we can use a method to embed the words into a numerically defined N-dimensional space in a manner that maintains some of their meaning in their new vectorised representation. Typically this is achieved in two or three steps @nlp_chapter, tokenization @tokenization_ref, vectorisation, and embedding @word2vec_processing; see @tokenisation_diagram.

#figure(
    image("tokenisation_diagram.png",  width: 100%), 
    caption: [The process of conditioning text data for input into a deep learning model. Text data is not intrinsically digestible by artificial neural network models, as artificial neurons can only process numerical inputs. Therefore, in order to apply deep learning models to text data, we must have some method of converting the data into a numerical format @nlp_chapter. Transformers expect a sequence of same-length vectors forming an input matrix, $matrixn(X)$. This diagram shows the process of converting text data into an input matrix. Typically, this conversion is completed in three steps, tokenization, vectorisation, and embedding. However, often, and in the case of the first described transformer model, vectorisation and embedding occur simultaneously and are often labelled simply embedding @attention_is_all_you_need. This is the method depicted in the diagram. In the example, we see the sentence "The quick brown fox jumped over the lazy dog." as it is prepared for ingestion by an NLP model. *Tokenisation* is the process of splitting one contiguous sequence of characters into a number of unique discrete tokens, $N$. This can be done at multiple levels but is usually done at the scale of words. Sometimes, especially with longer words, words can be split into multiple tokens, as is seen in this example where the word "jumped" is split into "jump" and "ed" @nlp_chapter. There are numerous algorithms to achieve this, which will not be discussed in detail. Every word, or word subset, within the training dataset, should have a unique token ID. Before running inference on new text data, that data must be tokenized, and each word in the new data will be mapped onto an existing token ID that was generated during the initial tokenisation process. Often some information-low words, known as "stop words", and punctuation are removed during the tokenisation process @nlp_chapter. In the example shown, the words "The", and full stops are removed from the input string. During *vectorisation*, each token is assigned a numerical vector, and *embedding* ensures that this vector is transformed into a meaningful vector space to allow for easier interpretation by the model. There are a number of methods to achieve both of these steps, some of which are simultaneous. In the example shown, each token ID is associated with a vector of tunable weights, as was the case in the first transformer paper. These vectors are randomised at the start of training, but as the process continues, they become tuned to values that represent the information contained by the tokens. In this manner, the vectorisation and embedding steps occur at the same time.]
) <tokenisation_diagram>

First, the input sequence must be tokenized @nlp_chapter. Tokenization involves splitting the input text into $N$ unique discrete tokens. Often the text is processed first, sometimes removing low information elements known as stop-words and punctuation. Then, using one of a variety of algorithms that will not be discussed here @tokenisation_review, the text is consolidated into $N$ tokens, often these could be whole words, but sometimes, depending on the tokenization algorithm and the size of the training data, word fragments could also be tokens. One will note that often words can have multiple meanings, which is a problem when trying to describe each token in a way that somehow presents the value it represents in a sentence. This can be a problem, and methods have been developed that can split identical sequences of characters into multiple tokens contextually. The very existence of this problem is an example of the contextual information that is provided by surrounding tokens in the sentence, this is the information that attention layers attempt to extract and distil.

After tokenization, we must convert each token into a unique vector. This can also be done through a variety of methods. In Vaswani _et al._ @attention_is_all_you_need each token has an associated vector in a look-up table, initially these vectors are randomly generated; however, the values in each of these vectors act as tunable parameters inside the model, so that whenever a certain token is present in a particular training example, the weights of its vector are tuned through the usual gradient-descent methods. In this way, as the training progresses, the vectors become, at least to the model, meaningful numerical representations of the value contained within their associated tokens. 

Gravitational-wave data is intrinsically vectorized so the embedding layer should not be much of a problem, however, it is not intrinsically discretised. Since transformers are sequence-to-sequence models, they ingest a series of N vectors forming an input matrix @attention_is_all_you_need @transformer_review, whereas gravitational-wave time series data is a single vector, at least when dealing with one detector. It is unclear how best to split the gravitational wave data into smaller vectors. We could simply cut along equally separated lines, "chunking" our data into smaller timesteps, or we could embed the data using some learned weights, for example with one or more dense or convolutional layers, in the latter case, feeding the transformer with feature slices at different timesteps; see @gw_embedding. Using different detectors as this extra dimension will only give us two to four features per timestep, which would be very small vectors for the transformer to work with.

#figure(
    grid(
        columns: 1,
        rows:    2,
        gutter: 1em,
        [ #image("skywarp_chunking.png",  width: 100%) ],
        [ #image("skywarp_dense_embedding.png",  width: 100%) ],
        [ #image("skywarp_convolutional_embedding.png",  width: 100%) ]
    ),
    caption: [Different embedding possibilities to discretise and embed gravitational-wave time-series data. _Upper:_ "Chunking" method of discretisation, where the input time-series is split into $N$ equal-length segments which can be fed into an attention-based model. This method would seem to have the disadvantage that it could split the waveform at any point, leading to chunks with very different waveform content depending on the waveform offset; it also assumes that the innate interferometer output vector is a good embedding for the attention mechanism, which is not necessarily true. _Middle:_ Embedding with dense layers, this setup is similar to the chunking method, but it applies one or more dense layers to each chunk so that the model can learn an embedding that will be better adapted to the attention mechanism in subsequent layers. Since the parameters of the dense layers are repeated for each chunk, this method is equivalent to a convolutional layer with $N$ filters and no overlap, where $N$ is the size of your embedded vector output. _Lower:_ Embedding with convolutional layers. This type of embedding involves creating feature maps of the input vector using a combination of convolutional and/or pooling layers. It is the equivalent of attaching a CNN head at the front of your model. The output of a 1D CNN would be a 2D matrix where one dimension, the depth, is different features, and the other is time. This can then be split into discrete vectors by splitting it along the time dimension to create vectors of features with length equivalent to the number of features.]
) <gw_embedding>

We have managed to transform our input text from a list of symbols into discrete tokens and finally into vectors that contain some aspect of the value represented by that token, and we have some ideas about how we might do the same with gravitational-wave data. However, unlike convolutional layers, attention layers treat each input element equally and intrinsically have no information about the location of the word in the sentence. We must use feature engineering to add to each vector, some information about the position of the token in the input sequence. 

=== Positional Encoding

Much information is embedded in the relative and absolute positions of tokens within text data @transformer_review. The same can said to be true of gravitational-wave data --- we would always expect the merger to come after the inspiral, for example. Whilst there is some possibility within the dense layers of a traditional CNN for the model to use this ordinal information in its classification, it might be a challenging process. We can use attention layers to look at the global information in a sequence, but since, unlike CNNs @conv_review, there is no structure inherent to the architecture that maintains information about the position of the inputs, if we feed in the word sequence as-is, we end up with a "bag of words" @bag_of_words_ref; see @bag_of_words. Whilst some models can do quite well with just a bag of words approach, able to infer context simply from the numbers of each word present @bag_of_words_ref, it is clear that some information is lost when discarding order.

#figure(
    image("bag_of_words.png",  width: 60%), 
    caption: [A "Bag of words". Without ordinality, the meaning represented by this sentence becomes significantly harder, if not impossible, to parse. If we had not already seen this sentence then we would not know if the fox was lazy or quick, or rather if it were the dog that was lazy or quick, and just who is jumping over whom? There are NLP models that are designed to use a bag of words as inputs, but it is easy to see that much information is lost when word order is discarded, thus we can infer that the order and position of the words contain a significant amount of information. The same can be true for time series, a CBC signal that contains a merger, an inspiral, and a ringdown, in that order, can probably be discounted as a glitch, but if we feed it in as a bag of words model, there could be no distinction between this and the expected arrangement.]
) <bag_of_words>

We solve this problem by adding extra information to our input embeddings with positional encoding @attention_is_all_you_need @positional_encoding_ref. To do this, we create a matrix that is the same size as our attention input matrix: [num_time_steps, num_feature_channels]. Each column in our matrix must have certain properties: it must be unique so that no two feature embeddings are given the same encoding, and it must convey information about the absolute and relative position of a given feature vector in the input sequence. We create this matrix using

$
    op("PE") (t,i) = cases(
        sin( t /(log(10000)^(i/d_"model") )) "if" i "is" "even",
        cos( t /(log(10000)^(i/d_"model") )) "if" i "is" "odd"
    )
$ <positional_encoding_eq>

where $op("PE") (t, i)$ is the positional encoding matrix, $t$ is the time index, $i$ is the feature index, and $d_"model"$ is the dimension of our model, the relevance of which will become clear later. The periodicity of sine and cosine functions enables a unique identifier for each vector whilst maintaining a consistent pattern that evolves across the time dimension @attention_is_all_you_need. This uniqueness ensures that absolute position is encoded; all feature vectors get a unique encoding, which will be the same independent of the vector's contents so the model can learn which encodings map to which positions. The logarithmic term, $log(10000)$ ensures that the variation in frequency between steps is large enough to be detectable by the model, whereas the scaling by $d_"model"$ ensures that the positional encoding values do not become too large and overshadow the feature vectors, or become so small they are undetectable. The relative position between any two vectors in the sequence can be estimated due to the linear superposition property of the sin and cos functions; the sum of the positional encodings will approximate $t_1 + t_2$, and the difference will approximate the difference, $t_1 - t_2$. Therefore, when the model adds or subtracts positional encodings (as it might do implicitly during training), the resulting encoding still carries meaningful positional information. This matrix is added to our sequence of input vectors by simple element-wise addition, therefore inherently encoding positional information into each feature vector. 

Fortunately, this embedding process is just as appropriate to use on gravitational wave data as it is on text data. In early testing, we found that including positional encoding improved model performance significantly.

By adding positional encoding to our input vectors, we have ensured that even if we (or a model) look at the vector in isolation we will still be able to know where in the vector it originated @attention_is_all_you_need @positional_encoding_ref. So we have stored extra information within the vector, however, if we look at this new vector in isolation, there is still much contextual information provided by the rest of the sequence that we cannot access alone. If we look at the word "dog" in isolation for example, even if we knew it was the ninth word in the sequence, we would have no idea that it was lazy, or that a fox was jumping over it. To embed this kind of information, we must turn to attention layers.

=== Attention! <sec-attention>

The global information provided by an individual element within a sequence is often greater than the local information contained within the isolated element @attention_1 @attention_2 @attention_is_all_you_need @attention_review. This extra information is stored contextually within the relationship between the given element and the other elements in the sequence, both within the information stored locally by the other elements and by the relative and absolute positions of the other elements.

The set of possible combinations of elements is large, even within relatively small sequences. Therefore, in order to enable a machine learning model to extract contextual information efficiently, a method must be implemented to determine which elements contribute the most contextual information to each element. This method is attention @attention_1 @attention_2 @attention_is_all_you_need @attention_review, a type of differentiable memory in which a global context vector is learned over an input sequence in the form 

$ matrixn(X) = mat(vectorn(x)_bold(1), ..., vectorn(x)_bold(i), ..., vectorn(x)_bold(n);) . $ 

The attention mechanism aims to embed global context locally; in order to do this, a comparison must be made between each element of the sequence and (in the case of self-attention) each other element of the same sequence. It is trivial to see that not every element in every sequence will be equally relevant and that this contextual dependence will depend on the information being extracted. In this way, one learns intra-sequence relations; long-term dependencies are captured because the entire input sequence is used to compute a single element of the output sequence. Ideally, this process makes the output elements, now with contextual information embedded locally, easier for other machine-learning methods to interpret. 

A transformer model is a machine learning algorithm that implements this method to localize global information using attention @attention_is_all_you_need @transformer_review. The output of a transformer block has the same dimensionality as the block’s input, as it retains the same number of elements. Ideally, each element has been transformed to contain a proportion of the relevant global information stored within the input sequence.

The question becomes, how can we calculate the attention? We can use an analogous problem to demonstrate the principle. In search and retrieval tasks, such as a search engine query, the user, in this case, a human, must generate a *query* phrase that can be used to find relevant information. This query phrase will not contain the entire information content of whatever document we are attempting to discover, if it did then we would not need to perform the search. Instead, it is generated using words and phrases that are associated with the information we are searching for. The search engine then has the unenviable task of searching through its entire library to find documents that might have information relevant to the query. 

The first instinct might be to look through every document and check to see if there are words and phrases in that document that match the content of the query. Immediately, we can tell that this will quickly become infeasible if the library is large, and/or contains large documents --- the process of searching would rapidly become very expensive. Instead, the search engine could have preprocessed these files, and in a similar manner to how the query was generated, it could pick out the key information content of each document in a distilled form that contains the information that it is most likely to match with a query. It generates a *key* or keys for that document, which can be checked against queries much more efficiently than searching the entire content.

Finally, the *value* of the information that the end user extracts from whatever document is returned, will not necessarily equate to the entire information content of the document. Depending on what information the user was originally searching for, and hence what query they entered into the search bar, they might only read a particular chapter of a book, or, even more specifically than that, they might only retain certain parts of information from that chapter that are relevant to their needs. During a search session, a user might enter a single query that matches well with multiple keys that return documents which the user then reads and summarises parts of the information in each document to gain new knowledge on whatever the original subject of their query was.

This analogy introduces the three key information concepts of the query, key, and value. We can use these concepts to build a deep learning layer that can, for every element of our input sequence, search through each element in the sequence and extract relevant contextual information that can then be embedded into that element, in a similar manner to how we can embed information about the elements position using positional encoding. In attention layers,  query, $vectorn(q)_bold(i)$, key, $vectorn(k)_bold(i)$, and value $vectorn(v)_bold(i)$ vectors are generated for each sequence element $vectorn(x)_bold(i)$, forming three matrices for the sequence as a whole: $matrixn(Q)$, $matrixn(K)$, and $matrixn(V)$. We create these matrices by multiplying three projection matrices with the input matrix, $matrixn(X)$, the query, $matrixn(W)_bold(q)$, key, $matrixn(W)_bold(k)$, and value, $matrixn(W)_bold(v)$, matrices. $matrixn(Q)$, $matrixn(K)$, and $matrixn(V)$ are generated with

$ matrixn(Q) = matrixn(W)_bold(q) matrixn(X), $ <weight-1-eq>
$ matrixn(K) = matrixn(W)_bold(k) matrixn(X), $ <weight-2-eq>

and

$ matrixn(V) = matrixn(W)_bold(v) matrixn(X). $ <weight-3-eq>

The elements inside these weights matrices are the only tunable parameters that are learned during the model training process @attention_1 @attention_2. During model training, the weights will adapt so that they can generate effective query, key, and value vectors that allow for proficient model function. Since this is a neural network and these are learned weights, multiplication by these weights matrices is equivalent to the application of a dense layer with no bias values.

The nature of attention layers makes it more difficult to draw artificial neuron connection diagrams as we have previously with perceptrons and CNNs, since the information flow is more complex. However, we can attempt to visualize the interaction between the various vectors as interacting functional elements, like machines in a factory, organelles in a cell, or gears in a clock; see @weights_matricies.

#figure(
    image("weights_matricies.png",  width: 70%), 
    caption: [Generation of query, key, and value vectors for each element in the input sequence of length, $N$. Before attention scores are calculated, each input vector, $vectorn(x)_bold(i)$ is multiplied by the learned query, $matrixn(W)_bold(q)$, key, $matrixn(W)_bold(k)$, and value, $matrixn(W)_bold(v)$, weights projection matrices to produce a query, $vectorn(q)_bold(i)$, key, $vectorn(k)_bold(i)$, and value $vectorn(v)_bold(i)$ vector for the input element, $vectorn(x)_bold(i)$. This operation is equivalent to the multiplication of the projection matrices and the input matrix, matrixn(X), to produce the query, $matrixn(Q)$, key $matrixn(K)$, and value $matrixn(V)$ matrices. The key takeaway is that the only tunable parameters are contained in the weights matrices, which act as projection functions to convert the input vector into functional vectors.]
) <weights_matricies>

#figure(
    image("q_k_v.png",  width: 40%), 
    caption: [Illustration of example query, key, and value vectors generated for the sentence "The quick brown fox jumped over the lazy dog.". After tokenisation and embedding, each vector in the embedded input sequence generates its own query, key, and value vector. Which together form query, key, and value matrices.]
) <qkv_diagram>

The query, key, and value matrices are used to calculate attention; see @qkv_diagram for an illustrative example of the projection matrices applied to the example sentence. The attention method aims to collect relevant information about a given sequence element within that element, extracting the information content from the position and meaning of the surrounding elements. Understandably, language does not have words for every possible concept, instead, it relies on combinations of words to provide many more concepts than single words could alone @linguistic_complexity. For example, language could have developed a single word for "lazy-dog" and "quick-fox"; but you would soon end up with an extraordinarily large vocabulary (assuming that new words were invented rather than just cheating and compounding words with a hyphen). If we wanted to include more complex concepts like "quick-brown-fox-that-jumped-over-the-lazy-dog" and "lazy-dog-that-has-had-quick-brown-fox-jump-over-it", the number of potential concepts becomes vast. Within the vector space, however, we are not limited by discretized tokens, and such concepts can all exist in a highly multi-dimensional space, since, in effect, we can add vectors together to sum their meanings. Attention layers essentially attempt to assemble these complex words @transformer_review.

In order to assemble these new vectors with embedded contextual meaning, we must work out the magnitude to which each other element affects the meaning of that element. This score is the "attention" for which the process is named @attention_1 @attention_2 @attention_review. In the example sentence, "The quick brown fox jumped over the lazy dog," we can see that almost all of the concepts are somehow interacting with each other in a significant manner. If we were to extend the string however say to, "The quick brown fox jumped over the lazy dog. Incy wincy spider climbed up the water spout.", we can see that tokens in the second sentence have very little effect on the concepts in the first sentence, so we might expect the attention scores between tokens in different sentences to be much lower than in the same sentence. Now in very advanced LLMs, there could be some cross-sentence attention as the model tries to determine why those two sentences in particular are next to each other, a question which could certainly hold some information, but this would be at a much higher level of abstraction than the simpler cases we have been discussing.

The query value for each sequence element is matched against the key value of each other element @attention_1 @attention_2 @attention_review; see @alignment-diagram. The alignment of the key and query determines a weighting for the value vector, a distilled representation of the relevant information contained within that element; see @scaling-diagram. The weighted value vectors are then summed to produce the new, contextually embedded, element. The two most common attention methods are dot-product @attention_2 and additive attention @attention_1, our models utilise the former and so we restrict our discussion to the work of Luong _et al._ and extensions. In either case, the function $alpha$ maps a set of query $vectorn(q)$, key $vectorn(k)$, and value $vectorn(v)$ vectors to a weighted sum of the values. This is given by

$ alpha(vectorn(q)_bold(i), matrixn(K), matrixn(V)) = sum_(j=1)^N a(vectorn(q)_bold(i), vectorn(k)_bold(j)) vectorn(v)_bold(j) $ <attention-eq>
  
where $a(., .)$ is called the alignment function and measures the similarity between the queries and keys. In the case of dot-product attention

$ a(q, k) = sigma((vectorn(q) dot vectorn(k)) / sqrt(d_k)) $ <alignment-eq>
  
where $sigma$ is the Softmax function; see @softmax-sec, and $d_k$ is the number of elements in the key vector, used to scale the value so that differences remain large when $d_k$ is large. This scaling was not a part of the original dot-product attention approach @attention_2 and was added by Vaswani _et al._ @attention_is_all_you_need, it has since become a common feature in attention layers.

#figure(
    image("alignment_function_diagram.png",  width: 100%), 
    caption: [Illustration of the operation of how the alignment function utilizes the query and key vectors to produce alignment scores for each sequence element. In dot-product attention @attention_2, this is achieved using @alignment-eq. Note that the numbers used here are for illustratory purposes only and not extracted from a real model.]
) <alignment-diagram>

#figure(
    image("scale_and_sum.png",  width: 80%), 
    caption: [Illustration of how the alignment scores are used to scale the respective value vectors for each sequence element, and are then summed to produce a new vector that contains global information embedded contextually. Each value vector is multiplied by the respective score, and then these scaled elements are summed together to produce the new vector.]
) <scaling-diagram>

This calculation is performed on each element of the sequence to produce a new sequence of equal length, hopefully with some contextual context embedded @attention_is_all_you_need @attention_review. Generalizing @attention-eq for the entire input matrix, $matrixn(X)$, we get

$ alpha(matrixn(Q), matrixn(K), matrixn(V)) = sigma((matrixn(Q) matrixn(K)^bold(T))/sqrt(d_"model")) matrixn(V) . $ <attention-eq-general>
  
Where again, $sigma$ is the Softmax function @softmax_ref. Combining @attention-eq-general with @weight-1-eq, @weight-2-eq, and @weight-3-eq gives a mapping between the attention input matrix, $matrixn(X)$, and the attention output matrix, $matrixn(Y)$.

$ matrixn(Y) = sigma(((matrixn(X) matrixn(W)_bold(q)) (matrixn(X) matrixn(W)_bold(k))^bold(T)) / sqrt(d_k)) (matrixn(X) matrixn(W)_bold(v)). $ 

The convenience that this complex procedure can be performed with a few matrix multiplications is one of the reasons for its great success. See @attention-diagram and @attention-network-diagram for illustrative diagrams.

#figure(
    image("attention_mechanism.png",  width: 100%), 
    caption: [Illustration of the operation of a single attention head. Here a very small three-element sequence is examined. Each element of the original input sequence is coloured differently, in red, green, and blue. All vectors and scalars associated with an input element are coloured similarly. The output sequence vectors are coloured with a mix of the input colours to show their new information content which consists of distilled global information. More detailed descriptions of the processes shown can be found in @weights_matricies, @alignment-diagram, and @scaling-diagram.]
) <attention-diagram>

#figure(
    image("attention_network_diagram.png",  width: 80%), 
    caption: [_Upper:_ Alternate method of visualizing attention mechanism as a network diagram. Although this is more similar to how networks have been displayed elsewhere in the thesis, it might obfuscate some aspects of the reasoning behind the attention layer operation. As in the @attention-diagram, this illustrates the operation of the attention mechanism on a sequence of length three, with each input vector coloured differently, in red, green, and blue. In this representation, the projection matrices, $matrixn(W)_bold(q)$, $matrixn(W)_bold(k)$, and $matrixn(W)_bold(v)$, are represented as dense layers, which are applied to each of the column vectors that comprise the input matrix in turn. It should be noted that although the dense layers are coloured differently as they are applied to each input element, this is just to show the different data flows, the weights are maintained by each application of each dense layer. The key, query, and value-dense layers, however, have different weights, and notably, no activation function, as they are just supplying a linear mapping rather than any more complex behaviour. _Lower:_ Abstraction of a single attention head layer, that will be used in future diagrams of models which contain attention layers, in order to limit diagram complexity.]
) <attention-network-diagram>

=== Multi-Head Attention

Thus far, the process we have described is the operation performed in a single attention head. We have worked under the assumption that all contextual information can be embedded locally with one pass from one head. In reality, this is not true, except for trivially simple sequences it would not be possible to embed all global information in one pass. In a similar manner to convolutional filters, wherein each filter looks at a particular feature of the input data, an attention layer typically has multiple heads each of which focuses on a particular information feature. One could look at colour for example, whilst another focuses on punctuation (if not removed in tokenisation), or sentence structure. 

In multi-head attention layers, the number of heads is a user-specified hyperparameter, N, just like the number of filters in a convolutional layer @attention_is_all_you_need @attention_review. Each head has independent weights for the query, $matrixn(W)_bold(q)$, key, $matrixn(W)_bold(k)$, and value, $matrixn(W)_bold(v)$, projection matrices, which are each tuned to find specific features in the data. After these heads have been applied the output is concatenated along the feature dimension, and then multiplied by a further weights matrix, used to mix the outputs of different heads and to reshape the output vector to a desired size, which does not necessarily have to be the same size as the input vector, though this is a common choice; see @multi-head-diagram for a representation of a multi-attention head.

It should be noted that in practice, all of the query, key, and value matrices for each head are calculated simultaneously with the same large weights matrices comprising the individual weights matrices from each head combined into large matrices. After multiplication with the input sequence, the large output matrices are split into separate matrices for each of the individual heads for the alignment scores calculation and vector summation. This is done to reduce the number of matrix multiplications required for the layer as a whole.

#figure(
    image("multi_attention_head.png",  width: 80%), 
    caption: [_Upper:_ Network diagram of multi-attention head. Similar to how multiple convolutional kernels work in tandem in convolutional layers, multiple attention heads work together in multi-attention heads to focus on different information aspects of the input vector. These are then concatenated along the feature axis before finally being multiplied by a further weights matrix, here shown as a dense layer, which serves to mix the output of the different heads and to reshape the output to a desired size. _Lower:_ Abstraction of a multi-head attention layer, that will be used in future diagrams of models which contain attention layers.]
) <multi-head-diagram>

=== Attention Blocks

Within transformers and other similar architectures, multi-head attention layers are often paired with a number of complementary layers within a residual block @attention_is_all_you_need @attention_review @transformer_review. The input and output matrices of this block usually have identical shapes so that the block can be repeated, $N$ times without having any intermediate reshaping layers. Attention blocks typically feature a number of dense layers with activation functions in order to perform non-linear computation, regularisation methods such as dropout and batch normalisation, and a residual skip connection wherein the block input is added to the block output, in order to reduce the vanishing gradient problem that can occur in very deep networks; see @attention-block-diagram.

#figure(
    image("attention_block.png",  width: 80%), 
    caption: [Typical attention block comprising multiple layers. Residual attention blocks vary in design between architectures but usually maintain the consistent elements shown. The skip connection is here represented by the encircling arrow, which shows that the input of the block is fed to the output before it is returned. There are also several regularisation methods present, batch normalisation, and dropout which help to reduce overfitting and ensure that values within the network remain bounded. Finally, the addition of dense layers and activation functions ensures that non-linear computation can be performed. Sometimes, if a reduction in total model parameter count and inference time is required, convolutional layers can be used in place of dense layers. The question marks indicate user-selectable hyperparameters. ]
) <attention-block-diagram>

=== Transformers <transformer-sec>

Since their introduction, attention mechanisms have been utilized in a number of different neural network architectures, including transformers @attention_is_all_you_need @transformer_review and generative diffusion models @diffusion_review. Transformers were first proposed by Vaswani _et al._ @attention_is_all_you_need to solve natural-language processing tasks, showing a significant improvement over previous recurrent and convolutional architectures. For these reasons, we decided to investigate a fully attention-based model, inspired by a Transformer encoder.

The transformer proposed by Vaswani _et al._ @attention_is_all_you_need consists of two branches each comprised of stacks of transformer blocks - the encoder and the decoder. See @transformer-diagram. Since the model is designed for sentence generation, its architecture is a little different from what we will use in Skywarp. The encoder model takes the input sequence, your input prompt adds positional encoding, and then runs that sequence through six encoder blocks, each consisting of a multi-attention head and two dense layers, both the multi-attention head and the two dense layers are surrounded by a residual skip connection and the outputs are normalized. The encoder converts an input sequence into a discrete latent space, which is then fed to the decoder, which similarly consists of a multi-attention head and two dense layers. In addition, the decoder has a cross-attention layer that receives the output of the encoder and converts that output into key-value pairs, matching them against the queries generated by the decoder sequence. Cross-attention allows attention maps to be generated between elements in different sequences, in opposition to self-attention where attention is only calculated within the sequence.

The design of this encoder-decoder serves the next token prediction capabilities of the transformer @attention_is_all_you_need @transformer_review. The purpose of the encoder is to encode the full training input sequence and provide a comparison for the decoder output. The decoder output then, gets a shifted input sequence, on the token to the right, with an additional start token. During the first attention layer in the encoder, a masking is added, so that no element can see any elements ahead of it in time. This is because whenever a token is at the end of a sequence it cannot see any future tokens. The outputs this process generates are then compared against the encoder output, which can see the whole input sequence, therefore the decoder learns to guess what the next tokens might be. That is a slight simplification but is an approximate description of the function of a standard transformer model. See @transformer-diagram.

#figure(
    image("transformer.png",  width: 100%), 
    caption: [The transformer model described by Vaswani _et al._ @attention_is_all_you_need. This encoder-decoder architecture can be used to generate predictions of the next token in a sequence. In the case of @attention_is_all_you_need, this sequence was natural language.]
) <transformer-diagram>

Since we are attempting to perform classification rather than next-element prediction, we will use only the encoder part of the transformer architecture, and adapt it to use gravitational-wave data.

== Transient Detection beyond CNNs <skywarp_review>

When expanding our literature search to include models outside of CNNs we encounter a large body of work that has expanded greatly in recent years, even since this work was carried out @gw_machine_learning_review. The volume presented makes it difficult to perform a complete review as it would be beyond the scope of this document. A review article @gw_machine_learning_review is suggested for a more extensive overview. 

An architecture commonly applied to time series problems is the Recurrent Neural Network @reccurant_neural_networks. Recurrent neural networks have an internal state determined by previous inferences, and thus, they can retain some information about previous data. In many ways, RNNs were the predecessor to Transformer models, largely because they are able, in some way, to make inferences from global information rather than being limited by the receptive fields of convolutional filters @cnn_review. There have been many studies on the application of RNNs to the transient detection problem. 

One of the most widely applied RNN subtypes is the Long Short Term Memory (LSTM) network @lstm_intro @lstm_chapter, which utilizes network controllable gates, which, based on learned computation, can control what data is preserved between inferences. Though this is not a permanent memory solution, as LSTMs will eventually lose recall ability from temporally distant inferences, it has the potential to store information across many thousands of inferences, hence the name Long Short-Term Memory. Bidirectional LSTMs, which are used almost exclusively throughout the literature, operate on sequences in both directions, allowing the model to see information both after and before the segment currently being analyzed. 

There have been several attempts to apply LSTMs to CBC detection. Schmitt _et al._ @bidirectional_lstm_comp_2 performed a comparison of multiple network architectures including bidirectional LSTMs and found that Temporal Convolutional Networks, which utilize a combination of convolutional and recurrent layers outperformed both the CNN employed by George _et al._ @george_huerta_cnn and traditional matched filtering approaches. Nigam _et al._ @bidirectional_lstm_comp perform a comparison of many different machine learning methods not limited to artificial neural networks or deep learning; whilst the results presented are limited and difficult to compare, they found CNNs and RNNs similarly effective. Chatterjee _et al._ @bidirectional_lstm present a model consisting of multiple convolutional and recurrent layers. They also present their model as a method for denoising signals for further analysis. They lack a robust discussion of False Alarm Rates. Lin _et al._ @bayesian_lstm present an interesting method that utilizes Bayesian networks along with LSTMs to incorporate uncertainty into the network. This allows the model to present its result along with variable confidence, though it demonstrates comparable predictive power it has a high false alarm rate, and they suggest that a larger training dataset is required to improve results.

Beveridge _et al._ @bidirectional_snr_lstm present a model that is imagined as a post-processing step that consumes the output of a matched filtering search, perhaps unsurprisingly considering their model input,  they achieve impressive accuracy scores down to very low FARs, although their method loses many of the benefits typically provided by deep learning searches: computational speed and cost, and is imagined as a way to improve current matched-filtering detection pipelines rather than replace them. 

Utilizing a different RNN architecture, Zhang _et al._ @bidirectional_gru demonstrated a Bidirectional Gated Recurance Unit (GRU) model, in comparison to a CNN and a Fourier Convolutional Neural Network (FCNN). They found the GRU to be the best-performing model and achieved an impressive performance of 89.6% accuracy at a False Alarm Rate (FAR) of #box($4.88 times 10^(−4)$ + h(1.5pt) + "Hz") on a pool of injections with SNRs between 5 and 20.

RNNs can be computationally expensive to train and run compared to other methods like CNNs @architecture_review, there have been several papers focused on performance improvements @accelerating_rnns @optimising_rnns_2.

There have also been attempts to apply LSTMs to burst detection problems, including a pattern-matching supernovae search @lstm_supernovae, and an unmodeled search using anomaly detection autoencoders @source_agnostic_lstm. The supernovae detection model by _Iess et al._ @lstm_supernovae demonstrates detection ability across several supernovae template banks but lacks false alarm rate discussion. Moreno _et al._ @source_agnostic_lstm use recurrent autoencoders for anomaly detection, autoencoders attempt to learn a function to project elements drawn from an input distribution into a dimensionally reduced latent space and then reconstruct the original input element from this reduced latent space. Because the encoder and decoder are trained on a specific distribution, if they are fed an element from outside the distribution, there will be a larger difference between model input and output, indicating an anomaly. In the case of burst detection, there would be no burst events in the training distribution, so if a burst event did come through the model, it would be out of distribution. They found the LSTM performed better than CNN and GRU autoencoders with accuracies of 30.4% for BBH signals, and 11.4% for BNS signals at a FAR of #box($0.01$ + h(1.5pt) +"Hz"). 

As a newer architecture, and one that is slightly less obvious to apply to time series data, since it is primarily used in NLP @transformer_review, there has been less attention on the application of transformers and attention-based models to transient detection problems. Yan _et al._ @coherent_snr_attention use attention layers in a coherence-based detection model. Zhao _et al._ @space_based_transformer focus on the problem of space-based detection. Jiang _et al._ @detection_conv_transformer, is the most relevant work, which was published around the time this investigation was being performed. They used a similar model architecture to that which is proposed by Skywarp, combining convolutional and attention layers and achieving results that are superior to purely convolutional models. During hyperparameter tuning, they found that the best performance was achieved by applying a single transformer layer after the convolutional head. One weakness of the paper is that their validation results are calculated using pools of mixed SNR, using the area under the ROC curve as their primary performance metric, this makes efficiency comparisons difficult.
 
== Skywarp Method <skywarp-method>

=== Skywarp Architectures <skywarp-model>

We have investigated a number of attention-based models and compared their performance to a CNN. We have investigated a fully attention-based model utilizing a dense embedding @skywarp_pure_attention, as well as a combined convolutional-attention model utilizing convolutional embedding, @skywarp_conv_attention. In response to Jiang _et al._ @detection_conv_transformer who found the best performance with a single attention block, a final model, with only one transformer block was created. For the pure CNN model, we adapted the model from Gabbard _et al._ @gabbard_messenger_cnn, with architecture as illustrated in @gabbard_diagram. We used this model as a reference CNN model to compare performance, this particular model was chosen as it is used often throughout the literature as a baseline CNN.

When transformers are utilised for Natural Language Processing (NLP) tasks, the input strings of natural language are first tokenised into discrete tokens before those tokens are fed into an embedding layer to convert the discrete tokens into continuous vectors that the network can ingest @tokenisation_review @attention_is_all_you_need. When adapting the architecture for use on time series data, there are some design decisions that must be taken. Tokenization, although still possible, is no longer required as the input data is initially in a continuous form. However, when deciding how to feed the series into the transformer, there are several options. Although it is possible to feed an attention block with the length of one vector from the input time series, it was found that this naive approach eliminated much of the transformer's potential for element-wise comparison. To resolve this, the method used by the vision transformer can be used; the input data can be segmented into $N$ segments, and then fed into the network. In addition or in place of such a segmentation, an embedding layer can also be employed to increase the dimensionality of the segments.

In the pure attention model, we reshaped the input time series (#box("1.0" + h(1.5pt) + "s") at  #box("8192.0" + h(1.5pt) + "Hz")) into 512 segments each consisting of 16 samples, these segments were then encoded into larger vectors with 128 elements by a single convolutional layer with a filter size of 1. This embedding was performed to allow sufficient size for the positional encoding to be added to each vector. This solution was found after trialling several variations. See @skywarp_pure_attention, for more detailed information on the network.

#figure(
    image("skywarp_pure_attention.png", width: 100%), 
    caption: [Skywarp pure attention model with dense embedding. This model architecture was used to test the purest application of attention layers to the detection problem by removing any convolutional layers within the model. The single convolutional layer was employed to increase the input dimensionality of the sequence elements from 16 to 128; this was necessary in order to add positional encoding of the appropriate size. Without positional encoding, models were almost impossible to train. The other hyperparameters were obtained using a rough trial-and-error search of the parameter space. Using a more sophisticated hyperparameters search, though a desired goal, proved difficult due to the increased computational requirements of attention-based models over CNNs. (Attention layer's memory usage scales approximately quadratically with input dimensionality, $N$: $O(N^2)$ @transformer_memory_usage similar to dense layers, as opposed to CNNs, which scale linearly with input dimensionality, $N$, and the number of filters, $F$, in the layers: $O(N times F)$ @cnn_memory_usage, so we encountered so difficulty fitting attention models in memory compared to CNNs.)]
) <skywarp_pure_attention>

During testing, we found that the pure attention model did not perform as well as the CNN model. It was found that the transformer model could much more easily overfit the training data, even with large training datasets. In order to combat this --- a combination convolutional-attention model was introduced. This model, described in @skywarp_conv_attention, feeds the output of the convolutional layers from the CNN described by @gabbard_diagram into the attention blocks described in @skywarp_pure_attention, in attempts to gain the benefits of both methods. The single-layer model shares the same architecture as @skywarp_conv_attention but with only a single attention block.

#figure(
    image("skywarp_conv_attention.png", width: 100%), 
    caption: [Skywarp convolutional attention with convolutional embedding. This model architecture was employed to test if a convolutional embedding scheme, using the proven CNN architecture of Gabbard _et al._ @gabbard_messenger_cnn embeds the input into feature maps which could then be input into attention layers. We have the greatest success with this model variety, again hyperparameters were manually tuned, so it is expected that with a more thorough hyperparameter investigation, a superior model could be found. ]
) <skywarp_conv_attention>

=== Skywarp Training, Testing, and Validation Data <skywarp-data>

The training dataset was very similar to the datasets used in previous sections. IMRPhenomD waveforms @imrphenom_d were generated using cuPhenom @cuphenom_ref and injected into real background noise taken from the LIGO Livingston detector from the 3#super("rd") LIGO-Virgo joint observing run using GravyFlow @gwflow_ref. The waveforms were generated with masses drawn from a uniform distribution between #box("5.0" + h(1.5pt) + $M_dot.circle$) and #box("95.0" + h(1.5pt) + $M_dot.circle$) for both companions and between -0.5 and 0.5 for the dimensionless spin component. A difference to note is that these waveforms were scaled with optimal SNRs drawn uniformly between 8.0 and 20.0, rather than between 8.0 and 15.0. There is no specific reason for this difference and were this experiment to be repeated these values would be standardised. Also, note that these experiments were performed with an earlier version of the GravyFlow data pipeline @gwflow_ref, so there may be some small inconsistencies, although both cuPhenom and the data acquisition functionality should operate very similarly to more contemporary versions. Another arbitrary difference is that real noise data was collected in batches of length #box("3600.0" + h(1.5pt) + "s") rather than #box("2048.0" + h(1.5pt) + "s"), this was changed after this work due to some stability improvements when working with power-of-two length data.

#figure(
  table(
    columns: (auto, auto),
    inset: 10pt,
    align: horizon,
    [*Hyperparameter*],  [*Value*],
    [Batch Size], [32],
    [Learning Rate], [10#super("-4")],
    [Optimiser], [ Adam ],
    [Scaling Method], [SNR],
    [Minimum SNR], [8.0],
    [Maximum SNR], [20.0],
    [SNR Distribution], [Uniform],
    [Data Acquisition Batch Duration], [ #box("3600.0" + h(1.5pt) + "s") ],
    [Sample Rate], [ #box("2048.0" + h(1.5pt) + "Hz")],
    [On-source Duration], [ #box("1.0" + h(1.5pt) + "s")],
    [Off-source Duration], [ #box("16.0" + h(1.5pt) + "s")],
    [Scale Factor], [10#super("21") ],
    
  ),
  caption: [The training and dataset hyperparameters used in Skywarp experiments. This is very similar to the data used for the burst and perceptron experiments seen in @perceptron-results. Note that the scale factor here refers to the factor used during the upscaling of the CBC waveforms and real interferometer noise from their extremely small natural dimensions to make them artificial neuron-friendly. This is done both to ensure that the input values work well with the network activation functions and learning rates, which are tuned around values near one, and to reduce precision errors in areas of the code that use 32-bit precision, employed to reduce memory overhead, computational cost, and duration. Data acquisition batch duration is a parameter of the GravyFlow data acquisition module. For speed, the GravyFlow @gwflow_ref data acquisition system downloads data in larger segments than is required for each training batch, then randomly samples examples from this larger segment to assemble each training batch. The data acquisition batch duration determines how long this larger batch is. Smaller values will result in a more evenly mixed training data set and a lower overall GPU memory overhead but will be more time-consuming during the training process. ]
) <skywarp-training-parameters>

=== Training Procedure <skywarp-training>

The training procedure is also very similar to previous experiments, using a learning rate of 10#super("-4"), a batch size of 32, and the Adam optimizer as well used industry standards, though it is possible that some of these hyperparameter decisions could be optimized with tuning. The training was performed in epochs with $10^6$ rather than $10^5$ examples, though the training validation pool remained at $10^4$.

== Skywarp Results <skywarp-results>

To profile the performance of Skywarp we compare it against a Convolutional Neural Network (CNN) with architecture taken from this early paper by Gabbard _et al._ @gabbard_messenger_cnn at different false alarm rates. As is the usual procedure, we first calculate the required model score thresholds to achieve various False Alarm Rates, see @skywarp_far_curve.

#figure(
  image("skywarp_far_curve.png", width: 100%), 
  caption: [Model False Alarm Rate (FAR) vs score threshold required to achieve that false alarm rate for the three Skywarp models and the recreated CNN model from Gabbard _et al._ @gabbard_messenger_cnn The four models display notably different FAR curve profiles, though it is important to note that a lower curve on this plot will not necessarily translate to model performance as it says nothing about the True Positive Rate, a classifier that labels everything as noise, for example, would be entirely flat on this plot, but would remain useless. Still, there is a notable difference between the curves, the pure attention model, consistently requires a much higher score threshold than the other three models, which will be seen in its poor efficiency performance.]
) <skywarp_far_curve> 

When we examine model efficiency curves in @skywarp_efficiency_curves, with key results extracted into @skywarp-results-table for easier reading of the small differences, the most evident takeaway is that the pure attention model utilizing dense embedding has by far the weakest performance, we can easily discount this method as ineffectual. It is not clear, however, whether this poor performance is due to the choice of embedding, or because attention layers alone are not adequate for gravitational-wave classification. We suggest that it is the former since there have been very successful image @vision_transformers and audio attention @audio_transformer models in other domains that do not utilize a convolutional head, and because, in theory, an attention layer is general enough to mimic any convolution. It is theorized that either the dense layer dimensionality increase that is performed to upscale the chunks enough to add positional encoding, is not adequate to convert the segments into a digestible feature space, or that the chunking is ineffectual because it splits the data without knowledge of the signal offset, creating sequence vectors which do not have consistent signal elements.

The other three models perform much more consistently. There is no clear advantage at a FAR of #box($0.1$ + h(1.5pt) + "Hz"), or #box($0.01$ + h(1.5pt) + "Hz"), although the attention-convolution hybrids consistently score higher, with the larger eight-layer model's advantage increasing with lower FARs. In many cases, this advantage is small however ranging from one to five percent. At 0.001 Hz, the lowest power-of-ten FAR where all models still have classification ability, it appears that the combined convolutional attention network has a more notable advantage over the other two competitive networks. However, this gap remains small enough to be somewhat unconvincing, and perhaps down to statistical variances in model training, which is not a perfect method and can fluctuate depending on the dataset order and parameter initialization. What can be said is that the advantage seen by the single attention layer over the larger eight-layer model that was suggested by Jiang _et al._ @detection_conv_transformer, was presumably down to their use of the area under ROC curves as performance metrics, since this metric benefits from higher efficiencies at lower FARs. At the lowest FAR presented #box($0.0001$ + h(1.5pt) + "Hz"), both the pure attention network and the single layer attention model ceased to have any classification ability, since their score thresholds have reached one. At this FAR the combined convolutional-attention model appears to greatly outperform the CNN. However, this is operating very close to the 32-bit precision limit, ($"threshold" > 0.999$), which would probably be uncomfortable for use in a transient search, as small statistical variations could have a large effect on model performance. The efficiencies are also quite low at this threshold, perhaps lower than one would wish to be to remain competitive with other detection methods.

#figure(
  table(
    fill: (_, row) => if calc.odd(calc.floor((row - 1)/4)) { luma(220) } else { white },
    columns: (auto, auto, auto, auto, auto),
    inset: 10pt,
    align: horizon,
    [*Model*], [*FAR (Hz)*],  [*Accuracy 6*], [*Accuracy 8*], [*Accuracy 10*],
    [Gabbard _et al._], [$10^(-1)$], [57.4], [91.5], [#text(red)[*99.4*]],
    [Skywarp Pure Attention], [$10^(-1)$], [41.5], [72.9], [91.2],
    [Skywarp Conv Attention Single], [$10^(-1)$], [#text(red)[*59.2*]], [#text(red)[*91.7*]], [99.3],
    [Skywarp Conv Attention], [$10^(-1)$], [57.2], [91.4], [#text(red)[*99.4*]],
    [Gabbard _et al._], [$10^(-2)$], [29.6], [76.6], [96.8],
    [Skywarp Pure Attention], [$10^(-2)$], [17.3], [50.0], [80.2],
    [Skywarp Conv Attention Single], [$10^(-2)$],  [#text(red)[*32.5*]], [78.6], [96.9],
    [Skywarp Conv Attention], [$10^(-2)$], [31.9], [#text(red)[*79.2*]], [#text(red)[*97.4*]],
    [Gabbard _et al._], [$10^(-3)$], [13.5], [57.6], [91.2],
    [Skywarp Pure Attention], [$10^(-3)$], [4.4], [22.0], [51.8],
    [Skywarp Conv Attention Single], [$10^(-3)$],  [13.3], [56.7], [90.2],
    [Skywarp Conv Attention], [$10^(-3)$], [#text(red)[*16.9*]], [#text(red)[*63.3*]], [#text(red)[*93.0*]],
    [Gabbard _et al._], [$10^(-4)$], [0.4], [8.7], [46.0],
    [Skywarp Pure Attention], [$10^(-4)$], [-], [-], [-],
    [Skywarp Conv Attention Single], [$10^(-4)$],  [-], [-], [-],
    [Skywarp Conv Attention], [$10^(-4)$], [#text(red)[*5.4*]], [#text(red)[*38.4*]], [#text(red)[*81.2*]],
  ),
  caption: [Accuracy results at different optimal SNRs from the four models tested at different FAR thresholds. Scores in red are the highest results for that SNR threshold at that FAR, in the one case where there is a tie, both scores are highlighted. With a very marginal lead, the single-layer attention-convolution hybrid appears the be the best model at a FAR of #box($10^(-1)$ + h(1.5pt) + "Hz"), only losing out to the CNN and deeper hybrid model by 0.1 percentage points at the highest SNR of 10. This is not a particularly useful FAR range, however, and as we decrease the FAR, the deeper attention layer seems to be victorious, but again the lead is quite small. This appears to show that the convolutional-attention model may have a slight advantage at lower FARs. At the lowest FAR presented, #box($10^(-4)$ + h(1.5pt) + "Hz"), the required score threshold for both convolutional-attention hybrid with the single attention layer, and the pure attention model, have reached one, and therefore lack any classification ability. For the remaining models, the required model score threshold is greater than 0.999, and although at this FAR the deep attention hybrid appears to be the clear winner, it is unclear whether victory at such a high score threshold is meaningful, or simply due to statistical variance in model training and the validation dataset. Although it should be noted that the lead is similar in all SNR bins, which were created independently, so if it is statistical variance, it is suggested that it probably originates in the training procedure.]
) <skywarp-results-table>

It should be reiterated that these are single-detector results, in a real search pipeline. The FAR would have a significant advantage. Assuming a detection pipeline with no overlap, #box($1.0$ + h(1.5pt) + "s") duration, and two detectors, the network FAR is given by $"FAR"_1 times "FAR"_2 times t$, we assume that $"FAR"_1 = "FAR"_2$ and our $ t = 1.0 s$ and since we have two detectors they can overlap in either direction, adding a factor of 2, therefore our network FAR is given by $"2.0 times FAR"^2 times 1 s$ which turns the presented FARs from #box($10^(-1)$ + h(1.5pt) + "Hz"), #box($10^(-2)$ + h(1.5pt) + "Hz"), #box($10^(-3)$ + h(1.5pt) + "Hz"), and #box($10^(-4)$ + h(1.5pt) + "Hz"), into #box($2 times 10^(-2)$ + h(1.5pt) + "Hz"), #box($2 times 10^(-4)$ + h(1.5pt) + "Hz"), #box($2 times 10^(-6)$ + h(1.5pt) + "Hz"), and #box($2 times 10^(-8)$ + h(1.5pt) + "Hz") respectively. The latter two of which begin to approach the regime of other detection pipelines; the threshold for significant burst detections used by the LVK collaboration is #box($3.9 times 10^(-y)$ + h(1.5pt) + "Hz") (once per month) for CBCs and #box($3.2 times 10^(-8)$ + h(1.5pt) + "Hz") (once per year) for bursts @ligo_far_threshold. However, in practice, it is expected that some overlap would be employed to reduce the change of a signal falling at a boundary. As well as efficiency curves, ROCs are also presented for comparison with other results, @skywarp_roc_curves. Due to the nature of the current calculation of the ROC curves, their maximum FAR resolution is considerably lower than what is prested in @skywarp_efficiency_curves, due to lower numbers of validation examples, so these plots cannot be as effectively used to compare performance at different FARs, but can give a general impression of model performance at higher FARs.

#figure(
    grid(
        columns: 1,
        rows:    2,
        gutter: 1em,
        [ #image("skywarp_efficiency_0_1.png", width: 100%) ],
        [ #image("skywarp_efficiency_0_01.png", width: 100%) ],
        [ #image("skywarp_efficiency_0_001.png", width: 100%) ],
        [ #image("skywarp_efficiency_0_0001.png", width: 100%) ],
    ),
    caption: [Efficiency curves of the Skywarp models and the recreated model from Gabbard _et al._ at different False Alarm Rates. For each of the 4 models trained, 61 efficiency tests are performed at evenly spaced optimal SNR values between 0 and 15. For each test, 16384 examples with signals of the relevant SNR are examined by the model. The most evident distinction is between the pure attention model utilizing dense embedding, and the other models, which are either purely convolutional or have a convolutional head. There is considerably less distinction between the other three models, which appear statistically indistinguishable at FARs of $10^(-1)$ Hz and $10^(-2)$ Hz. A slight advantage may arise between the combined attention-convolution model and the other two competitive models at the $10^(-3)$ Hz, which is perhaps the strongest evidence of an advantage, but this small difference is still too small to draw any definite conclusions of improved efficacy. At $10^(-4)$ Hz, the difference is much more apparent, but we are approaching 32-bit precision limits, so it is unclear exactly how seriously we should view these results. The efficiencies at this low FAR are also considerably reduced, reducing the appeal of the use of these models at this FAR. _First:_ Efficiency curves at a FAR of #box($10^(-1)$ + h(1.5pt) + "Hz"). _Second:_ Efficiency curves at a FAR of #box($10^(-2)$ + h(1.5pt) + "Hz"). _Third:_ Efficiency curves at a FAR of #box($10^(-3)$ + h(1.5pt) + "Hz"). _Fourth:_ Efficiency curves at a FAR of #box($10^(-4)$ + h(1.5pt) + "Hz").]
) <skywarp_efficiency_curves>

#figure(
    grid(
        columns: 1,
        rows:    2,
        gutter: 1em,
        [ #image("skywarp_roc_8_20.png", width: 100%) ],
        [ #image("skywarp_roc_12.png", width: 100%) ],
        [ #image("skywarp_roc_10.png", width: 100%) ],
        [ #image("skywarp_roc_8.png", width: 100%) ],
        [ #image("skywarp_roc_6.png",   width: 100%) ],
    ),
    caption: [Receiver Operator Curves (ROCs) generated for each of the four Skywarp models for a variety of different SNR pools. The story demonstrated by these plots is very similar to what is shown by the efficiency curves, @skywarp_efficiency_curves, albeit with less granularity. The pure attention model performs considerably worse than the other three models, which are nearly indistinguishable. The pure convolution model has a slightly higher area under the ROC curve, primarily boosted by higher efficiencies at higher false alarm rates and the combined convolutional-attention network achieves higher accuracies at the lower FAR ranges, with the single-layer convolutional-attention network somewhere between the two. _First:_ ROC generated with a pool of signals with optimal SNRs drawn uniformly between 8 to 20. _Second:_ ROC generated with a pool of signals with optimal SNRs of 12. _Third:_ ROC generated with a pool of signals with optimal SNRs of 10. _Fourth:_ ROC generated with a pool of signals with optimal SNRs of 8. _Fifth:_ ROC generated with a pool of signals optimal SNRs of 6.]
) <skywarp_roc_curves>

== Discussion <skywarp-discussion>

The use of attention layers in CBC detection models shows promise. Although these results fail to prove a decisive advantage, they demonstrate that even with rudimentary parameter optimization, a model can be generated to beat a CNN from the literature that is often used as a standard. One firm conclusion that can be drawn is that the use of a pure attention model, with the dense embedding presented, performs significantly worse than the other methods presented. 

Although it does not seem like there is a very significant advantage to using the Skywarp transformer architecture presented over the more traditional CNN model, it does appear that attention layers are better adapted for work in low FAR regimes. A more complete hyperparameter optimization procedure, and a more thorough investigation of signal embedding methods, as well as experiments in multi-detector inputs utilizing inter-detector cross attention, may yield more significant performance benefits.