#set page(numbering: "1", number-align: center)

#set math.equation(numbering: it => {[7.#it]})
#counter(math.equation).update(0)

= CrossWave: Cross-detector Attention for the Detection and Parameterisation of Overlapping Gravitational-Wave Compact Binary Coalescences <crosswave-sec>

Thus far, we have focused our attention on perhaps one of the simpler problems in gravitational-wave data analysis, transient detection; the fact remains, that many, more complex, tasks are yet to be satisfactorily solved. One of the largest and most intriguing of these is Parameter Estimation (PE) @parameter_estimation_review. Whilst detection merely identifies the presence of a signal, and, in a modeled search, tells us the type of signal we have detected, there is invariably other scientifically valuable information that can be extracted from a signal. During PE, we attempt to predict, with error margins, several parameters about a gravitational-wave-producing system. Typically this is a CBC system, although PE could also be performed on burst events if they were to be detected @burst_pe. Fortunately, CBCs can be described quite well in as few as 14 parameters that contain information both about the internal state of a CBC system, known as intrinsic parameters, and its relation to us as observers, known as extrinsic parameters @intrinstic_params. Care should be taken to distinguish between the parameters being extracted by PE, and the parameters of a neural network model, as they are unrelated.

Without further analysis, detection alone is useful for little more than rudimentary population analysis; PE, therefore, is a crucial part of gravitation-wave data science @gravitational_wave_science_overview. Extrinsic parameters, like the source distance and sky location, aid in population studies and multimessenger analysis @multimessenger_ref, and intrinsic parameters such as the companion mass and spin properties can help unveil information about the underlying physics of sources themselves @starquake_detection @neutron_star_equation_of_state_1 @neutron_star_equation_of_state_2, as well as their formation channels @gravitational_wave_population.

This section does not focus on a general PE method for either CBC or burst signals. Those have both been well investigated and although there is arguably a greater room for improvement and a larger need for innovation on these fronts than in detection alone it was not within the scope of this work. In this section, we present an analysis of a much smaller subproblem within PE; the detection and isolation, of overlapping signals contained within a single stretch of interferometer data. Because of the somewhat limited nature of the problem, it has not been studied as thoroughly as any of the other problems we have yet examined, which, in some ways, gives us more space for exploration, and an increased potential for novel scientific output.

== Frequency of Overlapping Compact Binary Coalescences (CBCs)

Significant improvements to our gravitational wave detection capability are anticipated within the next few decades, with improvements to existing detectors such as LIGO-Voyager @LIGO_Voyager, as well as future 3#super("rd") and 4#super("th") generation space and ground-based detectors such as the Einstein Telescope (ET) @einstein_telescope, Cosmic Explorer (CE) @cosmic_explorer, and the space-based Laser Interferometer Space Antenna (LISA) @LISA. Whilst the current detection rate ($1~2 space "week"^(-1)$ [BBHs]) and average detectable duration ($~7 s$ [BBHs]) of Compact Binary Coalescences (CBCs) is too low for any real concern about the possibility of overlapping detections @bias_study_one, estimated detection rates ($50~300 space "week"^(-1)$ [BBHs]) and durations ($40~20000 s$ [BBHs]) for future networks will render such events a significant percentage of detections @bias_study_one. See @overlaping-event-rate for a more detailed breakdown of overlapping event estimates. Contemporary detection and PE pipelines do not currently have any capabilities to deal with overlapping signals --- and although, in many cases, detection would still be achieved @bias_study_one @bias_study_two, PE would likely be at least somewhat compromised by the presence of the overlap, especially if more detailed information about higher modes and spins @bias_study_one are science goals.

#figure(
  table(
    columns: (auto, auto, auto, auto, 80pt, 70pt, 70pt),
    inset: 10pt,
    align: horizon,
    [*Configuration*],  [*Range (MPc)*], [*Cut Off (Hz)*], [*Mean Visible Duration (s)*], [*P(Overlap) ($"year"^(-1)$)*], [*$N_"events"$ ($"year"^(-1)$)*], [*$N_"overlaps"$ \ ($"year"^(-1)$)*],
    [aLIGO: O3], [611.0], [20], [6.735], [$3.9_(-1.3)^(+1.9) times 10^(-6)$], [$42.0_(-13.0)^(+21.0)$], [$0.0_(-0.0)^(+0.0)$],
    [aLIGO: O4], [842.5], [20], [6.735], [$1.0_(-0.3)^(+0.5) times 10^(-5)$], [$100.0_(-29.0)^(+56.0)$], [$0.0_(-0.0)^(+0.0)$],
    [aLIGO: Design], [882.9], [20], [6.735], [$1.2_(-0.4)^(+0.6) times 10^(-5)$], [$120.0_(-38.0)^(+60.0)$], [$0.0_(-0.0)^(+0.0)$],
    [LIGO-Voyager], [2684.0], [10], [43.11], [$2.3_(-0.8)^(+1.2) times 10^(-3)$], [$2700.0_(-38.0)^(+60.0)$], [$6.3_(-3.4)^(+7.7)$],
    [Einstein Telescope], [4961.0], [1], [19830.0], [$1.0_(-0.0)^(+0.0)$], [$15000.0_(-5000.0)^(+7100.0)$], [$15000.0_(-5000.0)^(+7100.0)$],
  ),
  caption: [Estimated overlap rates of BBH signals in current and future detectors, sourced from Relton @phil_thesis and Relton and Raymond @bias_study_two. Presented error values are 90% credible intervals. Note that these results, including past observing runs are estimates rather than real values, and are meant only as an illustration of the probable difference in overlap rates between current and future detector configurations. The number of overlapping signals, $N_"overlap"$, anticipated within one year is determined by the number of detections, $N_"events"$, and the visible duration of those detections, which are, in turn, affected by the detection range and lower frequency cut off the detector configuration in question. We can see that although with the current and previous detector configurations an overlapping event is extremely unlikely, it will increase with LIGO-Voyager to the point where we would expect $6.3_(-3.4)^(+7.7)$ overlapping signals per year of observing time, and further increase with the Einstein Telescope to the point where we would not expect any event to be detected without components of other signals also present in the detector. Similar overlaps are expected for LISA and Cosmic Explorer.]
) <overlaping-event-rate>

== Detection and Parameter Estimation (PE) of Overlapping Compact Binary Coalescences (CBCs)

Two studies examined the rate at which overlaps were likely to occur in different detector configurations along with the effect of overlapping signals on PE. Samajdar _et al._ @bias_study_one, determined that during an observing period of the future Einstein Telescope, the typical BNS signal will have tens of overlapping BBH signals and that there will be tens of thousands of signals per year that have merger times within a few seconds of each other. They found that for the most part, this had little effect on parameter recovery except in cases where a short BBH or quiet BNS overlapped with a louder BNS signal. Relton and Raymond @bias_study_two performed a similar study and produced the overlap estimates seen in @overlaping-event-rate. They found that PE bias was minimal for the larger of the two signals when the merger time separation was greater than #box($0.1$ + h(1.5pt) + "s") and when the SNR of the louder signal was more than three times that of the quieter signal. This bias was also smaller when the two signals occupied different frequency regions, and when the louder of the two signals appeared first in the detector stream. Despite this, they found evidence of PE bias even when the smaller signal was below the detectable SNR threshold. They found that overlapping signals can mimic the effects of procession; it will be important to be able to distinguish the two when detailed procession analysis becomes possible.

Much of the work in this area focuses on performing PE with overlapping signals, and there has not been as much attention to simply distinguishing pairs of mergers from single mergers. Relton _et al._ @overlapping_search measured the detection performance of both a modelled (PyCBC) @pycbc and unmodeled (coherent WaveBurst [cWB]) @cWB search pipeline when searching for overlapping signals. They determined that both pipelines were able to recover signals with minimal efficiency losses ($<1%$) although they noted that the clustering algorithm used in both pipelines was inadequate to separate the two events. They concluded that adjustments to clustering could be made to both pipelines in order to return both events given a sufficient merger time separation. Using these altered pipelines it would then be possible to separate the data into two regions, which could be used for independent PE.

Once an overlapping single has been identified, the next step is to deal with PE. Although in many cases, existing PE techniques may provide results with little bias @bias_study_one @bias_study_two, there are some situations in which this may not be the case. If the PE method can be improved in order to reduce that bias, it is useful so long as it does not result in a reduction of PE accuracy that is greater than the bias introduced by the overlapping signal.

There are four types of methods we can apply to alleviate the issues with PE @phil_thesis. 

+ *Global-fit* methods attempt to fit both signals simultaneously. There have been several studies investigating this method by Antonelli _et al._ @global_fit, which attempts to apply it to both Einstein Telescope and LISA data, @hieherachical_overlapping_pe_2 which compares this method to hierarchical subtraction, and several studies focusing solely on LISA data @lisa_global_1 @lisa_global_2 @lisa_global_3. This has the advantage of being somewhat a natural extension of existing methods, with no special implementation other than an increased parameter count, but that can also be its greatest disadvantage. The total number of parameters can quickly become large when an overlap is considered, especially if multiple overlaps are present which will be expected to occur in ET and LISA data.

+ *Local-fit* methods attempt to fit each signal independently and correct for the differences. The original proposal by Antonelli _et al._ @global_fit suggests using local fits to supplement a global-fit approach. This will reduce the number of parameters that you require your method to fit, but its efficacy is highly dependent on the proficiency of your correction method.

+ *Hierarchical Subtraction* methods suggest first fitting to the most evident signal, then subtracting the signal inferred from your original fit and repeating this process for all anticipated signals @hiherachical_subtration_overlapping_pe @hieherachical_overlapping_pe_2. This method would be effective at subtracting multiple sets of parameters for overlapping signals, assuming that the overlap does not cause bias in the initial fit, which the previously mentioned studies have shown is not always a correct assumption @bias_study_one @bias_study_two.

+ Finally, and most relevantly, *machine learning* methods can be employed as a global fit technique to try and extract parameters from overlapping signals. They come with all the usual advantages, (inference speed, flexibility, computational backloading) and disadvantages (lack of interpretability, unpredictable failure modes). Langendorff _et al._ @machine_learning_overlapping_pe attempt to use normalizing flows to output estimations for parameters.

Most of the aforementioned methods benefit from having prior knowledge about each of the pairs of signals, especially the merger times of each signal. As well as acting as a method to distinguish between overlapping and lone signals, CrossWave was envisioned as a method to extract the merger times of each of the binaries in order to assist further PE techniques. Crosswave was able to achieve this and also demonstrated some more general, but limited PE abilities.

== CrossWave Method

We introduce Overlapnet and CrossWave, two neural network models for the identification and PE of overlapping CBC signals. This section describes two complementary models, Overlapnet for the separation of the overlapping case from the non-overlapping case and CrossWave as a PE follow-up to extract the merger times of the overlapping signals in order to allow other PE methods to be performed. Overlapnet can differentiate between overlapping signals and lone signals with efficiencies matching that of more conventional matched filtering techniques but with considerably lower inference times and computational costs. CrossWave can extract the merger times of the two overlapping CBCs with an average error of less than 0.1 s. We suggest these two models or similar architectures may be used to augment existing CBC detection and PE infrastructure, either as a complementary confirmation of the presence of overlap or to extract the merger times of each signal in order to use other PE techniques on the separated parts of the signals.

Since the CrossWave project was an exploratory investigation rather than an attempt to improve the results of a preexisting machine learning method, it has a different structure to the Skywarp project. Initially, we applied architecture from the literature, again taking Gabbard _et al._ @gabbard_messenger_cnn, with architecture illustrated here @gabbard_diagram. This worked effectively for the differentiation of overlapping and lone signals. We named this simpler model OverlapNet. However, when attempting to extract the signal merger times from the data, we found this model to be inadequate, therefore, we utilized the attention methods described in @skywarp-sec, along with insights gained throughout other projects to construct a more complex deep network for the task, seen in @crosswave-large-diagram. We name this network CrossWave, as it utilises cross attention between a pair of detectors. It is hoped that this architecture can go on to be used in other problems, as nothing in its architecture, other than its output features, have been purpose-designed for the overlapping waveform case.

=== Crosswave Training, Testing, and Validation Data <crosswave-data>

The dataset utilized in this section differs from previous sections, in that it was not generated using the GravyFlow data pipeline. Since this was part of a PE investigation, the exact morphology of the waveforms injected into the signal is crucial to validating performance. The cuPhenom IMRPhenomD waveform generator that was developed for rapid waveform generation on the GPU has a relatively high degree of mismatch (~5%) with IMRPhenomD signals @imrphenom_d generated with LALSimulation @LALSimulation in some areas of parameter space. This is primarily thought to be caused by cuPhenom's @cuphenom_ref reduced precision (32-bit in most areas rather than 64-bit) and the lack of implementation of several post-Fourier conditioning steps. Whilst this mismatch was deemed to be mostly adequate for detection searches, especially for comparison of methods, we considered it inadequate for PE tasks. IMRPhenomD is also an older approximant, which does not take into consideration the latest improvements to waveform approximation including several physical phenomena, such as higher modes. Whilst there is currently no one approximant that can generate waveforms that include all physical effects, we opted to use IMRPhenomTPHM @imrphenom_future, which is a Time-Domain approximant that includes the physics of precession, which allows for studies of Higher Modes.

A static dataset was created using BBH waveforms generated using LALSimulation @LALSimulation and injected into Gaussian noise coloured by the LIGO Hanford and LIGO Livingston aLIGO design specifications @aLIGO_design_specificaton using the technique described in @noise_acquisition_sec but not with the GravyFlow @gwflow_ref pipeline. No BNS signals were considered. We used a #box("16" + h(1.5pt) + "s") on-source duration, to allow more space for different signal start times and to examine the effects of distant signal overlap on PE. We used a sample rate of #box($1024$ + h(1.5pt) + "Hz"), as this was considered adequate to contain the vast majority of relevant frequency content for the CBCs examined.

Unlike in the detection case, wherein our training distribution consisted of some examples with obfuscated signals and some consisting of pure noise, we assume that a detection has already been made by a detection pipeline, so our examples always contain signal content of some kind. This assumption was made to simplify the task to its minimal possible case. Our generated waveform bank consisted of $2 times 10^5$ IMRPhenomTPHM approximants. From that template bank, we constructed $2 times 10^5$ examples of lone signals injected into obfuscated noise and $2 times 10^5$ examples of pairs of signals injected into obfuscated noise, totaling $4 times 10^5$ training examples. In the latter case, each waveform was unique to a single pair, generating $10^5$ pairs, but each pair was injected into two different noise realizations in order to generate identical numbers of lone and paired templates. The use of the same waveforms in both the single case and the pairs was a conscious decision that was made in order to attempt to reduce the chance of the network overfitting to any particular signal morphologies that it learned to associate specifically with lone signals or pairs of signals.

The waveforms were generated with a wide parameter range drawn uniformly from across parameter space. The primary component of each waveform was generated with masses between #box("10.0" + h(1.5pt) + $M_dot.circle$) and #box("70.0" + h(1.5pt) + $M_dot.circle$), this is notably inconsistent with our previous studies, but was reduced to reduce task complexity and because this still covers most of the range that is of interest to PE studies. This also ensured that their visual duration, (starting at #box("20.0" + h(1.5pt) + "Hz"), which is both the whitening low-pass filter and around the limit that the detector design curve starts to make detection impossible), remained well contained within the #box("16" + h(1.5pt) + "s") on-source duration. Also unlike in our previous detection studies, the mass ratio was constrained between 0.1 and 1. Since the approximants were generated in an alternate method utilising luminosity distance as the scaling factor rather than SNR, the SNRs are not uniformly distributed, however, the Network SNR of any signal is not less than 5 or greater than 100. For each injection, luminosity distance in MPc was drawn from a power law distribution with base two scaled by 145, with a minimum distance of #box("5.0" + h(1.5pt) + "MPc"), this luminosity distance range was generated by a trial and error approach to achieve the desired SNR distribution. An overview of the parameters used to train both the CrossWave and Overlapnet models is shown in @crosswave-training-parameters.

A validation dataset was also generated with independent signals and background noise, with $2 times 10^4$ singles and $2 times 10^4$ pairs generated similarly to the training data but with different random seeds, totalling $4 times 10^4$ validation examples.

#figure(
  table(
    columns: (auto, auto),
    inset: 10pt,
    align: horizon,
    [*Hyperparameter*],  [*Value*],
    [Batch Size], [32],
    [Learning Rate], [10#super("-4")],
    [Optimiser], [ Adam ],
    [Scaling Method], [Luminosity Distance],
    [Min Luminosity Distance], [5.0],
    [Max Luminosity Distance], [N/A],
    [Luminosity Distance Distribution], [$ ("Power-Law (base 2)" times 145) + 5 "MPc"$ ],
    [Data Acquisition Batch Duration], [ N/A ],
    [Sample Rate], [ #box("1024.0" + h(1.5pt) + "Hz")],
    [On-source Duration], [ #box("16.0" + h(1.5pt) + "s")],
    [Off-source Duration], [ N/A ],
    [Scale Factor], [10#super("21") ],
    
  ),
  caption: [The training and dataset hyperparameters used in CrossWavea and Overlapnet experiments.]
) <crosswave-training-parameters>

In the case of the pairs of injections, the two waveforms are injected so that their merger times never have a separation exceeding #box("2" + h(1.5pt) + "s"). "Signal A" is defined as the signal whose merger arrives second at the Earth's centre, whereas "Signal B" is always defined as the signal whose merger time arrives first. This allows the model to differentiate between the two signals for the PE tasks. When only one waveform is present, that waveform is labelled "Signal A". It is possible, if the merger arrival time separation between signal A and signal B is low enough, that the order of A and B in the Hanford and Livingston detectors may be switched, if this is the case it would only happen in very few cases since the vast majority of training and validation examples with pairs of signals have merger time separations larger than the light travel time between detectors (~#box($0.01$ + h(1.5pt) + "s")).

#figure(
    grid(
        columns: 1,
        rows:    2,
        gutter: 1em,
        [ #image("single_example.png", width: 80%) ],
        [ #image("overlap_example.png", width: 80%) ]
    ),
    caption: [Two illustrative examples of 1the example used to train CrossWave, the upper demonstrates the single signal case, the lower the multiple signal case. Since the real data used to train CrossWave was unwhitened, it is not easy to parse by eye. Thus, as an illustrative example, these two examples are shown in whitened data generated using cuPhenom and GravyFlow. The example duration has also been cropped from #box("16" + h(1.5pt) + "s") to #box("5" + h(1.5pt) + "s"), since the merger times never have a separation greater than #box("2" + h(1.5pt) + "s") this is ample as an example. Both examples show time series from both detectors, simulating LIGO Livingstone and LIGO Hanford. _Upper:_ Single waveform injected into noise drawn from the two LIGO detectors. _Lower:_ A pair of waveforms injected into noise drawn from the two LIGO detectors. The waveforms are always injected with merger times less than #box("2" + h(1.5pt) + "s") distant.] 
) <overlap_injection_examples>

==== A note on Whitening

Interestingly, since the data was generated independently from GravyFlow, it was not whitened prior to model injection. Since this is not a comparison to other machine learning methods that use whitening, this is not particularly an issue, but it also can't tell us about the efficacy we have lost/gained due to the lack of whitening. Since this investigation does have positive results, this could potentially be an area for future experimentation, forgoing the whitening step before ingestion by a model would streamline a lot of the problems faced by low-latency machine learning pipelines. It should be remembered, however, that the training and validation data was generated using the unchanging PSDs of the aLIGO design specification @aLIGO_design_specificaton for each given detector. Attempting to train models with real or pseudo-real noise which is non-stationary, and in the former case contains non-linear glitches, may not be as viable.

The use of unwhitened noise, if possible, may have some benefits in the case of overlapping signal detection and PE. Because this work is only expected to become relevant in the regime of very long-lived signals, it may be difficult to get clean off-source data at a close enough time separation from the on-source data which is not also contaminated with other signals.

== Overlapnet Results

=== Classification

The first attempt to classify input examples generated with the method described in @crosswave-data utilized an architecture from the literature adapted from Gabbard _et al._ @gabbard_messenger_cnn, the model architecture of this model can be seen at @gabbard_diagram. To distinguish this model from later models, this model was named Overlapnet. We trained a binary classifier to output a score near or equal to one if there were two signals present in the input data, and a score near or equal to zero if there was only one signal in the data.

Since data for this experiment was generated independently, validation was also performed alternately. Since we are assuming the presence of at least one signal, in either case, the problem is not hugely asymmetric as it is in the CBC detection case. The penalty for incorrectly classifying a single signal as a double is much less than for classifying noise as a signal. This is because a detection model must examine many more noise examples than an overlap detection model would signals since we are assuming at least one signal has already been detected by another method. If we assume that misidentifying a single signal as a double is as undesirable as misidentifying a pair of signals as a single, we can set a classification threshold of 0.5 so that neither class is favoured (unless the model intrinsically favours one class over another, which is also possible), as an equal ratio between classes is usually recommended to maximize overall model performance @inbalanced_datasets. This means we can focus on optimizing our model to gain as high accuracy as possible, without needing performance in extremely low FAR regimes; therefore FAR plots are not particularly useful.

The trained model was run over the validation dataset consisting of $4 times 10^4$ examples generated independently but with the same method as the training data. The parameters for each waveform were recorded and compared with the classification results. 

This initial attempt at applying a preexisting model from the literature to the problem proved sufficient even in unwhitened noise. The model was able to correctly classify most pair examples where both of the optimal network SNRs are above 10, and correctly identify almost all single signals. See @overlapnet_classification_scores.

For the single signal validation examples, the model can correctly identify almost all cases, (assuming a detection score threshold of 0.5). We note that although the classification error very rarely exceeds 0.5, there is still some notable error. It is thought that this may be because of deficiencies in the construction of the dataset. Since there is very little to differentiate between pairs of signals where one signal is rendered almost indetecable due to a small SNR, and single signals; and between pairs of signals where both detectors have a low SNR and a single signal with a low SNR, this adds significant confusion to the training process, which encourages the model to show less confidence when classifying signals as singles. This could be ameliorated by increasing the minimum SNR threshold of the signals to the point where no (or fewer) training examples have one undetectable signal, although this change may come to the detriment of other classification abilities.

In the pair validation examples, the model has a much wider range of detection ability determined by the optimal network SNR of each of the examples' two signals. The model shows good performance when both signals have an optimal network SNR of at least ten, with a rapid decline below these values, which is roughly consistent with the efficiency curves we see in detection cases. This is anticipated. When one of the signals has a low SNR, the example becomes very similar to a single signal; when both of the signals have a low SNR, the example becomes indistinguishable from a single signal with a low SNR. In both of these cases, the model prefers to classify examples as single signals rather than double. This makes sense, the model will try to minimize the difference between its output and the ground truth values, half the examples in the training dataset are single signals, whereas considerably less than half the signals are pairs of signals with one low SNR --- if the model has to guess between the two, it is more likely that the example will be in the former category than the latter. This is also probably true for real signals, so this is possibly not a bad feature of the classifier. For the case when both signals in a pair have low SNR, it also makes sense that the classifier would want to classify these as single signals, as there are many more examples of a single signal with a low SNR in the training dataset than there are of a pair of signals both with a low SNR.

It is also speculated that the model may have learned to associate low overall excess power with single signals. Since the two classes were not normalized to contain roughly equal excess power, the average excess power found in pair examples will be double that of the average excess power found in single examples. This is certainly a feature that the classifier could have learned to recognize. This could be alleviated by normalizing the excess power between the classes, which would force the detector to rely on the signal morphologies alone rather than the excess power contained within the example. It is not clear whether this would be a useful feature or not. Certainly, in nature, overlapping signals would, in general, mean greater excess power, but this may have detrimental effects in model training.

#figure(
    grid(
      columns: 2,
      rows:    1,
      gutter: 1em,
      [ #image("crosswave_classification_corrected.png",  width: 100%) ],
      [ #image("overlapnet_zoomed_classification.png",  width: 100%) ],
    ),
    caption: [Classification error of Overlapnet output when fed validation examples, plotted with signal A optimal network SNR and signal B optimal network SNR. A total of $4 times 10^4$ validation examples were used to produce this plot. All examples consist of two-channel synthetic detector noise generated by colouring Gaussian white noise with the LIGO Hanford and LIGO Livingston aLIGO design specifications. Half the validation examples were injected with one each of $2 times 10^4$  IMRPhenomTPHM waveforms with no repetitions, these are the single injection examples, which only contain Signal A. In these cases the SNR of signal B is always zero, these signals are seen arranged along the bottom of the plot. The other half of the examples consist of two each of the same $2 times 10^4$ IMRPhenomTPHM waveforms with two repeats of the same pairs of signals injected into different noise realizations. A model score near one indicates the model has determined that the example has two hidden signals and a score near zero indicates that the model thinks the example has only one hidden signal. The classification score error shows the difference between the ground truth value and the predicted model output. Therefore an error nearer zero indicates good model performance, and an error nearer one indicates poor model performance. Assuming a classification threshold of 0.5 we can see that the model can successfully classify almost all single examples, and can successfully classify most pairs of signals when the Network SNR of both signals is above an optimal SNR of ten. We note that although classification is achieved in most cases, there is still substantial error in many cases, though mostly below the threshold required for an inaccurate classification, 0.5. It is theorised that this is because the model is trained with many examples of pairs of detectors with one low SNR that are hard to distinguish from single detectors with one signal. This confusion could add considerable uncertainty to the model predictions, and it is recommended that if this project were to be repeated the minimum SNR threshold for both of the signals should be increased. When either of the optimal network SNRs of one of the signals falls below 10, the rate of classification error increases in a curve that is consistently shaped with the detection efficiency curves discussed in previous sections. This is anticipated --- in the case that one of the SNRs becomes low, the signal will appear to look like a single signal as the other signal becomes hard to distinguish. In the case where both signals have a low SNR, both signals are hard to distinguish and it becomes difficult to differentiate between a single hard to identify signal and multiple hard to identify signals. In this latter case, where both signals have a low SNR, the model appears to favour classification as a single signal rather than double. It is hypothesized that this may be because the pairs and single examples were not normalized to have consistent excess power, meaning that the total excess power contained in the set of all two signal examples will be double that of the total excess power in all single signal examples. This might bias the network to associate low excess power with single signal examples. _Left:_ Full validation results. _Right:_ Zoomed result for increased detail below optimal network SNRs of 50.]
) <overlapnet_classification_scores>

We also plot several pseudo-efficiency curves for easier comparison to other results in @overlapnet_efficiency_plots. Since there are now two network SNR values for each pair example and one network SNR value for each single example, we present the result in five distinct efficiency curves, four curves for the pair examples, and one curve for the single example. The four curves for the pair examples are generated by sorting the dataset by the maximum network SNR of the pairs, minimum network SNR of the pairs, and the SNR of signals A and B, then by generating a rolling average of the model scores. The single signal SNR is generated by sorting by the lone SNR value and calculating a rolling average. Note that unlike previous efficiency curves, which displayed the percentage of results that were above a certain FAR-calibrated model threshold, these curves just plot the rolling average model predictions, which are correct at a score of one for the four pair curves, and correct at zero for the one single signal curve.

The minimum SNR signal curve reaches an average model score near one at a minimum network SNR of around 37, which is quite high. However, it still achieves relatively good scores above an optimal SNR of 16. The reason this curve appears with this different shape to the detection efficiency curves is presumably because there are factors other than the measured SNR that are relevant to the model performance. In all the pair examples, the other non-ranking SNR value will have a large effect on the model's ability to distinguish between the two cases, along with the merger time separation, and the parameter space difference between the two injections. Since the minimum network SNR is the only curve that reaches scores near one of the four curves for pair examples, we can infer that this is the bottleneck for detection ability. The other curves never reach one no matter how high the network SNR suggesting that a percentage of the other SNRs in the pairs are low and decrease detection ability.

The maximum SNR line, as expected shows, considerably lower performance at lower SNRs. In each of the examples on this line, the lower SNR signal, which we know is the limiting factor, is equal to or lower than the SNR metric. This also explains why the line starts at a higher SNR --- because the line is a rolling average of the examples sorted by SNR metric, the first average SNR value for the maximum signals will naturally be higher than the first average of the minimum signals. Interestingly, the maximum SNR curve reaches approximate parity with the lines plotting effficiencies when sorted by the SNR of signals A and B. This is presumably because there is a large range of SNRs less than the maximum, so the lower SNR is likely to still be detectable, with the difference between this maximum efficiency and one due to the undetectable percentage of SNRs under this maximum value.

Curves were also plotted by sorting the pair validation results when sorted by signal A SNR and signal B SNR. These were plotted to see if the model had any bias between the signals. Signal B is defined as the signal that arrives first in the detection, so it might in theory be possible for the classifier to develop bias toward one signal or another. However, the results show very little difference between the two curves, suggesting that the model does not have a preference between which signal has the higher SNR. Again the model does not reach an average model score of one, but this is because the other SNR in the pair is unconstrained, so a certain percentage of examples at each average calculation will have a pairing with an SNR that is undetectable.

Finally, an efficiency curve was plotted for the single signal examples. Only one curve was plotted for this example, as a single network SNR value can capture the entire SNR information content of the example. Note that in this case, a model prediction of zero rather than one is correct, so lower scores indicate a more accurate prediction. We see that the model performs best at SNR values less than 20, before plateauing and slowly increasing beyond that. This shape is created because the training data contained many pair examples with one low SNR value which would look very similar to single signal examples, creating confusion in the training process and leading the model to predict single signals with some uncertainty. The higher performance at low SNRs is presumably due to the excess power bias because there are considerably more single signal examples on the lower excess power end, the model can more confidently predict a single signal if the total excess power is low. For the same reason, model performance degrades at higher single signal SNRs as it is more likely there is higher excess power in double signal examples, although in double signal examples with high excess power the double morphologies are more likely to be visible, so this bias is considerably less than the low power bias demonstrated. 

#figure(
    image("overlapnet_efficiencies_2.png", width: 80%),
    caption: [Overlapnet pair efficiency plots created from the combined overlapnet validation data pool using rolling averages with a window of 1000 and plotting every 100#super("th") average value. This plot gives an approximate comparison to the efficiency plots generated in the detection cases; although generation was done with a rolling average over a pool of logarithmically distributed SNRs rather than with pools of discrete SNRs at specific test SNR values that have been used previously in the detection cases. Also note that this plots the model output score, rather than the percentage of cases which fall above a calibrated SNR threshold. These efficiency plots show the relationship between the SNR values of one of the signals and the model prediction. One of the five lines gives the rolling average model score when the validation results pool is sorted by minimum SNR value. This is perhaps the most useful of the four lines as it is the bottleneck in classification ability. It reaches a classification score of 1.0 at a minimum optimal network SNR of around 37. It remains above 0.9 for SNRs above 19 and increases slowly until 37. This separates it from the detection case and is presumably because there are extra factors not accounted for on this plot, primarily the SNR of the second signal, but also things like the parameter space difference of the two waveforms and the merger time separation of the two waveforms, which could both add increased difficulty without being visible on the efficiency plot. Two of the lines plot the rolling average model score when plotted with the SNR of one of the two signals, signal A and signal B. Signal B always arrives before signal A. The similarity between these lines shows that it is unlikely there is any bias between whether signal A has the lower SNR or signal B. The maximum scores achieved by these lines are less than the minimum, as there are always low SNR signals in the average used to calculate this. The last of the four pair example lines shows the moving average when the validation pool is sorted by the maximum SNR of the two injected signals. This is the lowest still, as it is more likely that the uncounted-for signals have low SNR. Lastly, we show the single signal SNR scores. Unlike the other signals, a lower score is better in this case, as a model prediction of zero indicates the lack of a second signal. We see that at low SNRs this score is lowest; this is expected as there are considerably more low SNR single signals in the dataset than pairs of signals, and this supports our hypothesis that the network is using excess power as a differentiation method. Above an optimal network SNR of 18 the classification score plateaus at an average of 0.2, as stated previously it is believed this is induced through confusing examples in the training dataset where it is almost impossible for the network to determine between a pair of signals where one signal has a low SNR and a single signal, teaching the network to guess with some uncertainty in all apparent single signal cases. We also see a slight decrease in prediction accuracy as SNR increases, again this probably results from the excess power bias. From this plot we can conclude that as expected the lowest SNR signal in the pair is the largest factor in model efficiency, but that other factors are probably also relevant.]
) <overlapnet_efficiency_plots>

We create additional plots to explore how classification performance varies with other areas of parameter space. First, we examine how the time difference between the merger arrival of signal B and signal A at the earth center (assumed to be very close to the arrival at any of the detectors. Only in a very small number of cases will the order of arrival at the Earth's center be different from the arrival time at any detector when using this range of time separations), which affects the classification performance. @overlapnet_classification_separation shows that there is little correlation between model performance and the arrival time difference, except when the time separation is very small. There appears to be some degradation of perfomance below #box("0.8" + h(1.5pt) + "s"), but this only becomes very significant below #box("0.2" + h(1.5pt) + "s"). Since the average model performance values are calculated using a rolling average, when we examine the distribution of individual example performance by eye this first bin also seems to be heavily weighted by examples whose separation is very close to zero. As the time separation moves toward zero, the model has less opportunity to use distinct merger peaks to aid in its classification and must begin to rely on morphology alone. Since the model maintains performance, though at a reduced efficiency, at separations down to zero seconds, we can determine that the model can use morphologies as well as distinct peaks, in order to distinguish between the two cases. Further analysis of a validation set consisting only of zero separated signals would be useful to examine this further. However, since signals arriving with such small separations are very unlikely even when detection rates are massively increased, this is not considered a priority.

#figure(
    image("overlapnet_classification_separation.png", width: 70%),
    caption: [Overlapnet classification results plotted against the time elapsed in seconds between the arrival of the merger of signal B and signal A. The coloured circles represent individual validation classification results colour-coded for visual clarity. The red line is the moving average model prediction error at the stated time separation with a window of 500 validation examples. Only pairs are plotted, as single examples have no time separation. We see that for time differences above #box("0.8" + h(1.5pt) + "s") the separation has little effect on the average prediction error. Between #box("0.2" + h(1.5pt) + "s")  and #box("0.8" + h(1.5pt) + "s") there is a slight but notable increase in error, and below a merger time difference of #box("0.2" + h(1.5pt) + "s") there is a more notable uptick in error. It appears that this uptick at lower time separations is mostly caused by signals that have very low separation ($<$ #box("0.1" + h(1.5pt) + "s")) --- this seems to be the only significant predictor of model performance, other than this, and the small decrease in performance below #box("0.8" + h(1.5pt) + "s") the classifier seems to work with equal efficiency across time separations. This is perhaps less of a correlation than might be expected, but it demonstrates that only very close signals are problematic if at detectable SNRs. This is a good sign for the chances of developing a useful classifier. ]
) <overlapnet_classification_separation>

For our final analysis of the classification results, we explore the parameter space of the waveform by examining model performance at different values of chirp mass and mass ratio; see @overlapnet_classification_mass_parameters. These plots are less illuminatory, the only visible correlation exists between a lower chirp mass in one or both signals and poor performance. This is likely caused because sources will have a lower $h_"rss"$ and therefore SNR (assuming identical detector noise conditions, sky localization, and polarization) if they are at the same luminosity distance as another signal with a higher chirp mass. This is also corroborated by the decrease in single signal classification performance at higher chirp masses, which is seen with higher SNRs. However, if luminosity distance is variable, which it is in the dataset, chirp mass alone does not correspond directly to SNR. Thus we don't see as strong of a correlation as we see in @overlapnet_classification_scores. 

Since there is no visible correlation along the line where the two parameters are equal to each other, we can conclude that both waveforms having similar mass parameters have relatively little effect on the ability of the model to correctly classify the signal. This is slightly surprising, as we would expect signals with similar frequency contents to be more difficult for the model to separate. However, in most cases within the validation dataset, the arrival time separation was large enough that the model could use the distinct merger peaks as evidence for mergers rather than relying on the morphologies alone. Which may explain this seeming lack of dependence on waveform parameters.

#figure(
    grid(
        columns: 2,
        rows:    1,
        gutter: 1em,
        [ #image("overlapnet_classification_chirp_mass.png", width: 100%) ],
        [ #image("overlapnet_classification_mass_ratio.png", width: 100%) ]
    ),
    caption: [Overlapnet classification results compared with the mass parameters of the constituent waveforms. _Left:_ Overlapnet classification scores plotted with source chirp masses for signal A and signal B. There appears to be some correlation between lower chirp masses and poor model performance, however, because there are highly performing examples even in cases where both chirp masses are low we can conclude that this does not explain the entire picture. It is hypothesized, that this correlation is primarily caused by the fact that lower chirp masses are more likely to produce a low SNR signal. If two sources were at the same luminosity distance but one had a higher chirp mass, the higher chirp mass would have a louder SNR (assuming identical detector noise conditions, sky localization, and signal polarisation). This hypothesis is supported by the lower model performance of single signals at higher chirp masses, as we have seen that single signal classification is slightly worse at higher SNRs. _Right:_ Overlapnet classification scores plotted with source mass ratio for signal A and signal B. This plot shows that there is very little, if any correlation between the mass ratio of the two signals, and model performance. This continues to show that signal morphology does not make a decisive difference in classification ability, which is primarily determined by the minimum SNR of a signal in the pairs, and secondarily weakened if the signals have a very small time separation. ]
) <overlapnet_classification_mass_parameters>

We conclude that Overlapnet is capable enough to differentiate between single and overlapping signals in the majority of cases, and with some adjustment to the training dataset, performance could probably be improved by removing the ambiguity generated by impossible-to-distinguish examples.

=== Regression

Following the relative success of Overlapnet in differentiating between examples with one CBC signal present and two overlapping CBC signals present, we attempted to use the same model, with an adjusted activation function on the last layer, (linear rather than softmax), to attempt a regression problem on pairs of overlapping signals, in an attempt to extract useful information that could be used in a parameter estimation pipeline. The most useful parameters that can be extracted from an overlapping signal, are the merger times of the two signals A and B. Using the same training dataset and procedure, we changed only the activation function applied to the model output to allow for regression rather than classification, the model loss function used, again to allow for regression, and the labels that the model was trained to output. Rather than output a score between zero and one, we trained the model to output a merger time for signal A, and a merger time for signal B. 

Due to an error in the data labelling procedure, which was not spotted until later experiments. Regression test results were extremely poor. Initially, it was thought that this was down to using an insufficient model for the task, therefore a much more complex and experimental network was constructed, utilizing much of the insight gained from previous experiments, as well as denoising autoencoder heads, and cross-attention layers between detectors. These concepts are explained in @additional-elements. We name this more complex network structure CrossWave. It should be noted that it is not proven that such a complex network is necessary for successfully overlapping parameter estimation of overlapping CBC signal with machine learning.

== Aditional Structural Elements <additional-elements>
=== Cross-Attention <cross-attention-sec>

The multi-head attention layers we have explored thus far in @skywarp-sec have all consisted of self-attention heads, which is the most natural application of attention @attention_is_all_you_need @attention_review. However, attention layers can also be used to compare two different sequences; this is a principle component of the archetypical transformer design @attention_is_all_you_need @transformer_review, wherein cross-attention layers compare the model-predicted vectors to the ground truth vectors of the training data. See @transformer-sec. Since we were not concerned with next-token prediction in @skywarp-sec we opted not to use cross-attention layers and instead focus entirely on self-attention. 

However, there is a scenario in gravitational-wave data science for which a natural application of cross attention can be applied --- between the detector outputs of the multiple interferometers of a gravitational-wave detection network. There are two ways in which we could deal with this scenario, we could apply the appropriate temporal encoding and add an encoding element informing the model which detector each sequence element originated from, or we could simply use cross attention between the multiple detectors. See @cross-attention-digaram> for an illustration of the cross-attention mechanism.

In cross-attention, query, key, and value vectors are still generated, but for two sequences instead of one @attention_is_all_you_need @transformer_review. The query vectors from one sequence are then compared with the key vectors of the other sequence, and the value vectors of that sequence are summed together similarly as in self-attention. What this does is it allows the attention layer to accumulate information from the other sequence that is relevant to vectors in the first sequence. Because the choice of which sequence will provide the query and which the key and value, matters, cross-attention is not commutative. After calculating the cross-attention between detectors, you can then add this result to the self-attention result, allowing you to accumulate relevant information both from other temporal locations in one detector and from information provided by other detectors. 

Overlapnet used both LIGO detectors as input since there was no need to try and optimise for low FAR. When attempting to improve on this network operation with attention layers, it is a natural choice to apply cross-attention.

#figure(
    image("cross_attention.png", width: 100%),
    caption: [Illustration of the action of a single cross-attention head. In contrast to a self-attention head, a cross-attention head takes two sequences as input: a querier sequence, and a queried sequence. The queryier sequence is converted into query vectors with a learned weights matrix, and the queried sequence is converted into key and value vectors. The rest of the attention mechanism functions identically to self-attention but uses query, key, and value vectors that originate from different sources. For more details on the self-attention mechanism see the description in @sec-attention.]
) <cross-attention-digaram>

=== Autoencoders and Denoising <autoencoder-sec>

Autoencoders are a family of artificial neural network architectures @autoencoder_ref. They can utilise many different layer types including pure dense layers, convolutional layers, and attention-layers, but they are defined fundamentally by their inputs and outputs and the shape of the data as it moves through the network. The vanilla autoencoder can be described as a form of unsupervised learning since the model input is the same as the model output, and therefore, although it has in some sense a model label --- its input, the data does not need to be labeled, as it is its own label.

A vanilla autoencoder attempts to compress the information content of its input into a latent vector that is typically significantly smaller than the input vector, then regenerate the original input vector from that reduced latent vector with as little loss as possible @autoencoder_ref. This has useful applications as a compression algorithm, but also sometimes in encryption and many other applications. Having access to a lower dimensional latent space that can represent elements of a unique distribution has many uses in generative models and classifiers. Many different subtypes of autoencoder try to regularise the latent space into a more useful format, the most common of which is the Variational AutoEncoder (VAE) @variational_autoencoder. 

Autoencoders can also be used for anomaly rejection @autoencoder_ref which has application in gravitational-wave analysis in both glitch @autoencoder_glitch_rejection and burst detection @source_agnostic_lstm. Because an autoendoer is trained to reconstruct data from a given distribution, if it is fed a result that lies outside that distribution this will likely result in a high reconstruction loss. The value of this loss then can be used to determine if the autoencoder has encountered something from outside its training distribution. In the case of gravitational-wave glitches, we can train a model on known glitch types or a single known glitch type. We can then reject glitches that the autoencoder can successfully reconstruct as specimens of known detector glitches. For anomaly detection, we can instead train the model to reconstruct a standard interferometer background, if the autoencoder fails to reconstruct a section of the background well, it could be an indication of the presence of an anomaly, which in some cases could be a gravitational wave burst. When detected in conjunction with coherent anomalies from multiple detectors, this could lead to a confirmed burst detection once glitches have been ruled out.

An autoencoder has three parts, an encoder, a decoder, and a latent vector @autoencoder_ref. See @autoencoder-diagrams. The encoder attempts to reduce the input vector into a smaller latent space vector. Performing a kind of dimensional reduction which hopefully preserves most of the information content of the input vector by representing it in a more efficient data format. In most distributions that are interesting, there is a significant structure that can be learned and used to compress the data. Similar to compression algorithms, if the input data is random and has no structure, there will not be a way to represent that data in a much more efficient way. The encoder commonly has an identical structure to the convolutional layers in a CNN. A function to compress the input data down into smaller feature maps is identical to what we require from our encoder. Encoders can also be built with dense or attention layers, and share most of the benefits and drawbacks of these previously discussed. The decoder is similar but acts in reverse to reconstruct the original input data from the reduced representation generated by the encoder. Often the decoder is a mirror image of the enoder and uses inverted layers such as transpose convolutional layers.

As well as acting as unsupervised models, it is possible to use pseudo-autoencoders which have the same structure as autoencoders but are not autoencoders in the truest definition, to produce an output that is not the same as its input, but instead an altered version of the input. This can be used to transform the input in some way, for example adding a filter to an image or audio input, or it can be used to try and remove noise from the original image. This latter type is known as a *denoising autoencoder* @denoising_autoencoder, and it is what we will be using as part of our expanded CrossWave architecture. Denoising autoencoders are no longer considered unsupervised models, as the labels must be denoised versions of the input vectors. During training, the denoising autoencoder learns to extract important features from the input image but ignores the noise, as it is not present in the output label and would be unnecessary information to propagate through the model. There have been some attempts to apply denoising autoencoders to gravitational-wave data in order to remove interferometer noise and reveal hidden gravitational-wave signals.

#figure(
    grid(
        columns: 1,
        rows:    1,
        gutter: 1em,
        [ #image("dense_autoencoder.png", width: 80%) ],
        [ #image("convolutional_autoencoder.png", width: 100%) ]
    ),
    caption: [Illustration of two trivial autoencoder architectures, one using only dense layers, the other using convolutional layers. Both networks have very few neurons and would likely not see use in any real practical application but are presented for illustration only. Autoencoders consist of an encoder that performs dimensional reduction on an input vector to reduce its dimensionality to a smaller latent space and produce a latent vector, this latent vector is then processed by the decoder which attempts to perform the inverse operation and reconstruct the original input image, or a slightly altered version of the input, for example a denoised version of the original input. Often the decoder is simply an inversed version of the encoder, which introduces the concept of transposed convolutional layers which perform the inverted operation of convolutional layers. _Upper:_ Illustrative dense layer autoencoder with a two-layer encoder and a two-layer decoder. The latent space of this autoencoder has two dimensions meaning the dimensionality of the input vector has been reduced from five down to two _Lower:_ Illustrative convolutional autoencoder with a two-layer encoder consisting of convolutional layers and a two-layer decoder consisting of transposed convolutional layers. The latent vector of this autoencoder has four elements, which means there has only been a reduction of one element between the input vector and the latent space.]
) <autoencoder-diagrams>

== CrossWave Architecture <crosswave-models>

The CrossWave architecture is the most ambitious model architecture presented in this thesis. It attempts to combine many intuitions gained throughout the research, with contemporary network features that are known to work well in similar domains. We utilize several new conceptual elements: denoising autoencoder heads, and cross-attention layers, which are described in more detail in @additional-elements.

The Crosswave architecture has a dual branch structure, which, rather than immediately combining both detectors into one multi-dimensional input stream, deals with input streams from both detectors in separate network branches in the first stage of the model. Each detector input is first fed into independent denoising autoencoder heads, with the idea that each autoencoder can learn to deal with the specificities of that particular detector noise, extract only the important signal features, and filter out detector glitches. These could first be trained independently to denoise signals before being used in the larger regression model, which was the original intention, however, due to time constraints, these were trained in unison with the greater model. The architecture of the autoencoder encoder is based on the model described in Gabbard _et al._ @gabbard_messenger_cnn, with the decoder consisting of the inverse of that architecture, using transpose convolutions in place of convolutions, and upscaling in place of pooling layers. This model was chosen as it is known to be able to effectively extract the features of a gravitational-wave signal.

In order to facilitate an effective comparison between the two detectors inside the cross-attention layers, it is desirable to have the features from both models mapped inside a shared latent space. For this reason, after the denoising heads, each branch was fed through an identical set of feature-extracting convolutional layers with shared weights. It is hoped that these shared weights will force the denoising heads to output vectors of the same form, which can then be processed by the shared feature extractor into the same latent space to allow for easier comparison by the cross-attention layers. In this manner, it is hoped that the independent denoising heads can deal with and remove differences in detector noise and configuration, whilst the shared feature-extracting layers can learn to recognize the features that are shared across detectors due to the detector-agnostic intrinsic properties of the waveforms proper. The feature-extracting layers are identical to the encoding layers inside the denoising heads, taken from Gabbard _et al._ @gabbard_messenger_cnn.

Two models were trialed, which are illustrated in @crosswave-small-diagram, and @crosswave-large-diagram. The small model only utilizes a single cross-attention layer, which combines the output of multiple self-attention blocks applied independently to each detector output. The larger model computes both self-attention and cross-attention inside each attention block, summing the results of both layers in order to combine both global contexts from other parts of the input vector, and the other detector into a single sequence for each detector. These results are finally concatenated after repetition and a final classification is performed with two dense layers. The larger model was found to have increased performance over the smaller one, and so was used for further experimentation. 

#set page(
  flipped: true
)
#set align(center)
#figure(
    image("crosswave_small_diagram_expanded.png",  width: 85%),
    caption: [Diagram of the network structure of the smaller of the two trialled CrossWave models. Both the CrossWave models have a novel structure with denoising heads, and feature extraction layers adapted from Gabbard _et al._ @gabbard_messenger_cnn, as well as utilization of cross-attention layers. The denoising heads are composed of an autoencoder structure, with one for each input detector. In this case, we have used simulated inputs from the LIGO Hanford and LIGO Livingston detectors so there are two autoencoding heads. Each autoencoder has independently trained weights. It is envisioned that during network training these will adapt individually to the peculiarities of the noise in its given detector, and, due to the shared weights utilized by the feature extractor, learn to output a standardized denoised version of the input from each detector, although it is expected this will not be a perfect correlated to a denoised detector stream since the autoencoders were not independently pre-trained before training of the larger model. After the autoencoding heads, several feature-extracting layers also adapted from Gabbard _et al._ @gabbard_messenger_cnn are used to embed the autoencoder outputs into two sequences that can be ingested by the attention layers. It is imagined, that because these feature-extracting layers share weights between detectors, they will map the output of the denoising layers into a shared latent space that can be interpreted similarly by the attention layers, and is therefore useful for cross-attention between detectors. 
    
    The core of the small CrossWave model utilizes a repeating block of self-attention layers applied repeatedly to each detector data stream, much like in the Skywarp transformer. These blocks are repeated three times. This repeating self-attention layer should help the model understand the global context of the data within each detector. After completion, these data streams are combined in a cross-attention block, and then processed by two dense layers to give the final regression output scores. This model was trialled and was somewhat performant, but the application of the cross-attention in this method was causing a lot of information to be lost, so the model was abandoned in favour of the larger variant shown in @crosswave-large-diagram.],
)  <crosswave-small-diagram>
#figure(
    caption: [Diagram of the network structure of the larger of the two trialled Crosswave models. Both the CrossWave models have a novel structure with denoising heads, and feature extraction layers adapted from Gabbard _et al._ @gabbard_messenger_cnn, as well as utilization of cross-attention layers. The denoising heads are composed of an autoencoder structure, with one for each input detector. In this case, we have used simulated inputs from the LIGO Hanford and LIGO Livingston detectors so there are two autoencoding heads. Each autoencoder has independently trained weights. It is envisioned that during network training these will adapt individually to the peculiarities of the noise in its given detector, and, due to the shared weights utilized by the feature extractor, learn to output a standardized denoised version of the input from each detector, although it is expected this will not be a perfect correlated to a denoised detector stream since the autoencoders were not independently pre-trained before training of the larger model. After the autoencoding heads, several feature-extracting layers also adapted from Gabbard _et al._ @gabbard_messenger_cnn are used to embed the autoencoder outputs into two sequences that can be ingested by the attention layers. It is imagined, that because these feature-extracting layers share weights between detectors, they will map the output of the denoising layers into a shared latent space that can be interpreted similarly by the attention layers, and is therefore useful for cross-attention between detectors.
    
    The core of the larger CrossWave block contains both self-attention blocks and cross-attention blocks in each iteration, this means that the model can compare data streams from both detectors multiple times, each time adding extra relevant information from the other detector into that detector's branch. Also, since the cross-attention is being performed in both directions, no information is lost as it was in the small model. Again, these blocks are repeated three times. After the repeating blocks, rather than using a cross-attention block to combine the branches, the outputs from each branch were concatenated before being fed into the dense tail, which then produced the final regression outputs.
    ],
    image("crosswave_large_diagram_expanded.png",  width: 96%)
) <crosswave-large-diagram>
#set align(left)

#set page(
  flipped: false
)

== CrossWave Dataset

We used the same independently produced dataset to provide the regression labels for the CrossWave regression models, described in @crosswave-models. Fortunately, the dataset was produced with comprehensive labels of relevant parameters, including luminosity distance, signal arrival time, companion masses, and spin parameters for each of the two waveforms in the pairs.

As opposed to binary classification, there is slightly more to consider when designing the form of the output labels. Initially, we must consider if we want to perform a classification at the same time as the regression of parameters. When training a model, the model can learn information from the input vector, but also from the labels. At the onset of experimentation with CrossWave it was thought that perhaps allowing the dataset more information about the data, in the form of more detailed labels of parameters, might aid in the classification process. This has several issues, primarily, if we wish to perform classification at the same time as a regression, we must include both pairs and singles in the training dataset. Therefore we must decide what to do with the parameter regression labels for signal B, in the case where there is no signal B. The natural decision might be to set these labels to zero, however, this might cause significant problems in the network. Whenever a training example without signal B is presented, the gradient descent algorithm will attempt to move the network toward producing zero for all of signal B's parameters, but zero is not an accurate representation of an undetectable signal for many of the parameters. For example, it might be more natural to put the luminosity distance very high for an invisible signal, but for other labels, it is not quite as easy to select a null value that won't disrupt the parameter estimation in some way. Joint classification regression trials proved ineffective across several different selections of null parameters for signal B, causing parameter estimation and classification confusion in the network. A method to gate outputs by multiplication with the classification output was also trialled, but this too proved ineffective.

Due to the failure of early experiments to perform parameter estimation with datasets comprised of both pairs of signals and single singles, the decision was made to focus on the more specific task of extracting parameters from pairs of signals, without the capability to perform parameter estimation on single signals, or classify between the two cases. Therefore all single signals were removed from the validation and training datasets, shrinking the training dataset size to $2 times 10^5$ examples, and the validation dataset to $2 times 10^4$ examples.

The second consideration is which parameters to attempt to extract. The inclusion of additional parameters did not appear to reduce the ability of the model to correctly extract other parameters, even if those additional parameters were difficult or impossible to extract. Neither did the opposite appear to be the case; the inclusion of additional labels did not appear to improve the ability of the model to classify other parameters. The following 24 parameters were selected for extraction by the models: ["*Signal A Geocentric Arrival Time (s)*", "*Signal B Geocentric Arrival Time (s)*", "*Signal A H1 Arrival Time (s)*", "*Signal B H1 Arrival Time (s)*", "*Signal A L1 Arrival Time (s)*", "*Signal B L1 Arrival Time (s)*", "*Signal A Companion Mass 1 ($M_dot.circle$)*", "*Signal B Companion Mass 1 ($M_dot.circle$)*", "*Signal A Companion Mass 2 ($M_dot.circle$)*", "*Signal B Companion Mass 2 ($M_dot.circle$)*", "*Signal A Luminosity Distance (MPc)*", "*Signal B Luminosity Distance (MPc)*", "*Signal A Dimensionless Spin Component Xß Companion 1*", "*Signal B Dimensionless Spin Component X Companion 1*", "*Signal A Dimensionless Spin Component Y Companion 1*", "*Signal B Dimensionless Spin Component Y Companion 1*", "*Signal A Dimensionless Spin Component Z Companion 1*", "*Signal B Dimensionless Spin Component Z Companion 1*", "*Signal A Dimensionless Spin Component X Companion 2*", "*Signal B Dimensionless Spin Component X Companion 2*", "*Signal A Dimensionless Spin Component Y Companion 2*", "*Signal B Dimensionless Spin Component Y Companion 2*", "*Signal A Dimensionless Spin Component Z Companion 2*", "*Signal B Dimensionless Spin Component Z Companion 2s*"].

Finally, we can consider if and how we want to normalize the values of the labels. Out of the selected parameters, many are in a different range, with the Spin Components between -1 and 1, the masses between 10 and 70, and the luminosity distance between 500 and 700. This is not necessarily a problem, and the model would still be able to produce these outputs assuming we use the right activation functions. However, this could cause the model to take a longer time to converge on correct solutions, as it would consider some parameters, with larger values, to be much more important the others. Ideally, we want the gradient descent mechanism to treat all parameters approximately equally. For that reason, all label values were normalized between 0 and 1. The only exception to this was the merger times for signal B, as these were incorrectly normalized leading to some ground truth values of less than one, which the model was unable to predict due to ReLU activation functions used on the output layer, which limit neuron outputs to values of zero or greater.

== CrossWave Results

=== Merger Time Parameter Estimation Results <crosswave-merger-time-sec>

Although a single model to predict all aforementioned parameters was produced, we shall focus on examining the merger time predictions before we look at the other results. These are the more important outputs of the model, as they fulfil the initial goal of providing more information for use in more established parameter estimation methods. The model was trained to output merger arrival time predictions for the LIGO Hanford Detector, the LIGO Livingston Detector, and the Earth's centre. Results are consistent between detectors, so for the first set of comparisons we compare only the predictions for the LIGO Hanford arrival time to the ground truth LIGO Hanford arrival time.

First, can examine the arrival time predictions compared to the individual signal SNRs --- unlike in the classification case where only a single result is output, (albeit from two output neurons normalized by a SoftMax layer), the model is now tasked to output two regression values (along with the other parameters): a merger time for signal A and a merger time for signal B. Therefore, we have generated two plots, one for each signal; see @crosswave_regression_snr.

Examining the results, we observe that there are a few outliers with high errors above #box("0.25" + h(1.5pt) + "s"), but the majority of merger times were predicted with errors under this margin. On average, errors were notably worse for signal B than for signal A, but their magnitudes do not seem to be correlated to the signals' network SNR so it is unclear whether this increased error in signal B lies in the method or is a result of systematic training degradation introduced by the normalisation error. Another possibility is that this increased error arises from the asymmetry caused because signal B always arrives in the detector before signal A (except in very rare edge cases previously discussed); because of this, the entirety of signal B is always contaminated with the inspiral of signal A, whereas signal often has at least some clear signal after signal B has merged and the ringdown has petered out. Some errors could originate from the misidentification of signals A and B. However, if this were to be the case, we would expect to see this error correlated with SNR, but we do not, at least on first inspection.

#figure(
    grid(
        columns: 2,
        rows:    2,
        gutter: 1em,
        [ #image("regression_snr_A.png", width: 100%) ],
        [ #image("regression_snr_B.png", width: 100%) ],
        [ #image("regression_snr_A_less.png", width: 100%) ],
        [ #image("regression_snr_B_less.png", width: 100%) ]
    ),
    caption: [CrossWave merger time prediction error of Signal A, _upper left_, and Signal B, _upper right_. Compared to the classification results, the merger time errors look more consistent. This is primarily because the model output is not restricted between one and zero like it is in classification, so a few outliers with very high errors saturate the colour map. Given this, we have also plotted the same results with all examples that have errors greater than #box("0.25" + h(1.5pt) + "s") removed, for a more granular view of the bulk of the regression prediction errors. These are the lower two plots. In these focused plots, we can see that a significant number of results have a regression error of less than #box("0.1" + h(1.5pt) + "s"), which could be helpful to aid a secondary parameter estimation method. On these lower plots, there is also a notable difference between the average error on signal A merger time predictions, and the average error on signal B merger time predictions, with a higher average error on signal B. It is unclear exactly why this is the case, but we speculate that this is because signal B arrives first in the detector, meaning that the inspiral of signal A can interfere significantly with signal B, whereas the opposite is only the case when the merger separation is very small. It is also possible that sometimes, signal A can be misclassified as signal B. We would expect this latter confusion to have some correlation to SNR, but this does not seem to be the case. It could also be due to the aforementioned normalisation error reducing model training efficacy for signal B merger time predictions. Interestingly, the relationship between signal SNR and regression error appears low. This suggests that the substantive cause of regression error lies elsewhere, we plot additional comparisons to further investigate.]
) <crosswave_regression_snr>

Next, we can plot the rolling average prediction error as it changes with SNR; see @crosswave_regression_efficiencies. The picture is mostly as anticipated, with the determining factor for high merger time error at low SNRs ($<20$) being the SNR of the signal whose merger time is being predicted. This suggests that the expected SNR relationship is present in @crosswave_regression_snr, but is hidden under variance created by other factors. At optimal Network SNRs between 20 and 60, the results are roughly consistent between signal A prediction error and signal B prediction error. With a high SNR in the opposing signal leading to higher error. Above an optimal network SNR of 60, the pictures change, with the signal A prediction error average roughly equal whether ranked by signal A SNR, signal B SNR, or maximum SNR. This suggests estimation of signal A merger time is independent of signal B. For signal B however, high signal A SNR increases error on signal B, suggesting that as theorized, the inspiral of signal A can interfere with the parameter estimation of signal B.

#figure(
    grid(
        columns: 1,
        rows:    2,
        gutter: 1em,
        [ #image("regression_efficiency_A.png", width: 80%) ],
        [ #image("regression_efficiency_B.png", width: 80%) ]
    ),
    caption: [CrossWave rolling average merger time prediction error plotted when ranked by different SNR combinations. Since the model now has two outputs, one for each merger time in the input example, a plot was generated for each merger time prediction. A plot showing signal A merger time prediction on the left, and a plot showing signal B merger time prediction on the right. At low SNR, the error is dominated by the SNR in the given signal, which is anticipated --- a low SNR in a given signal would, evidently, make it difficult for the model to detect, and hence, estimate the merger time, of that signal. We can also see the notable difference in average prediction error between the upper signal A plot and the lower signal B plot. Interestingly, we see that the error on the signal B merger time increases when the SNR of signal A is higher. This seems to be the case regardless of the SNR of signal B. Since signal B always arrives first in the detector, this could be because a loud signal A inspiral obfuscates the presence of signal B, rendering the signal B merger time hard to identify.]
) <crosswave_regression_efficiencies>

As was the case with classification, we might expect the difference in merger times to affect merger time estimation ability. Thus we have created similar plots to determine the effect of the difference in merger arrival time between the two signals on model prediction ability; see @crosswave_merger_times. In both cases, there is a sharp peak in the magnitude of the prediction error when the merger time separation nears zero. As would be expected, if it gets difficult to determine if there are one or two signals at a particular spot, with perhaps another smaller SNR signal hiding elsewhere, the model becomes confused when trying to predict the merger time. In the case of signal A, defined as the second signal to arrive in the detector, when incorrect, the model tends to predict the signal will arrive later than it does, and vice versa in the case of signal B. We also note that in both cases, though more distinctly in the signal B case, a cluster of errors falls along the line where error equals the merger time separation, we can label these events as misidentifications, where signal A has been misidentified as signal B or vice versa. In both cases, there seems to be a very slight uptick in error at high separations, this could be due to a smaller number of examples present in these areas of parameter space, leading the model to think these are unlikely parameters. 

#figure(
    grid(
        columns: 2,
        rows:    1,
        gutter: 1em,
        [ #image("error_time_difference_signal_a.png", width: 100%) ],
        [ #image("error_time_difference_signal_b.png", width: 100%) ]
    ),
    caption: [CrossWave merger arrival time prediction errors compared with the time separation between signal A and signal B merger arrival times in the LIGO Hanford detector. _Left:_ Error on signal A merger time prediction compared with the time separation between the two mergers. _Right:_ Error on signal B merger time prediction compared with the time separation between the two mergers. The colour of the plotted examples depicts the absolute error between the model prediction and the ground truth value, and the red line shows the rolling average absolute prediction error. For both merger times, we can see a spike in erroneous merger time predictions when the time separation is near zero. This is similar behaviour to what is seen in the classification examples. It is also expected here, since if the mergers are hard to distinguish from each other it will be difficult to determine the specific merger times. An asymmetry arises in which way the model will incorrectly predict the merger, in signal A, defined as the second to arrive in the detector, the model often predicts the signal will arrive later than it does, and for signal B, the model often thinks it will arrive earlier than it does. Since B always arrives first, these are logical assumptions for the model to make in both cases. In both cases, we also see lines of erroneous predictions where the model error equals the time separation. These are believed to be cases where the model believes signal A to be signal B and vice versa. This line is more pronounced for signal B errors, suggesting that signal B's are more commonly mistaken for signal A's than the other way around.]
) <crosswave_merger_times>

We have decided not to plot the merger times against the mass parameters as we did in the classification case, as these did not seem to have much of an effect on classification ability.

Finally, we finish with a direct comparison of the model prediction to the ground truth value; see @crosswave_arrival_time_prediction_error. This is plotted to align with the further presentation of parameter estimation results in @crosswave-further-pe-sec. We have plotted the predicted merger time in the LIGO Hanford detector compared to the ground truth, as well as the predicted merger time in the LIGO Livingston detector compared to the ground truth. The results for Earth centre merger arrival times are omitted, as these were not considered particularly relevant to this analysis. A full table of parameter estimation results is given by @crosswave-regression-results.

#figure(
    grid(
        columns: 2,
        rows:    2,
        gutter: 1em,
        [ #image("h1_signal_a_arrival_time.png", width: 100%) ],
        [ #image("h1_signal_b_arrival_time_2.png", width: 100%) ],
        [ #image("l1_signal_a_arrival_time.png", width: 100%) ],
        [ #image("l1_signal_b_arrival_time_2.png", width: 100%) ],
    ),
    caption: [CrossWave signal merger time parameter estimation results. Each pair of plots shows the merger time estimate of signal A (_left_) and signal B (_right_). For each validation example, the ground truth value is represented on the x-axis, and the model prediction is on the y-axis. Each represents the signal merger time in seconds. The colour of each circle depicts the absolute difference between the ground truth value and the model prediction, which will be zero if the point falls on the line of $x = y$, which is also shown on the plot as a dashed grey line. Due to an error in label normalisation, some ground truth values for signal B were less than zero. Unfortunately, due to the choice of loss function used for the regression (ReLU), the model could not output predictions below zero, this meant that it was unable to predict these values correctly. This error may have interfered slightly with the rest of the training process, however other than its inability to classify these examples, there does not seem to be a significant reduction in the performance of classification of signal B merger times. Validation examples with round truth values below zero, and their associated predictions have been omitted from signal B plots for visual clarity. If training were to be repeated this error could be easily rectified, either by correcting the normalization or by altering the choice of activation function. _Upper Left:_ Predicted against actual signal A merger time in the simulated LIGO Hanford output. _Upper Right:_ Predicted against actual signal B merger time in the simulated LIGO Hanford output. _Lower Left:_ Predicted against actual signal A merger time in the simulated LIGO Livingston output. _Lower Right:_ Predicted against actual signal B merger time in the simulated LIGO Livingston output. ]
) <crosswave_arrival_time_prediction_error>

=== Other Parameter Estimation Results <crosswave-further-pe-sec>

As well as attempting to predict the merger arrival times for both signal A and signal B, the model was also tasked to output several other parameters. This was initially done to attempt to increase the model's knowledge about the task at hand but was found to have no significant positive or negative effect on the estimation of the merger time parameters. Thus, it was kept as a feature of the final model as a potential feature of interest for future development into a more advanced fully machine learning-based parameter estimation model for overlapping signals.

A full table of the results of the CrossWave parameter estimation model when run on the $2 times 10^5$ pair validation examples and compared to the ground truth labels can be seen in @crosswave-regression-results. For each parameter, an $R^2$ score is plotted as well as a Mean Absolute Error (MAE). The $R^2$ score or the "coefficient of determination" is a measure of the goodness of fit of a model. It provides an indication of how well the independent variables in a regression model explain the variability of the dependent variable. An $R^2$ score of one indicates a perfect predictor, whereas a $R^2$ score of zero indicates the model is doing no better than outputting the mean value, and a negative value indicates that the model is performing worse than outputting the mean and so possibly indicates an error in training. The MAE simply indicates the average magnitude of the difference between the model prediction and the ground truth value.

#figure(
  table(
    columns: (auto, auto, auto, auto, auto),
    inset: 10pt,
    align: horizon,
    [*Parameter*], [*$R^2$ score A*], [*Mean Absolute Error A*], [*$R^2$ score B*], [*Mean Absolute Error B*],  
    [H1 Time], [0.968], [0.100 s], [0.963], [0.0967 s],
    [L1 Time], [0.963], [0.0915 s], [0.963], [0.0965 s],
    [Geocent Time], [0.963], [0.0923 s], [0.963], [0.0974 s],
    [Luminosity Distance], [-0.834], [23.2 MPc], [-0.791], [22.7 MPc],
    [Mass 1], [0.613], [#box("7.95"  + h(1.5pt) + $M_dot.circle$)], [0.623], [ #box("7.79" + h(1.5pt) + $M_dot.circle$)],
    [Mass 2], [0.718 ], [#box("5.59" + h(1.5pt) + $M_dot.circle$)], [0.715], [#box("5.47" + h(1.5pt) + $M_dot.circle$)],
    [Spin 1x], [-0.00897], [0.162], [-0.0119], [0.165],
    [Spin 1y], [-0.0780], [0.174], [-0.0749], [0.178],
    [Spin 1z], [0.268], [0.273], [0.234], [0.280],
    [Spin 2x], [-0.0117], [0.161], [-0.0114], [0.163],
    [Spin 2y], [-0.0709], [0.179], [-0.0705], [0.179],
    [Spin 2z], [0.0699], [0.311], [0.0620], [0.316],
  ),
  caption: [Results of the CrossWave parameter estimation model. For each of the model's outputted parameters, a Mean Absolute Error (MAE) along with an $R^2$ score is presented. The MAE indicates the average magnitude of the errors between the model's predictions on the validation dataset and the corresponding ground truth values. It is a measure of average prediction accuracy, though it doesn't distinguish between overestimation and underestimation. The $R^2$ score quantifies how well the model's predictions explain the variance of the ground truth values in the validation dataset. An $R^2$ score of one signifies perfect prediction accuracy in the validation examples used. In contrast, a score of zero suggests the model's predictive capability is no better than simply using the mean value of the validation examples. Negative $R^2$ values indicate that the model performs worse than a model that would always predict the mean, possibly signalling errors in the training process or model selection. ]
) <crosswave-regression-results>

Out of the parameters that the model was trained to predict, the most accurate are the merger times, with $R^2$ scores between 0.96 and 0.97, and MAEs between #box("0.1" + h(1.5pt) + "s") and #box("0.09" + h(1.5pt) + "s"). As we have seen from previous analyses of these results, this average is driven up by outliers. There does not seem to be a particular detector that performs worse than any other, however, the Hanford signal A MAE is notably higher than the other predictions, whether this is driven up by statistical variance or some other factor is unknown. These results are discussed in more detail previously in @crosswave-merger-time-sec.

The parameters that the model was next most proficient at extracting, were the mass parameters of each of the two component masses in each of the two binaries, generating four mass values in total: signal A mass 1, signal B mass 1, signal a mass 2, and signal B mass 2. The $R^2$ scores are lower and the MAE values are higher for mass 1 than for mass 2. This is probably because during parameter generation mass 1 is always forced, by convention, to be the higher mass, meaning that, in general, mass 1 has a larger range of possible values than mass 2. Because the model can use its prediction of mass 1 to constrain mass 2, it can reduce its error. This constraint also allows for a better guess at the average mass for mass 2, since its values have a smaller distribution than mass 2 values. These results can be seen in @crosswave_mass_prediction_error. The error margins may be low enough for these results to have some limited usefulness, however, since they lack any form of uncertainty it is unclear exactly what that would be. Perhaps they could be used to inform the priors of another parameter estimation search.

#figure(
    grid(
        columns: 2,
        rows:    2,
        gutter: 1em,
        [ #image("signal_a_mass_1.png", width: 100%) ],
        [ #image("signal_b_mass_1.png", width: 100%) ],
        [ #image("signal_a_mass_2.png", width: 100%) ],
        [ #image("signal_b_mass_2.png", width: 100%) ],
    ),
    caption: [CrossWave companion mass parameter estimation results. Each pair of plots shows the companion mass estimates of signal A (_left_) and signal B (_right_). For each validation example, the ground truth value is represented on the x-axis, and the model prediction is on the y-axis. Each represents the companion mass in solar masses. The colour of each circle depicts the difference between the ground truth value and the model prediction, which will be zero if the point falls on the line of $x = y$, which is also shown on the plot as a dashed grey line. After the merger time predictions, the mass plots show the greatest promise, able to predict component masses with a moderate degree of accuracy. Without a comparison to another parameter estimation method, it is unclear exactly how much use these results can be. _Upper Left:_ Predicted against actual signal A companion 1 mass. _Upper Right:_ Predicted against actual signal B companion 1 mass. _Lower Left:_ Predicted against actual signal A companion 2 mass. _Lower Right:_ Predicted against actual signal B companion 2 mass.]
) <crosswave_mass_prediction_error>

Beyond the companion mass parameter estimation results, CrossWave's parameter estimation ability is very limited and is explored here as a demonstration of its lack of proficiency, rather than as a suggestion of usefulness. Of the attempted extraction of the companion spins, only the Z components show any signs of successful estimation. In particular, the Z component of the larger companion shows partial predictive power, although the usefulness of this extraction is questionable. These results are shown in @crosswave_spin_error.

#figure(
    grid(
        columns: 2,
        rows:    6,
        gutter: 1em,
        [ #image("signal_a_spin_1_x.png", width: 100%) ],
        [ #image("signal_b_spin_1_x.png", width: 100%) ],
        [ #image("signal_a_spin_1_y.png", width: 100%) ],
        [ #image("signal_b_spin_1_y.png", width: 100%) ],
        [ #image("signal_a_spin_1_z.png", width: 100%) ],
        [ #image("signal_b_spin_1_z.png", width: 100%) ],
        [ #image("signal_a_spin_2_x.png", width: 100%) ],
        [ #image("signal_b_spin_2_x.png", width: 100%) ],
        [ #image("signal_a_spin_2_y.png", width: 100%) ],
        [ #image("signal_b_spin_2_y.png", width: 100%) ],
        [ #image("signal_a_spin_2_z.png", width: 100%) ],
        [ #image("signal_b_spin_2_z.png", width: 100%) ],
    ),
    caption: [CrossWave regression results for the dimensionless spin components of the two companions in each binary merger, A and B. The left plots show the parameter extracted from merger A, whereas the right results show the same parameter extracted by CrossWave from merger B. The plots show the ground truth value of the dimensionless spin component plotted against the predicted value of the dimensionless spin component. The colour of each validation example indicates the difference between the ground truth and the predicted value, in this case, equivalent to the distance the point is from the line of $x = y$.  The results are in the following order from upper to lower: 

    + Mass 1 Spin Component X [_Left:_ Signal A, _Right:_ Signal B]
    + Mass 1 Spin Component Y [_Left:_ Signal A, _Right:_ Signal B]
    + Mass 1 Spin Component Z [_Left:_ Signal A, _Right:_ Signal B]
    + Mass 2 Spin Component X [_Left:_ Signal A, _Right:_ Signal B]
    + Mass 2 Spin Component Y [_Left:_ Signal A, _Right:_ Signal B]
    + Mass 2 Spin Component Z [_Left:_ Signal A, _Right:_ Signal B]
    
    There appears to be little difference in classification ability between signal A and signal B. The X and Y components show no classification ability, with the model finding an approximate output value to omit for all validation examples. It was known that extracting the spin parameters from the injected signals would be a challenging task, so this is anticipated. The model appears to show limited classification ability for the Z components, with the Z component for the more massive companion extracted with a stronger correlation than the lower mass companion, for which CrossWave shows only very slight predictive ability.
    ]
) <crosswave_spin_error>

Finally, CrossWave attempted to extract the luminosity distance of the source. This extraction has failed in an unusual manner. Not only did the model fail to correctly predict the luminosity distance to any degree, but also the results produced a $R^2$ score, which indicates the model did not even find a good mean value. Why this is the case is unknown. The luminosity distance could be difficult to extract due to its degeneracy with the source inclination angle, a parameter for which prediction was not attempted. The negative $R^2$ score could be due to another normalisation error which further investigation may reveal. These results can be seen in @crosswave_luminosity_distance_error.

#figure(
    grid(
        columns: 2,
        rows:    1,
        gutter: 1em,
        [ #image("signal_a_luminosity_distance.png", width: 100%) ],
        [ #image("signal_b_luminosity_distance.png", width: 100%) ],

    ),
    caption: [CrossWave model predicted luminosity distance vs ground truth luminosity distance of simulated BBH waveforms. _Left:_ Predicted signal A luminosity distance. _Right:_ Predicted signal B luminosity distance. The colour of each example point indicates the difference between the predicted and the ground truth value for that example. These plots indicate that there is almost no correlation between the predicted luminosity distance and the ground truth value. The model outputs a very similar value independent of luminosity distance, it is unclear whether this inability arises from a problem with model training and/or data processing, or whether luminosity distance is too difficult for the model to determine because of degeneracy with other parameters such as inclination. ]
) <crosswave_luminosity_distance_error>

== Discussion and Limitations

Overlapnet shows us that a machine learning model can be trained to distinguish between a single signal and a pair of signals separated by time separations of less than #box("2.0" + h(1.5pt) + "s") and greater than #box("0.1" + h(1.5pt) + "s") if the minimum SNR of the signal was sufficient for differentiation. With greater hyperparameter tuning and adjustments to the training dataset, substantive improvements in differentiation ability could be made. This suggests that a machine learning method such as Overlapnet may be a good addition to a future CBC pipeline which has to contend with the possibility of overlapping signals. It could act alone or as one of a suite of methods to switch between alternate parameter estimation methods designed to deal specifically with overlapping signals.

CrossWave has shown that machine learning methods can be used to extract the merger times of two overlapping signals with moderate success. Again, CrossWave or an improved model could be used as part of a larger parameter estimation pipeline, to provide priors to a more established parameter estimation method, once a pair of overlapping signals has been identified in the data.

CrossWave has also demonstrated limited parameter estimation ability of its own, across a few other parameters. It most successfully extracted predictions for the masses of the companions of signals A and B with some accuracy. It also showed limited potential to extract predictions of the Z dimensionless spin component. This method of directly extracting parameters using CrossWave has limited application, as it cannot deal with uncertainty or multi-modal probability spaces, which are needed for a robust and modern parameter search. It is possible that a machine learning method could be created with these features, through the use of Bayesian Neural Networks (notably distinct from Bayesian networks), or a multimodal latent space perhaps within an autoencoder framework, similar to the function of the VITAMIN parameter estimation method @vitamin.

The large model developed for CrossWave has performed well, and warrants further investigation, and comparison against other detection methods in the single signal detection problem are recommended and were desired but abandoned due to time constraints.