{'fraction_non_alphanumeric': 0.030516732914476215, 'fraction_numerical': 0.015001923323503013, 'mean_word_length': 4.6686046511627906, 'pattern_counts': {'":': 0, '<': 4, '<?xml version=': 0, '>': 3, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Broad Absorption Line Quasar catalogues with Supervised Neural Networks

We have applied a Learning Vector Quantization (LVQ) algorithm to SDSS DR5 quasar spectra in order to create a large catalogue of broad absorption line quasars (BALQSOs). We first discuss the problems with BALQSO catalogues constructed using the conventional balnicity and/or absorption indices (BI and AI), and then describe the supervised LVQ network we have trained to recognise BALQSOs. The resulting BALQSO catalogue should be substantially more robust and complete than BI- or AI-based ones.

The most straightforward explanation for the differences between QSOs and BALQ-SOs is a simple orientation effect. Thus all QSOs may undergo significant mass loss through winds, but BALs are only observed if the central continuum and/or emission line source is viewed directly through the outflowing material. Viewed in this context, BALQSOs may be the only available tracers of a key physical process common to all AGN. Moreover, the fraction of QSOs displaying BAL features ( f BALQSO ) may provide a direct estimate of the opening angle of these outflows.
Historically, most BALQSO samples were selected on the basis of the so-called balnicity index (BI; [2]) or similar metrics. These samples consistently yielded BALQSO fraction estimates in the range f BALQSO ≈ 0.10 − 0.15 ([2, 7, 4, 8]). In a previous paper ( [9]; Paper I), we showed that the BI and a recently defined metric, the absorption index (AI; [10]), are to strict or relaxed respectively when selecting BALQSOs.
Here we will use the hybrid-LVQ approach from Paper I using a combination of the classic BI, a simple neural network and visual inspection to produce BALQSO samples that are more robust than AI-based ones, but more complete than purely BI-based ones using the QSO sample associated with Data Release 5 (DR5) of the Sloan Digital Sky Survey (SDSS; [11,12]). Our catalogue contains 3505 BALQSOs selected from 28,421 objects in the SDSS DR5 QSO sample in the redshift range 1.7 < z < 4.2.

THE INPUT QSO SAMPLE
The SDSS DR5 QSO catalogue contains over 77,000 objects in total [12]. However, for the purpose of constructing a uniform BALQSO catalogue, we only consider objects whose spectra fully cover the C IV 1550 Å resonance line, which displays a particularly deep and well-defined absorption through in the spectra of most BALQSOs. Given the wavelength range covered by the SDSS spectra, this implies an effective redshift window of 1.7 < z < 4.2 for our QSO parent sample, which contains spectra of 28,421 objects.
Our BALQSO classification method works on continuum-normalised spectra covering the wavelength range 1401 -1700 Å with 1 Å dispersion. It also uses the associated BIs for training the neural network and to flag borderline cases requiring visual inspection. We therefore normalise all QSO spectra using the method described in [9,13], in which each spectrum is fit with a modified DR5 QSO composite allowing for object-toobject differences in reddening and overall spectral slope [14]. We then interpolate each spectrum onto the new wavelength grid and estimate the BI in the same way as [2].

HYBRID-LVQ SELECTION OF BALQSOS
The method we use to classify BALQSOs has already been described in detail in [9], so we only provide an overview of the key points here. Briefly, our method is a hybrid of BI-based, neural network and visual classifications. It is designed to produce a more complete BALQSO sample than a pure BI selection without significantly increasing the number of false positives. Starting with a BI-based classification, we use a simple neural network-based machine learning algorithm called "learning vector quantization" (LVQ, [15]) to identify objects that might have been misclassified by the BI. All such objects are then inspected and classified visually. The way in which we train our LVQ network to recognize BALQSOs has been described in detail in Paper I. Note that redshift uncertainties are explicitly taken into account by our network. Below, we will sometimes refer to the full hybrid method as "LVQ-based", but it is always worth keeping in mind that LVQ is only one part of a process also involving the BI and visual inspection. Fig. 1 shows a flow diagram of the steps involved in creating the final DR5 BALQSO catalogue using the hybrid-LVQ network. Our LVQ-based method classifies 3,385 of the 28,421 QSOs (11.91% ∓ 0.21%) in our DR5 parent sample as BALQSOs and the catalogue can be found online 1 . It is reassuring to note that the LVQ classifications and the BI ones tend to agree for over 92% of the objects. The ones which the methods disagree on are visually inspected for their classifications. Overall we find that 29% of the BI > 0 objects require visual inspection (848), whilst only 5% (1224) of the BI = 0 do. At first one might think that this is quite a high fraction for the BI > 0 objects.  [9]). The other panels in Fig. 2 show composites created for higher AIs and BIs. The composite containing the most QSOs is the middle-left, which could also be considered as being the region in AI-BI space including the most borderline cases. This again helps explain the high fraction of objects which the BI and LVQ disagree.

PROPERTIES OF THE FINAL BALQSO CATALOGUE
In order to explore our results in more detail we present the composites in Fig. 3. The top row of panels show composites produced with QSOs which were finally tagged as BALQSOs, whilst the bottom row with non-BALQSO objects. As expected the topcentre composite displays a very narrow absorption, which will be missed by the BI calculation due to it's conservative definition. Next we consider the bottom-centre panel displaying the composite of objects with BI > 0 but a non-BALQSO final tag. The spectrum is redder compared to the non-BALQSO composite, but has no clear signs of line absorption.

CONCLUSIONS
Compiling BAL quasar catalogues and determining the observed BALQSO fraction is a challenging task. Most of the problem resides in the ambiguity one encounters when  attempting to classify individual absorption features. After all, it is in the human nature to be subjective, even if one tries the opposite. The judgment will be based on previous examples of what is and what is not a BAL. This is why we have used a hybrid method consisting of a simplified neural network together with visual inspection to create a hopefully near-optimal BALQSO catalogue.
In [9] and this work we have shed light on many classification problems when dealing with BALQSOs taken from contemporary surveys. We showed that when the recently introduced "absorption index" (AI) is used to classify BALQSOs, the resulting log AI distribution is clearly bimodal. Both modes containing comparable amount of objects, but only the high-AI mode clearly being associated with genuine BALQSOs. Moreover, in our previous paper, we showed how even the traditional "balnicity index" (BI) produces incomplete BALQSO samples. It is likely that due to the diverse nature of observed BAL throughs, conventional metrics are no longer appropriate given the large data volume increase in observed QSO samples. Also it seems even more unfeasible to define new metrics in order to deal with problems caused by the old ones. Here we have shown how a hybrid algorithm can overcome these problems, taking into account the increasing data volume gathered by contemporary astronomical surveys.
The observed fraction is, however, still subject to serious selection effects. In [9] we have explored these and corrected for colour-, magnitude-and redshift-dependent selection biases on the DR3 dataset. After applying the corrections we reached the conclusion that there is no compelling evidence for redshift evolution in the intrinsic BALQSO fraction.