{'fraction_non_alphanumeric': 0.03355722802190391, 'fraction_numerical': 0.009333441234388066, 'mean_word_length': 5.13165729407873, 'pattern_counts': {'":': 0, '<': 2, '<?xml version=': 0, '>': 2, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Open set learning with augmented category by exploiting unlabelled data (open-LACU)

Considering the nature of unlabelled data, it is common for partially labelled training datasets to contain samples that belong to novel categories. Although these so-called observed novel categories exist in the training data, they do not belong to any of the training labels. In contrast, open-sets define novel categories as those unobserved during during training, but present during testing. This research is the first to generalize between observed and unobserved novel categories within a new learning policy called open-set learning with augmented category by exploiting unlabeled data or open-LACU. This study conducts a high-level review on novelty detection so to differentiate between research fields that concern observed novel categories, and the research fields that concern unobserved novel categories. Open-LACU is then introduced as a synthesis of the relevant fields to maintain the advantages of each within a single learning policy. Currently, we are finalising the first open-LACU network which will be combined with this pre-print to be sent for publication.


Introduction
Inductive reasoning is the process of generalising principles to match specific observations with their respective outcomes. Supervised learning is an inductive machine learning policy wherein pre-annotated input-output sample pairs (i.e. observations-outcomes) are provided to networks for back-propagated learning [19]. Specific to classification, these input-output labelled pairs are input data samples with their respective output class/category labels. Supervised classifiers learn from labelled samples to relate data with similar patterns (i.e. those belonging to the same category) and distinguish between data with different patterns (i.e. samples belonging to different categories). In turn, classifiers generalise knowledge about labelled/known categories so that they can be deployed to classify never-before-seen testing samples.
Supervised learning has three significant obstacles: 1) supervised classifiers generally require many labelled training samples to learn from [44]; 2) it is costly and sometimes impractical to assume that every category has been labelled [12,2,14]; and 3) many data-driven applications are volatile leading to never before seen patterns appearing after training [20,13,18]. These three obstacles have been respectively addressed in research by 1) semisupervised learning (SSL), 2) learning with augmented category by exploiting unlabeled data (LACU), and 3) novelty detection/open-set recognition (OSR) [17,18]. Although these fields have unique advantages, no learning policy has coherently synthesised all three. This study introduces such a synthesised learning policy named open-set learning with augmented unknown category by exploiting unlabeled data or open-LACU.
In supervised learning, training labels represent all possible categories of the input data. In other words, every input sample encountered by the classifier is assumed to belong to one of the known categories. This setting, known as testing under a closed-set, rarely holds in practice due to the intrusion of outliers, anomalies and novel categories within data domains [20,5]. Outliers, anomalies and novel categories should not be classified into any of the training labels as these do not match the patterns of the known categories. Instead, outliers, anomalies, and novel categories should be separated into different piles. In this study, we specifically focus on novel categories, which are groups of anomalous samples that have similar patterns. Novel categories appear in a domain for one of two reasons: 1) these categories did not yet exist at the time of training and so could not have been labelled [20,18] or 2) these categories were not of interest to an application and were left unlabelled for cost-saving purposes [2,14].
In literature, the first and most common form of novel categories are unobserved novel categories. Unobserved indicates that these categories do not have any samples that belong to them within the training set as these categories only appear during testing. The second less common form of novel categories are observed novel categories. Observed indicates that these categories do have samples represented in the training set; however, these observed categories are still novel and do not belong to any training labels. Consequently, observed novel categories can only be represented in partially labelled datasets as their training samples are always unlabelled. To the best of our knowledge, no previous study has generalised the difference between unobserved and observed novel categories. However, the unique attributes of unobserved novel categories make them worth separating from all categories represented during training, including observed novel categories.
Observed novel categories are most prevalent within big datasets that represent a plethora of categories [14]. In these datasets, gathering labelled samples from every category is expensive making it is more practical to only gather labels for those categories relevant to a specific application. However, only labelling some but not all categories means observed novel categories will be scattered in the unlabelled training data. In contrast, unobserved novel categories represent new and interesting patterns/discoveries that appear over time [13]. A significant example of an unobserved novel category is the sars-cov2 virus, which did not exist within the domain of respiratory diseases before its discovery in December 2019 [1]. For processes such as early detection, completely separating unobserved novel categories from the training domain provides practitioners with the necessary access to samples of the newly discovered categories. However, to achieve this separation, unobserved novel categories must also be separated from observed novel categories.
Novelty detection is commonly left as a stand-alone process [20,13]. However, several research fields study the classification of multiple known categories with simultaneous detection of novel categories. These fields focus on either observed novel categories or unobserved novel categories, but never both. More specifically, open-set recognition (OSR) focuses on supervised classification with detection of unobserved novel categories [18], while learning with augmented category by exploiting unlabelled data (LACU) focuses on semi-supervised classification with detection of observed novel categories [14]. This study uniformly synthesises OSR and LACU to train classifiers under the cost-sensitive semi-supervised training scheme of LACU while maintaining the enhanced applicability of OSR. This new learning policy, open-LACU, specifically requires the classification of multiple known categories, detection of observed novel categories, and separate detection of unobserved novel categories.
Open-LACU defines two augmented categories to encapsulate observed and unobserved novel categories. More specifically, in a domain with K known categories, a K + 1'th augmented category is defined for all observed novel categories, and a K + 2 augmented category is defined for all unobserved novel categories. However, it is important to re-iterate that the K + 1 and K + 2 augmented categories have no labelled training samples. Instead, the K + 1'th augmented category only has unlabelled training samples, while the K + 2 augmented category does not have any training samples. To generalise these augmented categories, samples that belong to the K + 1'th category must be filtered out of the unlabelled training set and be pseudo-labelled into the K + 1 category. Subsequently, the K + 2 augmented category can either be generalised by placing probability thresholds on classifications overall K + 1 categories [31,18], or by generating and pseudo-labelling samples that exclusively belong to the K + 2 augmented category [32,7].
The remainder or this pre-print study provides 1) the relevant background on classification within machine learning, 2) a brief review on the various fields within the novelty detection, specifically focusing on the training-testing criteria of various fields, 3) and formally introducing the open-LACU policy. The authors of this pre-print or currently designing the first open-LACU network, which we aim to append to this pre-print and send for publication. Open-LACU synthesis two application-grade learning policies to maintain the advantages of both within a single network. Open-LACU certainly allows for an extensive number of applications, especailly so for the big data domains.

Background
Inductive learning trains classifiers to learn general rules of a domain so that these rules can be applied on never before seen data [44,20]. Example applications of inductive classifiers include self-driving cars [37], disease detectors [34], and remote sensors [43], wherein never before seen input data is streamed over time and must be classified accordingly. To ensure safe classifications, inductive learning policies define their testing phase as isolated from the training phase to simulate the model in a real-world environment. Beyond isolated testing, testing sets regularly include novel categories as trained classifiers can encounter new novel patterns after being deployed. However, some learning policies do not include novel categories in their criteria, while others deal with novel categories in conflicting ways. This section provides the necessary background on popular inductive learning policies.
Supervised learning has a training set, D train , and a separate testing set, D test . Both sets contain input samples paired with their respective output category labels, i.e. (x, y) with x the input sample and y the label. However, only the labels from D train are available to the classifier for backpropagated learning. The testing labels are hidden from the classifier as these need to be inferred for evaluation measurements. Therefore, for clarity here and further on, the labels in D test are reconstructed as anticipated labels, (y a ), which indicates that the sample belongs to a certain category, yet this category is unknown to the classifier. For supervised learning, each output label represents one of K known categories described in C K := {1, 2, ..., K}. More formally, supervised learning has all {y ∼ D train } ∈ C K and all {y a ∼ D test } ∈ C K . It is clear that general supervised learning does not contain novel categories within its training-testing criteria.
Semi-supervised learning (SSL) trains classifiers using partially labelled training sets, D train := D lab-train ∪ D unlab-train [44]. Samples in D lab-train are labelled pairs, (x, y), while samples in D unlab-train are only input samples, (x), without any labels. Assuming that unlabelled training samples belong to certain categories, the unlabelled training samples can be appended with anticipated labels, i.e. (x, y a ), similar to the testing set. Semi-supervised models combine supervised learning, applied on D lab-train , and unsupervised learning, applied on D unlab-train , in a single training scheme. In turn, considerable fewer labelled training samples are required for high-accuracy classification. After training, SSL has the same testing phase as supervised learning. Formally, SSL has all {y ∼ D lab-train } ∈ C K , all {y a ∼ D unlab-train } ∈ C K and all {y a ∼ D test } ∈ C K , showing again that there are no novel categories within general SSL training-testing criteria.
The supervised and SSL policies do not have novel categories within their training-testing criteria. As to say, supervised and SSL test classifiers under the assumption of a closed-set. However, many application-grade datasets contain samples that belong to novel categories which can either appear in a partially labelled training set [14] or the testing set [30,23,28]. In the case of supervised learning, all training samples are labelled and can only belong to known categories. For SSL, however, the unlabelled training samples in D unlab-train cannot be guaranteed to belong to a known category by nature of them being unlabelled [2,14,8]. After training, testing samples in D test for both supervised and SSL can also not be guaranteed to belong to a known category [46,18]. In other words, all samples with anticipated labels could belong to a known or novel category, with classifiers having no pre-determined annotation of which belongs to which.
Studies concerning novel category detection regularly use inconsistent naming conventions leading to complicated and convoluted literature reviews [46]. For example, out-of-distribution detection [17], anomaly detection [6], and novel category detection [13] regularly have the same experimental setups in their respective studies. Yang et al. [46] addressed the inconsistent naming conventions by proposing the generalized out-of-distribution detection taxonomy. However, Yang et al. only considered the research fields where novel categories are unobserved during training and not the fields wherein novel categories are observed during training. These fields include learning from positive and unlabelled data (PU learning) [2], mismatched semi-supervised learning (MSSL) [8,21,27], and learning with augmented class by exploiting unlabelled data (LACU) [12,14].

Novel category detection
Geng et al. [18] defined four category types within classification tasks: 1) known known categories, which are the K known categories with annotated class labels; 2) known unknown categories, which are the known novel categories, meaning these categories do have labelled training samples available for them, but these labels are all for the background augmented category; 3) unknown known categories, which are categories without any samples available during training, but some semantic side information about these categories are available; and 4) unknown unknown categories, which are those completely unobserved during training but are encountered during testing. These category definitions do not take partially labelled datasets into consideration. This section provides a reformulation of these category definitions. Subsequently, the relevant research fields for unobserved and observed novel category detection are discussed in detail.

Category types
The following definitions extend on the category types defined by Geng et al. [18] to respect novel categories within partially labelled datasets. Within these definitions, we introduce the taxonomy of observed and unobserved categories. These definitions provide a clearer definition by 1) replacing the redundancy of, for example, 'known known categories' to instead be 'observed known categories', and 2) allowing for the necessary distinction between novel categories represented during training and novel categories not represented during training. Our category definitions also include an ad-hoc definition of augmented categories: • Augmented categories: these categories encapsulate any group of categories. Augmented categories must be generalized so that their class boundaries encapsulate all categories assigned to them. The most popular augmented category is the 'background/other/unknown' category which is regularly used to encapsulate all novel categories in a domain [18,14]. Note that labelled training samples might or might not be available for an augmented category, as is specified by the learning policy.
• Observed known categories: here, observed indicates that samples which belong to these categories exist in the training set, while known indicates that these categories are defined in the list In other words, these categories are the K number of labelled/known categories, each with at least one annotated sample in D lab-train . Formally, all [(y) ∼ D lab-train ] ∈ C K , which represents all the available training labels. For unlabelled training sets, D unlab-train , practitioners can assume that all {(y a ) ∼ D unlab-train } ∈ C K if, and only if, C K extensively represents all categories in the domain. If not, practitioners must assume that D unlab-train contains samples that belong to observed known categories and samples that belong to observed novel categories, which are defined next.
• Observed unknown/novel categories: here, observed indicates that samples which belong to these categories do exist in the training set. However, novel indicates that these categories do not belong to any training labels. In other words, training samples from these categories only exist as unlabelled samples in training sets. Observed novel categories can be treated in one of two ways: 1) in the case of class-incremental learning [11,48], models can append observed novel categories to C K as they are discovered; 2) observed novel categories can be detected and separated into an additional K + 1'th augmented category. The second option is the focal point of novel category detection studies and is certainly a pre-step for the first option. Concerning only the second option, it is important to note that some studies assume that labelled samples are available for observed novel categories -i.e. with a K + 1 label [16,38]. However, gathering these labels increases annotation costs and can result in inefficient representation for the K + 1'th label []. Therefore, most studies assume that there are no labels available for observed novel categories [2,14,21,47]. Also, note that the number of observed novel categories in a domain is generally unknown but can be statistically approximated by comparing D lab-train with D unlab-train [4].
• Unobserved known categories: here, unobserved indicates that there are no input samples that belong to these categories in the training set. However, considering that these categories are known, they are defined in C K and so do have training labels. In other words, output training labels are available for these categories, but input training samples are not. These categories exclusively fall under the field of zero-shot learning ( [42], wherein side semantic information about each category is incorporated within training schemes. Thus, classifying networks can generalize these categories without input samples using the additional information.
Note that these categories are not of concern to this study.
• Unobserved unknown/novel categories: here, unobserved indicates that no samples belong to these categories within the training set. Again, novel indicates that these categories do not belong to any training labels. In other words, these categories appear over time, meaning they only appear during testing. These categories are the most common definition for novel category detection studies [36,18,46,20]. More formally, unobserved novel categories lie within the low-density region of the training data, and each category represents a new and interesting discovery within the domain. Similar to observed novel categories, unobserved novel categories can either be appended to C K as they are individually discovered, or they only need to be detected and separated into an additional augmented category. Please note that in the inductive setting, there is no way to approximate the number of unobserved novel categories.
The learning policy introduced in this study focuses on classifying multiple observed known categories (i.e. K > 1) while detecting and separating observed and unobserved novel categories. Within research literature, novel category detection is split into two research paths: 1) those that exclusively study unobserved novel categories, and 2) those that exclusively study observed novel categories. These research branches and their multiple sub-branches are briefly reviewed below. Table 1 then summarises the review's findings, which specifically highlight the types of categories in the training and testing criteria of all the relevant learning policies.

Unobserved novel category detection
Unobserved novel category detection is the process of separating samples that belong to observed known categories from samples that belong to unobserved novel categories. Gruhl et al. [20] described two different types of unobserved novel categories: 1) unobserved novel categories lie in high-density regions of the training data and represent conceptual drifts within the structure of observed known categories, or 2) unobserved novel categories lie in low-density regions of the training data and represent new and interesting discoveries within the domain. Considering that concept drifts are structural changes of the observed known categories [13,20,29,25], this study does not agree with the notion that concept drifts are novel categories. Instead, and as indicated in the definitions above, we follow the more commonly used second definition, wherein unobserved novel categories are new discoveries that appear over time.
It is essential to distinguish between unobserved novel categories that appear due to a semantic shift and those that appear due to a covariate shift [46]. Covariate shifts occur in transfer learning and domain adaption scenarios. In these cases, trained classifiers encounter multiple domains during testing [49,15]. An example of a covariate shift scenario is when a classifier was trained using data from one hospital but then deployed on data from various other hospitals. In contrast, semantic shifts occur within inductive classification scenarios where new and interesting categories appear during testing, even though the classifier is trained and tested on the same domain. The running example for semantic shifts is the sars-cov2 virus, which did not exist within the domain of human respiratory diseases until its discovery in December 2019 [1]. This study focuses on inductive classification, and so unobserved novel categories only occur due to semantic shifts in the domain.
Unobserved novel category detection is commonly treated as a binary classification problem [36]. In binary novel category detection, classifiers must determine whether a sample is from a novel category or not. However, this study focuses on classifying multiple observed known categories (i.e. K > 1) with simultaneous detection of novel categories. This task places us in the sub-field of novelty detection called open-set recognition (OSR) [18]. OSR aims to train multi-category classifiers under the assumption of an open-set, meaning unobserved novel categories are present in the domain. In general, OSR models define an additional K + 1'th augmented category to represent all unobserved novel categories. More formally, OSR has all {y ∼ D train } ∈ C K and all {y a ∼ D test } ∈ C K ∪ {K + 1}. The K + 1'th category is then generalised either by applying probability thresholds on a K category classifier or by using generative models to create samples that can be used to generalise the unknown space [7].

Observed novel category detection
Observed novel categories exist in partially labelled datasets amongst observed known categories. In our previous study [14], observed novel categories were described to appear in partially labelled datasets for one of two reasons: 1) practitioners were unaware of their existence during the labelling process, and so could these categories were not given training labels, and 2) these categories were not of interest to the application and so were purposefully left unlabelled for cost-saving purposes. No matter the reason, the fact that observed novel categories exist in the same domain as observed known categories means that observed novel categories can only have a semantic shift from observed known categories. Similar to unobserved novel category detection, observed novel category detection is either a binary or multi-category classification problem. Each of these problems and their respective learning policies are discussed.
Training classifiers using partially labelled datasets generally falls under semi-supervised learning (SSL) [44]. However, state-of-the-art SSL methods were designed under the assumption that there are no observed novel categories in the training domain. Consequently, state-of-the-art SSL models perform poorly in the presence of observed novel categories in the domain [33]. To address this collapse, researchers have developed a new learning policy called mismatched semi-supervised learning (MSSL) [21] (also referred to as safe semi-supervised learning [8] and universal semi-supervised learning [27]). In MSSL, the term 'mismatch' describes the relationship between the labelled training set, D lab-train , and the unlabelled training set, D unlab-train . If the categories represented in D lab-train do not match the cate-gories represented in D unlab-train , then there is a mismatch between these two sets. More formally, if D unlab-train contains samples that belong to observed novel categories, then there is a mismatch between D lab-train and D unlab-train .
Three different mismatch settings have been described: 1) a subset mismatch, which means that D unlab-train contains samples that belong to observed novel categories and samples that belong to all observed known categories, 2) an inter-sectional mismatch, which means that D unlab-train contains samples that belong to observed novel categories and samples that belong to some but not all observed known categories, and 3) a complete mismatch, which means that D unlab-train only contains samples that belong to observed novel categories [27]. However, due to the unknown nature of observed novel categories, it is difficult to determine which type of mismatch a training set has. Consequently, methods that train using mismatched partially labelled datasets should either be robust to each mismatch setting or compare D lab-train with D unlab-train to statistically approximate the type of mismatch [4].
None of the fields discussed in this sub-section contains unobserved novel categories. Therefore, for this discussion, the K + 1'th augmented category is defined to only encapsulate observed novel categories. To describe different mismatch settings, a new set C M is also defined, which is a sub-set of C K that contains the observed known categories that are not represented in D unlab-train . Subsequently, a mismatched partially labelled training dataset has all {y ∼ D lab-train } ∈ C K and all {y a ∼ D unlab-train } ∈ (C K − C M ) ∪ {K + 1}. A subset mismatch would then have an empty C M , an inter-sectional mismatch would have C M containing some but not all observed known categories, and a complete mismatch would have C M = C K . Again, C M cannot be explicitly known. Therefore, models trained using mismatched partially labelled training datasets must either include statistical approximations of C M by comparing D lab-train with D unlab-train , or be robust to any C M .
Mismatched semi-supervised learning (MSSL) aims to alleviate the performance degradation of general semisupervised learning (SSL) in the presence of observed novel categories. However, MSSL does not require trained models to detect and separate samples from observed novel categories during testing. In other words, MSSL includes observed novel categories during training but not during testing, as only the observed known categories are of concern to the trained classifier. Formally, MSSL has all {y ∼ D lab-train } ∈ C K , all {y a ∼ D unlab-train } ∈ (C K −C M )∪{K +1} and all {y a ∼ D test } ∈ C K . To achieve MSSL, researchers generally apply a pre-training filtering step to remove samples out of D unlab-train that belong to observed novel categories [26,8,21]. With these samples removed, practitioners can train a general semi-supervised classifier without fear of performance degradation. However, MSSL is not a solution to observed novel category detection, as trained models need not detect and separate these categories.
To the best of our knowledge, only two research fields train classifiers using mismatched partially labelled datasets and also require models to separate samples from observed novel categories. These two fields are positive and unlabelled (PU) learning [24,9,2] (also known as semi-supervised novelty detection [4]), and its multi-category extension of learning with augmented category by exploiting unlabelled data (LACU) [12,35,40,14] (also known as the inappropriately named open-set semi-supervised learning [47]). PU learning places all observed known categories into the positive augmented category (C + ) and all unobserved novel categories into the negative augmented category (C − ).
In contrast, LACU requires models to classify each of the K number of observed known categories while placing all observed novel categories into the K + 1'th augmented category.

Training-testing criteria
Open-LACU is an inductive learning policy that trains classifiers off partially labelled datasets to classify multiple known categories while simultaneously separating novel categories. Unlike other learning policies, open-LACU assumes the presence of observed novel categories and unobserved novel categories within its domain (see Table 1). To ensure generalisation between the two novel category types, open-LACU requires models to separate observed novel categories and unobserved novel categories into two different augmented categories. By doing so, new discoveries are detected quickly and efficiently, improving the application, while the training domain does not need to be completely labelled concerning each category.
In summary, open-LACU models must generalise between 1) K number of observed known categories, 2) all observed novel categories placed into the K + 1'th augmented category, and 3) all unobserved novel categories placed into the K +2 augmented category. Note that labelled training samples are only available for the K observed known categories but not for the two augmented categories. Furthermore, considering a partially labelled training set, unlabelled training samples can belong to either an observed known category or the K + 1'th augmented category, with no pre-determined annotations of which belongs to which. Thus, formally, open-LACU has all {y ∼ D lab-train } ∈ C K , all {y a ∼ D unlab-train } ∈ (C K − C M ) ∪ {K + 1} and all {y a ∼ D test } ∈ C K ∪ {K + 1, K + 2}.

Evaluation measurements
Open-LACU has K + 2 number of labels within the testing set. Although a general accuracy measurement over all labels might seem intuitive, it is important to note that the spread of samples overall K + 2 labels will rarely be equal. More specifically, because the K + 1 and K + 2 labels represent augmented categories, these could include multiple observed novel and multiple unobserved novel categories respectively. Therefore, it is safer to assume that more samples will have the K + 1 and K + 2 labels compared to each observed known category. In such cases, a normalised accuracy measurement over all labels is necessary.
OSR studies use multiple normalised evaluation measurements to address the unequal spread of labels [18]. Of these, the most common measurement is the F1 score which is the harmonic mean of the classifier's precision and recall [41]. The F1-score is calculated either as a macro average F1 (MF1) or a micro average F1 (mF1). The micro-average favours categories with a large number of samples. In contrast, the macro-average ensures that each category is weighted equally regardless of the number of samples. Consequently, to ensure the evaluation measurement is aware of the unequal spread of labels, MF1 is most appropriate. However, Mendes et al. [31] proposed an altered version of MF1 for OSR, referred to as open-set MF1 (OSMF1), to respect the nature of unobserved novel categories.
Mendes et al. [31] argue that true positive classifications of samples that belong to unobserved novel categories (i.e. the K + 2 augmented category in our case) are insignificant for evaluation measurements. This argument is founded on the commonly used reject option for OSR [18]. Previous studies have applied reject options on models that did not generalise the unknown space for unobserved novel categories. These models detect unobserved novel categories through a probability threshold enforced on the closed set classifier. Classification scores below these thresholds are rejected from being classified into any training labels. However, more recent OSR studies have shown that even without any representation during training, unobserved novel categories can be generalised into a bounded unknown space [7].

Benchmark experiments
Benchmark experiments for OSR and LACU are generally conducted using fully labelled classification datasets such as MNIST and CIFAR10 (each having 10 distinct categories) [18]. To simulate an OSR environment, K < 10 number of categories are randomly selected as observed known categories while the remaining categories are set as unobserved novel categories. Subsequently, all samples that belong to unobserved novel categories are removed from the training sets, while the labels of their testing samples are changed to K + 1. Similarly, to simulate a LACU environment, K < 10 number of observed known categories with the remaining categories set as observed novel categories. Subsequently, some labels are masked from training samples that belong to observed known categories, while all labels are masked from training samples that belong to observed novel categories. Furthermore, labels of testing samples that belong to observed known categories are changed to K + 1. • MNIST-KMNIST: MNIST contains 10 categories of hand-written digits with samples being 28 x 28 black and white images (see Fig. ?? for example data points). Each of the 10 categories has ±7000 labelled samples available, which are generally split as 6000 samples for training/validation and 1000 samples for testing. A similar dataset to MNIST was recently constructed called Kuzushiji-MNIST (KMNIST) [10].
KMNIST is a similar dataset to MNIST in that it also contains 28 x 28 black and white images that belong to 10 distinct categories. However, the categories within KMNIST represent 10 different hand-written cursive Japanese characters. Because both MNIST and KMNIST represent hand-written characters, these two sets are considered from the same domain but with semantic shifts from one another. Consequently, MNIST and KMNIST are combined for open-LACU experiments. However, to ensure consistency, MNIST categories are always used for randomly selecting observed known and observed novel categories, while KMNIST categories are always selected as unobserved novel categories.
• QuickDraw: As an extension to the MNIST-KMNIST experiment, we propose using a dataset with considerable more samples and categories. Specifically, the QuickDraw dataset is used, which also consists of 28 x 28 black and white hand-drawn images [22]. However, the QuickDraw dataset has ±70000 labelled images per category with 345 different categories of various hand-drawn images (see Fig. ?? for example data points). With so many categories, QuickDraw does not need to be combined with any other dataset. Furthermore, given such a large number of available samples within the QuickDraw dataset, experiments on this dataset will provide researchers with some indication of model performance on big data.
To the best of our knowledge, open-LACU is the first learning policy that generalises between observed and unobserved novel categories. Considering that big datasets generally include both novel category types within their domains, open-LACU is arguably a more suitable learning policy for applications. Concerning experiments within this study and future studies, the evaluation measurement proposed above are simple to implement and urged to remain consistent. Furthermore, bar the datasets used, proposed open-LACU methods should challenge 1) the number of labelled training samples per observed known category (the less, the better to decrease annotation cost), and 2) the relationship or the mismatch between D lab-train and D unlab-train , whereby methods that are robust to any mismatch setting are considered most favourable. With open-LACU formalised, we now move to the first proposed open-LACU network.

Conclusions
As a brief summary of this pre-print study, in this research we introduced the open-LACU learning policy. Open-LACU is the first policy to generalize between observed and unobserved novel categories, which greatly expands the application of resulting networks. Fundamentally, open-LACU is a synthesis of open set recognition (OSR) and learning with augmented unknown category by exploiting unlabelled data (LACU). OSR ensures classifiers are able to separate new categories that appear after training while LACU provides a more cost-sensitive solution to the training data setup in respect to the category labels. Open-LACU maintains both these advantages. We argue that open-LACU open up new research paths, especially so for big data classification.