{'fraction_non_alphanumeric': 0.03128182744071002, 'fraction_numerical': 0.010233279984480334, 'mean_word_length': 5.073637702503682, 'pattern_counts': {'":': 0, '<': 35, '<?xml version=': 0, '>': 31, 'https://': 2, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Annotated Dataset Creation through General Purpose Language Models for non-English Medical NLP

Obtaining text datasets with semantic annotations is an effortful process, yet crucial for supervised training in natural language processsing (NLP). In general, developing and applying new NLP pipelines in domain-specific contexts for tasks often requires custom designed datasets to address NLP tasks in supervised machine learning fashion. When operating in non-English languages for medical data processing, this exposes several minor and major, interconnected problems such as lack of task-matching datasets as well as task-specific pre-trained models. In our work we suggest to leverage pretrained language models for training data acquisition in order to retrieve sufficiently large datasets for training smaller and more efficient models for use-case specific tasks. To demonstrate the effectiveness of your approach, we create a custom dataset which we use to train a medical NER model for German texts, GPTNERMED, yet our method remains language-independent in principle. Our obtained dataset as well as our pre-trained models are publicly available at: https://github.com/frankkramer-lab/GPTNERMED


Introduction
In situations of low resource languages, neural baseline techniques for specific tasks in natural language processing (NLP) are often difficult to be applied successfully due to the lack of sufficient and adequately annotated training data. While English can be perceived as the most relevant language in the field of NLP research as being a high-resource language, effectively any other language can be considered as rather low-resource language in contrast. Yet, the abundance of plain textual resources is no uniquely decisive factor when it comes to dealing with embedded NLP problems in real-life applications. In this regard, a domain-specific dataset needs to be obtained to match to the applied context and the underlying data acquisition process can involve access to highly restricted data, manual engagements from domain experts or time-and costintensive data gathering. Another concern relates to the actual NLP objective of the use case and usually heavily determines the final design of the obtained dataset and its collection of task-related annotations.
We study the use case to annotate certain medical entity classes in German throughout this paper since it is an instance that suffers from all formerly mentioned challenges. In this work, we demonstrate an effective method for synthesizing a custom, domain-aligned dataset with annotation information in an unsupervised fashion. Furthermore, we show evidence of its effectiveness by training a generic medical model for German medical named entity recognition (NER) by finetuning a pre-trained language model. Due to the inherently generic nature of our work, we do not see fundamental obstacles in apply the approach on related entity classes in medical or even non-medical tasks, or for different non-English languages of similar quantitative level of resource abundance.

Medical Datasets
In NLP, deep learning-based methods have been proven as highly effective in order to tackle frequent tasks, most notably the self-attention-mechanism-based transformer architecture. [42] One fundamental problem of deep learning-based methods remains to be the need for vast amount of data for training, including corresponding annotations for supervised learning.
In English medical NLP, these challenges have been addressed to a certain extend by the availability of annotated datasets, such as the MIMIC-III [27] and MIMIC-IV [17] datasets or n2c2 datasets from the i2b2 challenges [16]. In general, multilingual textual datasets are available that carry medical texts from multiple languages. The datasets often entail parallel corpora for translation tasks and lack semantic annotation like the UFAL Medical Corpus 1 for the WMT'17 biomedical challenge [45]. Driven by manual annotation work, Mantra GSC [19] is a public gold-standard annotated corpus with multilingual texts based on prior parallel corpora and provides limited UMLS information.
For German medical NLP, the field has made notable advances in terms of available datasets. While work in this field of NLP has been published, internal and proprietary datasets are frequently used as underlying datasets. [15,33,44,21,41,7,12,22,11,24,26,20] In recent years, semi-publicly available datasets like BRONCO [18] and GGPONC 1.0 [6] and 2.0 [6] have been made available. While BRONC is advertised based on real discharge letters with annotations, other datasets like GGPONC originate from synthetic data sources like clinical practice guidelines, assembled from multiple or crawled data from the web. If annotation data is provided, such metadata differ in terms of entity types, entity type definitions or their overall task objectives. Hence, a direct comparison of datasets and corresponding models cannot be made directly with respect to NER F1/tagging scores, or entity linking to different onthologies. Only metrics of rather limited interest such as test set performance of trained models, or token size and number of entities for a dataset are directly derivable for comparison. For an extensive overview on the recent state of German medical datasets, we point to [6].

Medical Models and Applications
We restrict our focus on models and applications to items of general interest and practical applicability. Most works from the presented dataset section develop accompanied models to the datasets and publish internally evaluated scores. However in many cases, the reproducibility of the described results is not possible since models are not made publicly available along with the paper. Furthermore, some models or systems are designed for narrow NLP tasks and are not of interest for general application in the field, like cardiography texts [41]. Since models are trained on sensitive training data, privacy concerns arise from the fact that potential training data extraction attacks could uncover patient-related data. This concern is amplified by the increasing use of finetuning larger language models that are susceptible to such attacks [9]. In the German domain, the neural German model GERNERMED [14] avoids this issue by using public data from English in combination with neural machine translation to be the first publicly available model with unrestricted access and further improved their method for stronger models [13]. Authors from GGPONC [6] and BRONCO [18] provide access to their own models after registration or signed user agreement. On a broader perspective, the software mEx [31] provides a entire stack of different models and dockerized software layers to serve an integrated text processing system, their models can be obtained on request through signed user agreement [32]. Commercial applications from Health Discovery (Averbis) 2 and SparkNLP (John Snow labs) 3 are available but are purely proprietary applications. Contrary to perceptions of domain experts and reviewers, Amazon Comprehend Medical 4 does not support German texts at the time of writing. Popular, open solutions like Apache cTAKES [34] and MetaMaps [3] do not exist for the German community. Due to the rapid change in the field, we do not consider this list of available models and software as conclusive. We point to [32] and [6] for a more exhaustive enumeration of available models and systems.

Language Model-based Dataset Generation
Data augmentation is a popular technique in the Machine Learning community, in which the objective is to sample new data points from the manifold that models the set of known data points. In computer vision, semantic invariance applies to basic image transformation in many situations [40]. However in NLP such basic techniques cannot always be applied if semantic information of sentences needs to be preserved, but more sophisticated approaches are used such as back-translation [39] of words or phrases through translation, yet failures in translation can jeopardize the augmentation method [4]. The idea to use pretrained language models for data augmentation has been proposed as effective method for augmenting small datasets [2,30] or even create datasets nearly from scratch. [36,37]. With the increasing popularity of large, prompt-based language models like GPT-2/3 [29,8] and open source counterparts [43,5], methods with various objectives have been developed to improve the quality and usefulness of the models in different contexts such as sentence similarity estimation [37]. In addition to classical few-shot text generation, task instruction-driven zero-shot methods are likewise an active field of research [28,25,37]. For medical NLP purposes, text generation has been shown for synthesizing EHR reports [23] and its application for downstream tasks [1] using an GPT-2 model. To the best of our knowledge, we are the first team to expand the general idea to the field of German medical NLP.

Methods
In this work, we leverage the capabilities of pre-trained language models in regards to their example-driven few-shot learning for text generation. The method follows the basic idea implemented in various related contexts [23,2,37]. We apply the GPT NeoX language model from EleutherAI [5] for input processing and text generation. The model implementation is kept close to the GPT-2/3 architecture, an autoregressive model which is closely related to the vanilla Transformer architecture [42] with decoder-only blocks. Note that we do not perform gradient-based fine-tuning of the model on novel data, but the model is only used for inference. In difference to other models like GPT-3 ( [8]), the internal model weights are publicly available similar to its smaller GPT-J [43] model. We decided to use the NeoX model over GPT-J due to its larger size 5 which has been shown to exceed the performance of GPT-J on several tasks [5] yet being sufficiently small to run on our local instance. In addition, large multilingual language models are able to improve task performance on low-resource languages (e.g. German) by the multilingual knowledge transfer from a highresource language (e.g. English) [38].
As previously discussed, LM-based text generation models are used to generate their respective text output by conditioning on an input text sequence, highlighting two main aspects on the input sequence design. First, the sequence can carry a task description in natural language to advise the model on its task objective. While writing an obvious prompt command seems obvious to a normal person, the performance of language models vary between different autoregressive Language Model (GPT NeoX) Output: <s>The <class="animal">cat</class> sat on the mat.</s> <s>Alice likes <class="animal">dogs</class>.</s> <s><class="animal">Pigs</class are smart.<s/> <s>Bob's <class="animal">hamster</class>....

Prompt:
<s>The <class="animal">cat</class> sat on the mat.</s> <s>Alice likes <class="animal">dogs</class>.</s> <s> Input Prompt Generated Text with Annotations Figure 1: Synthesis of markup-based text with annotation information: The input prompt consists of markup-encoded text of few pre-written sentences. The set of sentences is augmented by the language model that generates new data (red) token-wise in an autoregressive fashion.
semantically equivalent task instructions [37]. Second, the input can inject information on the task during the prediction of the next word by providing text examples in the input sequence.
In this work, we do not focus on tuning task instructions in natural language but rather demonstrate that straightforward few-shot-learning-based example prompting as model input suffices for synthetic dataset generation within the scope of our use case. To avoid the issue of only generating plain natural text without valuable annotation metadata, we design our input prompt in the style of a simple markup language, where the language model reads the data as a collection of sentences. Each sentence is enclosed by <s> and </s> signs and separated by a line brake. For each sentence every word from a certain label class l is enclosed by <class="l"> and </class> respectively. We select a small set of exemplary sentences, encode them according to the basic markup rules and append the opening sentence tag <s> to the prompt to indicate the start of an additional sentence. The whole process is illustrated in Figure 1.
In language generation, the unnormalized probabilities over tokens, referred to as logits, are normalized and smoothed by the last softmax layer in the network where n is the number of tokens in the vocabulary, l i is the unnormalized predicted probability for token i. The temperature parameter τ is used for smoothing the normalized probability distribution. In this, higher values of τ increase the probabilities for less probable tokens at the expense of highly probable tokens. We can utilize the parameter to reduce the risk of generating invalid markup-based text data by setting the temperature to 0 < τ < 1.0 in combination with top-p < 1.0 prior to token sampling.
After collecting the output data, we parse the markup text to obtain a synthetic, silver-standard corpus along with its corresponding annotations. For a further data cleansing, we only keep sentences that fulfill the following requirements: First, the sentence needs to have a closing </s> tag. Second, the parsing of the sentence can succeed and the annotations are provided by valid <class="l"> and </class> tags. Third, the sentence has at least one annotation. Fourth, all annotation labels are part of the pre-defined set of label classes. Fifth, duplicate sentences are reduced to unique occurrences (deduplication).
The synthesis of annotated sentences from a large language model and its transfer to a smaller, more efficient model can be considered as a high level form of knowledge distillation: For the very purpose of developing a German NER model for medical entities, we are able to transform the implicit knowledge of the 20B parameter model about this very context into a dedicated NER model with an faster, less resource-intensive computational footprint. In fact, these properties align well to the aim of practical applicability of our method and its resulting model in dedicated domain contexts. For the development of a robust NER model, we train a neural-based NER parser from the open-source SpaCy NLP library on our dataset. While the NER parser component is trained from scratch, its input vectors are generated through an pre-trained BERT-based encoder model to improve the performance of the final model through transferlearning and contextualization. The BERT-based encoder is fine-tuned to the data by gradient update during training procedure.

Results
We provide the GPT-based NeoX model with an input sequence of twelve sentences in German language, encoded in the described markup style. The sentences are pre-annotated with the label classes Medikation (medication/drug), Dosis(dosage/strength) and Diagnose (diagnosis). The prompt is displayed in Figure 2.
During inference we set τ to 0.8 and top-p to 0.9 for language generation and sample 1000 different outputs with a maximum length of 768 tokens each, and additional 100 outputs with an increased temperature τ set to 0.9. Given the parameters, we obtain a raw baseline dataset of 17776 sentences which we reduce to 9845 sentences after the different filters were applied, as shown in Table 1. The final dataset consists of 245107 tokens with annotations for Dosis (# 7547), Medikation (# 9868) and Diagnose (# 5996).
We evaluate the performance of the respective models on precision, recall and f1-score on the testset. The evaluation is computed in strict mode as a character-wise classification task, meaning that exact overlaps and label classes are considered. The results are shown in Table 2. The results indicate strong performance of the models on all label classes, with gbert and GottBert as the models with the best averaged f1 scores. As a significant caveat, while the dataset is split into training, validation and test set and no samples are shared across these sets, the synthesized dataset contains structurally similar sentences that allows the models to potentially overfit implicitly by learning syntax and structure of such homogeneous sentences instead of overfit to certain words directly. The homogeneity could be reduced by various techniques including increasing the temperature τ at the expense of increasing the probability of generating invalid sentences.  Table 2: Results on the test set: The total results are based on the labels' frequency-weighted average. The label annotations are evaluated characterwise.
We further evaluate the models on a small gold-standard German dataset proposed in [13] as an out-of-distribution (OoD) dataset. Since the dataset contains label annotations largely compatible to the n2c2 2018 ADE dataset [16], we cannot directly compare all label classes, yet in the interest of an OoD performance evaluation, we assume that the label class Drug shares significant semantic overlap with the label class Medikation. The results are provided in Table 3. Beyond the expected drop in terms of the Medikation scores across all models, gbert and GottBERT are identified as the models with the best f1 scores, with GottBERT surpassing gbert by 2.6% in f1 score (test set reference: -0,6%).  Table 3: Results on the out-of-distribution dataset: As caveat, the label definitions of Medikation (ours) and Drug(from the 2018 n2c2 ADE dataset [16]) is inaccurately assumed to be equivalent for comparison. The label annotations are evaluated character-wise.

Discussion
We demonstrate the effectiveness of our method for utilizing pre-trained language models for dataset synthesis by training a neural NER model on this dataset, yet the limited availability of annotated German medical NLP datasets with ill-defined or even dissimilar label classes remains a major obstacle when it comes to a more exhaustive, yet reliable evaluation of the trained NER model for all label classes. Given the evaluation scores on the Drug/Medikation labels it must be considered that our method achieves these results based on twelve initial sentences. Aside from the evaluation, we did not further perform hyperparameter search for dataset synthesis on parameters like temperature τ or top-k/top-p sampling or beam search due to the high computational costs of running the NeoX model as well as due to limited access to GPU resources. Even though the initial need for computational resources is a major downside of our method, we believe that this factor becomes negligible with respect to the fact that the method can operate without input from costly human annotators. For very domain-specific contexts, such as German medical texts, this not only provides an opportunity to work on NLP approaches independent of external monopoly-like data sources and medical institution that also constitute a severe asymmetry in academic competition. Yet it also allows the further use of the dataset without additional efforts in pseudonymization and legal ramifications that are usually unavoidable when working with datasets originating from real patient data. Therefore, we are able to provide the synthesized corpus and the trained models to third party use publicly without further access restrictions. While our NER model exhibits strong performance in general and proves the dataset to comprise useful and valid data for text and corresponding annotation, the dataset remains synthetic in nature and thus cannot be considered as an gold standard-level dataset. The question to which degree the corpus carries additional domain knowledge remains open for future work.

Conclusion
In this work, we leveraged the few-shot ability of the pre-trained language model GPT NeoX to generate an annotated dataset for German medical texts without the need of manual annotations by introducing few annotated text samples to the language model in a simple markup format. We further used the dataset to train NER models by fine-tuning three pre-trained BERT encoder models. Our evaluation on testset as well as OoD set indicates a robust performance of the NER models even for shifts in the dataset. We discussed the disadvantages and advantages of our method as well as its potential implications for the German medical NLP research community and beyond. The corpus and the trained models are publicly available on GitHub at: https://github.com/frankkramer-lab/GPTNERMED