{'fraction_non_alphanumeric': 0.02072155411655874, 'fraction_numerical': 0.00547641073080481, 'mean_word_length': 5.387615220987946, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 0, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 1, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Validation and Transparency in AI systems for pharmacovigilance: a case study applied to the medical literature monitoring of adverse events

Recent advances in artificial intelligence applied to biomedical text are opening exciting opportunities for improving pharmacovigilance activities currently burdened by the ever growing volumes of real world data. To fully realize these opportunities, existing regulatory guidance and industry best practices should be taken into consideration in order to increase the overall trustworthiness of the system and enable broader adoption. In this paper we present a case study on how to operationalize existing guidance for validated AI systems in pharmacovigilance focusing on the specific task of medical literature monitoring (MLM) of adverse events from the scientific literature. We describe an AI system designed with the goal of reducing effort in MLM activities built in close collaboration with subject matter experts and considering guidance for validated systems in pharmacovigilance and AI transparency. In particular we make use of public disclosures as a useful risk control measure to mitigate system misuse and earn user trust. In addition we present experimental results showing the system can significantly remove screening effort while maintaining high levels of recall (filtering 55% of irrelevant articles on average, for a target recall of 0.99 on suspected adverse articles) and provide a robust method for tuning the desired recall to suit a particular risk profile.


Introduction
Medical literature monitoring (MLM) of adverse drug reactions and special situations is the task of periodically searching the scientific literature for potential safety events associated with a product of interest with the goal of reporting findings to regulatory authorities for appropriate action, or gaining insights on a product's safety and efficacy profiles. Periodic MLM is a regulatory requirement for marketed products in many jurisdictions (ex: [1,2]) and also performed by regulators, the European Medicines Agency being an example [13]. This activity is typically performed by a team of experts in a pharmacovigilance team with responsibility for a marketed product.
A typical MLM workflow is shown on Figure 1: a pharmacovigilance specialist will craft a database search based on query strings targeting the product of interest and relevant terms indicative of adverse events or special situations (off-label use, paediatric, pregnant or elderly populations, etc). This search is then run on a number of scientific databases, results are extracted, de-duplicated and undergo initial screening by a specialist based on the title and abstract of the article. Should the article be deemed relevant the process continues by carrying out a deeper inspection based on the full text of the article and subsequent quality control. Screening of abstracts will vary according to the task objectives: for periodic surveillance of individual case safety reports (ICSR) specialists look for safety events in articles where identifiable patients can be found. Other screening tasks may look for aggregated safety data from patient studies or target the discovery of possible new safety signals. What is common however is the activity will demand more resources as the input grows in volume. Indeed, this volume continues to grow with more publications being available and more products entering the market every year, increasing the demand for specialist resources in pharmacovigilance teams. At the same time, recent surveys indicate pharmacovigilance workflows still lag in automation adoption [3] presenting an ideal opportunity to consider Artificial Intelligence (AI) solutions to deliver productivity gains and higher quality results.
Reaching this goal requires technology development coupled with adequate process maturity that ensures validated and trustworthy AI systems, attracting interest from regulators and industry bodies who have put forward guidance in developing AI systems for pharmacovigilance [6,7,19]. Leveraging these developments, we describe an AI system for medical literature monitoring of the scientific literature and corresponding development process serving as a case study operationalizing current guidance. In addition we present experimental results demonstrating how the system can realize productivity gains in MLM workflows, and discuss lessons learned and future development opportunities.

Considerations and Guidance for AI Systems in Pharmacovigilance
Software engineering practices for pharmacovigilance typically follow established computer systems validation (CSV) guidance in the International Society for Pharmaceutical Engineering's Good Automated Manufacturing Practices (ISPE GAMP [5]). To address specific considerations of AI software systems, the guidance proposed by TransCelerate [6], extends CSV leveraging existing regulatory guidance and best practices, in particular FDA's seminal work on AI/ML software as a medical device [7]. Despite the TransCelerate guidance being particularly well suited for AI in pharmacovigilance, this remains an active area where regulatory agencies and industry bodies continue to seek feedback and evolve guidance on the use of AI. We refer readers to a recent survey of ongoing activities on this topic in [19].
We also observe considerations from the AI transparency literature may contribute to the same goals of guidance for AI in pharmacovigilance: from [6], clearly stating a model's intended use is an important requirement for validated systems and expected to be part of a system design specification, while the public disclosure of an AI system's intended use is encouraged as a transparency measure that makes clear the system applicability and trade-offs, reducing the risk of misuse [14]. AI disclosure considerations and proposals can be seen in the work on model FactSheets [16], Model Cards [20], Datasheets for Datasets [21] and the AboutML initiative [14].

AI Development Approach
Our AI development approach has the following goals: (a) incorporate existing best practices in computer systems validation and supporting processes for change control, risk management, etc. (b) ensure decisions at important stages of the development process are traceable and can be later interrogated, (c) consider the existing guidance for developing AI systems for pharmacovigilance, notably considerations presented in [6] and finally (d) leverage AI transparency best practices to mitigate risk and improve trustworthiness of the system.
The processes supporting the lifecycle of the AI system are heavily anchored in artifacts maintained in a quality management system (QMS), where documentation changes are fully traceable and undergo established approval processes. This ensures decisions relating to the system design, data sourcing and labelling, model training and testing have a full audit record. Using a QMS also facilitates the integration and reuse of supporting processes for CSV such as change control, risk and documentation management.
The backbone of our process is described in the AI development lifecycle document: a multistage development process that delivers milestones iteratively through planned releases. Each stage is common in scope to traditional software development, but includes considerations and best practices specific to AI development. Table 1 describes process stages and included deliverables of particular interest to AI.
Deriving directly from existing documentation in the QMS, we include public disclosures as part of our release process: technical details of the AI system and models are documented in a factsheet 1 derived from the one proposed in [16]. Corresponding user-facing product documentation is also updated as appropriate, as illustrated in Figure 2. In addition to documentation the code, data and models must also follow standardized control mechanisms that ensure an auditable record of changes. A combination of git [8] and the open source Data Version Control tool (DVC) [9] provided a suitable toolkit for ensuring there is an auditable record of changes across these artifacts. Git is a widely used distributed version control system providing a permanent change record of code and other artifacts in its repository; DVC follows a "git-like" workflow to accomplish similar goals for data and model artifacts with additional features specific for data science workflows such as running repeatable experiments. By not deviating significantly from typical software engineering workflows this approach makes adoption easier and leverages existing tooling and best practices built by the developer community. Table 1 outlines traceability controls in place across the development lifecycle stages: Changes undergo traceable review and approval process via pull requests.

Testing
Unit testing is automated and integrated into the code review process.
Automated model testing reports (integration testing) produced with every release; results are version controlled.
Functional testing for computer system validation includes AI-specific features, artifacts are tracked as controlled documents in the QMS. (Key deliverables: user and functional requirements matrix, installation qualification, operational qualification and validation summary report)

Release
Releases follow change management process and are traceable via controlled documents; Code releases undergo review and approval process from GitHub.

Monitoring
Periodic monitoring is carried out via functional testing and benchmarking experiments. Results are tracked as controlled documentation in the QMS.

System Description
The system acts as an additional stage to the original MLM workflow, shown in Figure 3, where model predictions filter results retrieved from searching the literature according to whether these may contain relevant safety information. Using the title and abstracts of an article as input the system issues a prediction on whether the article contains one or more suspect adverse event. Suspected adverse articles proceed for further screening. To ensure compatibility with as many data sources as possible, no additional article metadata is used by the system other than the text of the title and abstract.

Inference Pipeline
The inference pipeline comprises a pre-processing stage to clean and tokenize the input, detect abstract language and perform entity extraction of patient mentions used in later stages. The model inference stage encodes the normalized text into features and runs the prediction step of the machine learning models, producing raw model predictions. Next, a post-processing rulesbased stage produces the final predictions and an explanation step computes metadata to help users interpret model predictions. The design of the pipeline is modular so that each stage can produce additional outputs to be consumed downstream. For instance, additional machine learning models and entity extractors can be added, and later combined at the rule-base stage. The current implementation uses two machine learning models at the model inference stage: The main adverse detection model is a refinement of our previous work [4] and employs a multilayer neural network (NN) architecture organized as follows: an initial embedding layer converts tokens into vector representations using a combination of pre-trained word embeddings built with a biomedical text corpus [10] and additional trainable embedding layers derived from partof-speech tags and dependency parsing tags. The embeddings are combined and processed by a series of convolutional layers followed by a LSTM recurrent layer [17] and an attention layer. Regularization is applied across the network architecture by using drop out during training and the use of batch normalization layers. Complementing the NN model a second, separate model uses bag-of-words (BOW) representation as features from 1-gram and 2-grams and is trained with a random forest estimator.
The rule-based stage of the pipeline takes the output of all the previous stages as input, and computes predictions by evaluating a set of pre-defined rules created in collaboration with pharmacovigilance experts. This approach is used to override model predictions when SMEs deem it is the safer option, or to ensure predictions are only made within a pre-defined operating envelope. In the case when rules override machine learning predictions, the execution log provides an additional layer of explainability to results. Figure 5 outlines the rules implementation for issuing a suspect adverse prediction: input attributes extracted during preprocessing are verified against the model's operating envelope. Currently this comprises of verifying the inferred language of title and abstract is English, the input size is within boundaries seen during training, or input is of a known invalid format (ex: abstract errata descriptions).
The prediction scores from the machine learning models are then checked against thresholds configured as rules. If a prediction using the NN model is not adverse, it is then checked against the BOW model at a threshold level deemed of high confidence. Finally, the system checks if an identifiable patient was found by the entity extraction pre-processing stage (ex: "46-year-old male patient"). This rule reflects pharmacovigilance experts advice in that identifiable patients appearing on case report or case series articles should undergo a closer inspection. At the explanations stage we employ a variation of the LIME technique [11] to extract sections from the abstract most relevant to the predicted class. LIME is a model-agnostic explainability technique that generates predictions in the vicinity of the input by randomly applying small modifications to the input text. Explanations to the model prediction can be derived from the modifications that affected the predicted label most significantly. From experimentation we find the technique is best applied in our case when using entire sentences as the unit of modification as opposed to individual tokens.

Data Labelling
The labelling approach must take into account the fact that abstracts may contain only partial safety information: it is not always possible to fully determine the details of an adverse event from the abstract or title, and further inspection in the article full text may be needed. For the same reason a suspect event may apply to any product being mentioned in the text -ie. it may not be possible to establish causality. Minimizing the risk of filtering relevant articles is also an important consideration, and finally the model should be broadly applicable to other related MLM activities introduced in Section 1.
Training data was curated and labelled in advance by pharmacovigilance specialists and does not dynamically change with user input, thus categorizing the resulting models as "static". Static systems are desirable from a validation standpoint as they align with guidance from [6] and can better follow testing and change management practices for computer systems validation.
The machine learning task is a binary text classification problem where the prediction identifies an abstract as suspect adverse or not. For data labelling this is defined as an abstract that either contains explicit mentions of an event or implicit mentions that may be fully described in the full text of the article. This definition allows for building classifiers that can be used in many scenarios that need the detection of safety events while at the same time minimizing the risk of false negatives.
The dataset is built from a selection of publicly available abstracts from the biomedical literature. Articles were iteratively selected using pre-defined keyword searches that (1) reflected compounds belonging on one or more categories for therapeutic use from the MeSH taxonomy [12], or (2) were part of the European Medicines Agency list of products under surveillance for medical literature monitoring [13] or (3) were retrieved using commonly used adverse terms or terms for special situations (paediatric, off-label, pregnancy-related terms, etc.). The data labelling and training objectives are recorded in the QMS and form part of a continuous process of model re-training and release. The entire labelling process is the responsibility of pharmacovigilance specialists, and all labelled data undergoes sampled quality checks by a second annotator to ensure labels remain consistent. Inter-annotator agreement is monitored using Coen's Kappa metric, and additional reviews are triggered should it fall below an agreed threshold. "Twenty-two patients experienced peripheral neuropathy, and two had severe neuropathy".
(event may be associated to treatment) "Although dose reduction was required in 20% of the patients, no adverse event that led to the discontinuation of treatment was observed." (more ambiguous, implies adverse event led to dose reduction)

Performance Metric
Mistakes in the prediction of safety events have an asymmetric risk profile: articles falsely identified as suspect adverse (false positive) would incur incremental screening effort, but articles falsely identified as not suspect adverse (false negative) negatively impact what safety information is detected. In deciding the performance metric to optimize, the least risky approach is one that minimizes false negatives, even at the expense of additional effort.
The adverse model is parametrized for a desired target recall, ensuring false negatives remain low: with recall fixed at a sufficiently high level, a metric that reflects the additional effort caused by false positives should be minimized. We use the false positive rate (fpr), defined as the ratio of false positives (FP) to the number of ground truth negative examples (N) given by: Where TN is the number of true negatives. Thus, the performance target is the minimization of false positive rate at a desired target recall.

Risk and Controls
Framing current guidance considerations as risks addressed by one or more controls provided a useful mechanism to translate guidance into action items during the development life cycle. This approach is summarized for our system in Table 3. We note that in many cases public disclosures can be used as an effective risk mitigation tool. Intended use The AI system is applied outside of its intended use and operating envelope. [6,14] Inference pipeline verifies input within operating envelope (rules stage).
Publicly disclose intended use in the system's fact sheet.

Intended use
User over-confidence on predictions from the AI system, performance on data not seen in training is not available. [6,18] Publicly disclose performance metrics and justification in system fact sheet.
System supports different levels of supervision to facilitate validation and adoption.

Intended use
Full automation derived from model predictions may hide incorrectly predicted documents from users. [6] System supports diverse levels of supervision and validation, and does not require mandatory full automation.
All historical results are auditable irrespective of prediction result.

Data
Inconsistency in labelling training data may lead to poor performance. [6] Data labelling performed by pharmacovigilance specialists with MLM background; quality checked and monitored for inter-annotator agreement.

Data
Data used in training is not representative for the intended use, or is biased towards certain scenarios, leading to unexpected results. [6] Training data composition, intended use and domains not covered in the training data are disclosed in the system fact sheet.

Data
Input data is not in expected format and may generate spurious predictions. [18] Inference pipeline performs check of input data for language and operating envelope.
AI system is integrated into data pipeline where ingested data is from verified sources.

Data
Poor quality of training data may affect model performance. [6] All training data collected is labelled by pharmacovigilance specialists.
Data labelling process includes quality check and verification of inter-annotator agreement.

Model
Model over fitting [6,18] NN model trained with regularization and supplemented by additional BOW model.
Model performance is reported against a left-out test set, and an independent dataset from another distribution.

Model
Data leakage into training set may lead to misleading performance data. [6,18] Integration testing includes validation against data leakage.

Model
Model drift: age/relevancy of training data. [6,18] Periodic model releases are envisaged with corresponding updates to public disclosure.
Training data is regularly updated following AI lifecycle and change management process.

Model
Robustness to adversarial attacks. [6] The model is currently operating on input data sources vetted by the engineering team.

Transparency
Model predictions are not easily understood, and less trustworthy to users [6] Model predictions are accompanied by explanatory statements derived from the rule-based stage; using LIME technique for model explanations.

Transparency
Model design and validation decisions are not visible or clearly justified, becoming less trustworthy to users [6,14,18] Model design and validation decisions are publicly disclosed in the system fact sheet; In-depth documentation is tracked in the QMS and auditable upon request.

Experimental Results
The training set is built by randomly sampling 90% of all available labels (n=20,531). Sampling is stratified across the labelling categories used to select data discussed in Section 4; the minority label is randomly up-sampled to produce a balanced number of samples for each label.
The remaining 10% of the original dataset is split evenly into a validation set and a test set (each with n=1141). The training set is used to train the core machine learning models, while the validation set is used for hyper parameter tuning. The test set is set aside solely to report model performance results.
Once the BOW and NN models have been trained, a hyper parameter tuning step employs grid search to determine suitable values for model thresholds ( Figure 5) that minimize the false positive rate on the adverse class for a target recall. The experiment is repeated 10 times by randomly re-sampling the test/validation sets. Figure 6 shows the trend for the desired target recall (x-axis) vs. obtained recall on test set (y-axis) using hyper parameter search for NN and BOW as standalone models. The box plot shows inter-quartile ranges across all 10 x experiment runs, with minimum and maximum values extending from the box.   In Figure 8 we compare results of NN and BOW against the strategy of combining the machine learning models with an extra rule based step (NN+BOW+Rule) from the inference pipeline. Results are shown for the 0.9-0.99 target recall range. In Table 4 we present average false positive rates from the 10 experiments.

Discussion
Results indicate our approach can yield productivity gains for high volume MLM workflows while maintaining high levels of recall. Taking as an example the results for a target recall of 0.99 from Table 4, the system would operate with an average of 45% false positive rate (45% of non adverse abstracts predicted as suspected adverse), thus correctly predicting 55% of non-adverse articles which could then be automatically filtered or undergo less time consuming forms of screening.
When comparing machine learning models, the NN approach provided improvements over a benchmark BOW model, but these improvements became less significant at higher levels for target recall (target recall = 0.99). Combining the BOW and NN at rule-base stage provided more robust results across all values tested except 0.99, suggesting further opportunities to experiment with combining other machine learning techniques and model ensembles.
Using hyper parameter searching to find suitable threshold values for a target recall has approximated to reasonable obtained recall values in the test set as seen in Figure 6, indicating it is a viable method for tuning the system to the particular level of risk (when lower recalls are tolerated, more productivity gains can be had). This method reflects similar studies of AI in pharmacovigilance [15] and can help support the validation needs of different applications.
With respect to risk management and controls identified in this study, we observe that public disclosures play an important part in risk mitigation: they contribute to informed decision making by end users by clearly stating the AI system's intended use, operating envelope, architecture and performance metrics. Another observation is that controls can be applied based on how the AI system is operationalized to end users -ie. the overall system it is embedded in. For instance, an interface that enables different levels of human supervision can support the progressive adoption of AI automation (requiring progressively less supervision) to suit a user's desired strategy.
In our experience the continuous involvement of pharmacovigilance experts is a critical success factor: this collaboration provided tangible benefits implemented as risk control measures across the system. Examples include refinements to the system's the intended use, data labelling and the rule-base stage in the inference pipeline. Our experience resonates with establishing multistakeholder development processes to improve transparency [14]. Furthermore, AI systems are often operationalized as subsystems within an end user application, as is our case 2 . Here, subject matter expertise can too provide valuable feedback on the application user experience to minimize misuse risks from AI automation.

Conclusions
In this study we presented an AI-based system for improving productivity in medical literature monitoring workflows by filtering irrelevant articles from human screeners with high levels of recall. The system is designed and built taking into particular consideration requirements and constraints from existing guidance for validated AI systems in pharmacovigilance and best practices from the AI transparency literature. We believe critically evaluating AI systems that operationalize emergent industry guidance with concrete implementations are a valuable step in driving adoption of AI technologies for pharmacovigilance and hope our study provides a useful case study to compare and contrast real world implementations of existing guidance.
We aim to further develop the system's inference pipeline by experimenting with recent neural architectures that have provided improvements on the state of the art on a number of NLP tasks, and by using more sophisticated robustness testing for input validation. We also wish to refine our public disclosures based on proposed guidance from [20,21] extending data disclosures and giving greater consideration to model bias and fairness evaluation.