{'fraction_non_alphanumeric': 0.022052621008234585, 'fraction_numerical': 0.008555934926692106, 'mean_word_length': 4.874469089193015, 'pattern_counts': {'":': 0, '<': 2, '<?xml version=': 0, '>': 0, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Data Augmentation for Personal Knowledge Graph Population

A personal knowledge graph comprising people as nodes, their personal data as node attributes, and their relationships as edges has a number of applications in de-identification, master data management, and fraud prevention. While artificial neural networks have led to significant improvements in different tasks in cold start knowledge graph population, the overall F1 of the system remains quite low. This problem is more acute in personal knowledge graph population which presents additional challenges with regard to data protection, fairness and privacy. In this work, we present a system that uses rule based annotators to augment training data for neural models, and for slot filling to increase the diversity of the populated knowledge graph. We also propose a representative set sampling method to use the populated knowledge graph data for downstream applications. We introduce new resources and discuss our results.


INTRODUCTION
Cold start knowledge graph population is the problem of populating a knowledge graph from unstructured documents. It involves tasks like Entity Recognition, Entity Classification, Entity Resolution, Relation Extraction, and Slot Filling.
The NIST TAC Knowledge Base Population (KBP) challenges remain a popular example of the problem. As [23] showed, the problem remains largely unsolved, with the F1 of the overall relation and slot filling system achieving only 26.7% F1, which is not sufficient for real world applications. They also introduced a new TACRED dataset for the relation extraction task, which addressed some of the requirements like provenance of the relations, annotations for negative samples, larger number of samples (106k) and longer average sentence length (36. 4).
For a number of applications in Data Protection, fraud prevention and business intelligence, there is a need to extract Personal Data Entities, classify them at a fine grained level, and identify relationships between people. Manually created Ontologies and Knowledge Graphs are typically used for this purpose.
Cold start becomes more challenging in Personal Knowledge Base population where the entities are people and they are related to other people in the real world. Constructing such a graph from unstructured documents like emails in an organization, government data of citizens, personal conversations etc is a significantly harder problem than populating general purpose world knowledge graphs.
Many of the resources available for populating such a graph with distant supervision are unavailable for personal KBP. This is not because description of the person entities are unavailable (they may have biography pages, resumes and corporate profiles), but entity linking to world knowledge graphs will have low recall. We use the terms Knowledge Base and Knowledge Graph interchangeably in this work.
The first challenge in personal KBP is in identifying attributes at a fine grained level. In Figure 1, Brigham Young University could be classified coarsely as ORGANISATION by a Named Entity Recognizer. In recent years, a number of Neural Fine Grained Entity Classification (NFGEC) models have been proposed, which assign fine grained labels to entities based on context. They could type Brigham Young University as /org/education. However the focus of such systems has not been on PDEs. They do not treat the problem of identifying PDEs any different from other entities. In our example, we want the model to assign educated_at as the class for Brigham Young University.
In typical relation extraction tasks, a person and their place of birth could be considered a relation. However we treat personal knowledge graph as a property graph and want to have only people as nodes and relations between people as edges. Other entities like place of birth could be considered attributes of the Person node.

Privacy, fairness, and Data Protection
Considering people knowledge graphs need to have sufficient personal information to be of any practical use, they also have to protect the privacy of the people concerned. So such Knowledge Base population typically happens in-house and remains a largely manual process till now.
Even within an organization, populating personal knowledge graphs can only happen with the informed consent of the employees. Customer data of companies cannot be used for populating such graphs because of privacy reasons and also because Data Protection laws like GDPR forbid such usage of data without the consent of the customers. Although, customers of a company may also benefit from such a personal knowledge graph, to serve Data Subject Access Requests (DSAR) for example, obtaining informed consent from customers will be very hard.
Further, while training models and algorithms to populate a personal knowledge graph, it is desirable to use synthetic data rather than actual data of the organisation. Hence in this work we propose a way to extract personal knowledge graph from the publicly available wikipedia, wikidata, and web crawl data of elected representatives.
Even while populating such a knowledge graph from synthetic data, we have to strive to avoid bias in the training data on gender, age, ethnicity, location, religion, sexual orientation among other attributes. While there are methods available to detect bias in models, eliminating bias in the training data can have more desirable outcomes. Business Intelligence and other applications will also benefit from eliminating or reduce bias in the data. Hence in this work, we present ways to augment the training data of neural models used for knowledge graph population, as well as for using the graph as a dataset for downstream applications.
We summarize our contributions in this work as follows: • We present a system to populate a personal knowledge graph with 20116 nodes, 36 attribute types and 10 relation types. This graph can be used a dataset for downstream tasks. • We share our results from augmenting the training data with rule based systems on the Entity Classification task as well the overall KBP evaluation task.
• Finally, we present a representative set sampling method that can be used to sample the dataset while maintaining the diversity achieved by our augmentation. [2] introduced the concept of Personal Knowledge Graphs. A typical pipeline to populate a knowledge graph comprises of Entity Recognition, Entity Classification, Entity Resolution, Slot Filling, Relation Extraction and Link Prediction. [13] introduced personal knowledge graphs in the context of conversational systems. In this paper, we describe a system to extract personal knowledge graph from the Wikipedia pages of elected representatives and people related to them. We treat this as an acceptable proxy for real world enterprise documents like emails, internal wiki pages, organization charts which cannot be used for research purposes. Entity classification is a well known research problem in Natural Language Processing (NLP). [15] proposed the FIGER system for fine grained entity recognition. [21] showed the relevance of handcrafted features for entity classification. [19] further showed that entity classification performance varies significantly based on the input dataset (more than usually expected in other NLP tasks). [5] and [1] introduced model improvements to achieve better results on OntoNotes and Wiki datasets respectively. [7] proposed a neural architecture to populate a Person Ontology.

RELATED WORK
[3] introduced a AI Fairness toolkit called AIF360 which provides a number of algorithms for detecting and mitigating bias against protected attributes like gender, age, ethnicity, location, sexual orientation and few others. [9] proposed a method using counter factual data to improve fairness. [15] proposed the FIGER entity type hierarchy with 112 types. [10] proposed the Google Fine Type (GFT) hierarchy and annotated 12,017 entity mentions with a total of 89 types from their label set. These two hierarchies are general purpose labels covering a wide variety of domains. [5] proposed a larger set of Personal Data Entity Types with 134 entity types. We have selected the 34 personal data entity types, as shown in Figure 2 that were found in our input corpus.

Personal Data Annotators
Any system that assigns a label to a span of text can be called an annotator. In our case, these annotators assign an entity type to every entity mention. We have experimented with an enterprise (rule/pattern based) annotation system called SystemT introduced by [4].
SystemT provides about 25 labels, which are predominantly coarse grained labels.
We use these personal data annotators in 3 ways: • To annotate the dataset with entities for the entity classification task.
• As part of the Personal Data Classification pipeline, where for some of the classes, the output of these PDAs are directly used as entity types. These are types like email address, zip codes, number where rule-based systems provide coarse labels at high precision. • To create a series of labeling functions that annotate relations between entities. These relations are used to bootstrap link prediction models, which in turn populate the Ontology Graph. While neural networks have recently improved the performance of entity classification on general entity mentions, pattern matching and dictionary based systems continue to be used for identifying personal data entities in the industry. We believe our proposed approach, consisting of modifications to state-of-the-art neural networks, will work on personal datasets for two reasons. [21] showed that hand-crafted features help, and [19] have shown that performance varies based on training data domain. We have incorporated these observations into our model, by using coarse types from rule-based annotators as side information.
We used our Personal Data Annotators to create a number of labeling functions like those shown below to create a set of relations between the entities. We have created this dataset from the Wikipedia page of US House of Representatives and the Members of the European Parliament. We obtained the names of 1196 elected representatives from the listings of these legislatures. These listings provide the names of the elected representatives and other details like contact information. However this semi-structured data by itself cannot be used for training a neural model on unstructured data.
Hence, we first obtained the Wikipedia pages of elected representatives. We then used Stanford OpenNLP to split the text into sentences and tokenize the sentences. We ran the Personal Data Annotators on these sentences, providing the bulk of the annotations that are reported in Table 1. We then manually annotated about 300 entity mentions which require fine grained types like /profession. The semi-structured data obtained from the legislatures had name, date of birth, and other entity mentions. We needed a method to find these entity mentions in the wikipedia text, and assign their column names or manual label as PDEs.
We used the method described in [4] to identify the span of the above entity mentions in wikipedia pages. This method requires creation of dictionaries each named after the entity type, and populated with entity mentions. This approach does not take the context of the entity mentions while assigning labels and hence the data is somewhat noisy. However, labels for name, email address, location, website do not suffer much from the lack of context and hence we went ahead and annotated them.

EXPERIMENTS 4.1 Entity Classification
We use the neural network model from [5], which consists of an encoder for the left and right contexts of the entity mention, another encoder for the entity mention itself, and a logistic regression classifier working on the features from the aforementioned encoders.
The above model improves on the work by [19]. The major drawback in that model was the use of custom hand crafted features, tailored for the specific task, which makes generalization and transferability to other datasets and similar tasks difficult. Building on these ideas, we have attempted to augment neural network based models with low level linguistic features which are obtained cheaply to push overall performance. Below, we elaborate on some of the architectural tweaks we attempt on the base model. Similar to [19], we use two separate encoders for the entity mention and the left and right contexts. For the entity mention, we resort to using the average of the word embeddings for each word. For the left and right contexts, we employ the three different encoders mentioned in [19], viz.
The results on Elected Reps dataset as can be seen in Table 2, clearly show the same trend, i.e. adding token level features improve performance across the board, for all metrics, as well as for any choice of encoder. The important thing to note is that these token level features can be obtained cheaply, using off-the-shelf NLP tools to deliver linguistic features such as POS tags, or using existing rule based systems to deliver task or domain specific type tags. This is in contrast to previous work such as [15], [21] and others, who resort to carefully hand crafted features.

Relation Extraction
While extracting entities from unstructured text can be improved as we have shown in 2, extracting people to people relations is a harder problem. The state of the art methods like TACRED perform reasonably well, relations are often not mentioned in text. The UDBMS dataset based on the person data in DBPedia has over 500k nodes and only 28k edges.
In enterprise data like emails, controls, policies too, the entities are present far more than the relations between people. Hence we to augment the knowledge base with relations extracted from other sources. In our wikipedia based elected representatives dataset, we follow [11] approach of using wikidata as a source of relations. In addition, we use the Wikipedia API to hyperlinks between persons in our dataset. However, we restrict this to only people who are mutually linked. We leave the task of assigning a type to the mutual wikipedia links between people as future work.
A similar approach can be adopted in enterprise datasets, using email sender, receiver information, organization charts, and newsletters to extract and type relations between people in the personal knowledge graph.

Dataset
Model Accuracy TACRED TACRED 56.84 Table 3: Relation extraction performance obtained by us using TACRED model

REPRESENTATIVE SET SAMPLING
In the process of generating the knowledge graph, we might want to manually inspect the generated entities to have an understanding of what kind of entities are generated and how good are the attributes associated with them. Inspecting all the entities is a tiresome job and hence we might want to inspect a small subset of the entities.
To get meaningful insights about the dataset, we need the samples that we select to be representatives of the entire dataset. One simple approach to sampling could be random sampling where we select samples at random. There can be other deterministic approaches like select every k t h entity in the dataset and inspect them etc. But, there are some issues associated with the simple approaches above.
• Redundancy : If we assume that the entities are generated form a probability distribution P, then it is more likely that random sampling would yield samples with patterns around the mean/mode etc. Hence the samples in the subset would mostly convey redundant information. • Coverage : Because random sampling yields samples with patterns around the mean/mode, we are more likely to ignore data patterns with lesser probabilities in the subset. Hence to have a better coverage, we would want the samples to be representatives of most of the patterns in the dataset.
To circumvent the issues above, we have developed a sampling algorithm that employs some heuristics to reduce the redundancy and increase coverage of patterns among the samples that are selected. The desiderata for the sampling algorithm are as follows : • Select at least one sample which represents the most frequent pattern in the dataset. This ensures coverage of the dataset. • No two patterns selected in the subset should be similar by more than a threshold θ . This avoids redundant patterns in the subset.
Our sampling algorithm is inspired from matrix sketching [14], which is a linear time algorithm to find the most frequent patterns in the dataset. More formally, given a matrix A ∈ R n×m , the algorithm finds a smaller sketch matrix B ∈ R l ×m such that l << n and A T A ≈ B T B. Here we can observe that the matrix B tries to capture most of the variance in B. In other words, each row of B represents a frequent direction in A and also because B is obtained by performing SVD on rows of A, each row of B is orthogonal to other rows of B. Our intuition is that, once we get the frequent directions of A, we can easily select data points along that direction and thereby select samples representing the frequent patterns in the dataset. The sampling algorithm expects the input to be in numerical form only. We convert each categorical attribute to one hot embedding and normalize each numerical column to be between [0, 1] and feed it as input to the algorithm. We drop other text attributes. Hence, input to the sampling algorithm is a matrix A that is scaled for numerical attributes and one-hot embedded for categorical attributes respectively. Now, we explain the sampling algorithm in several steps.
(1) Dimensionality reduction : Going forward in the algorithm, we would apply matrix sketching on the input matrix A, which would in turn apply SVD. Hence to keep the eigen problem tractable, we do feature agglomeration to reduce the dimensionality of A. In our experiments we retained only the top 100 features of A. (2) Clustering : One of the popular methods in literature to select representatve points is to partition the dataset into many clusters and select representatives from each of them. We also adapt the same paradigm. We first cluster the datWset and apply sampling algorithm in each cluster independently. Thereby, we select representative points from each cluster.
In our experiments, we run KMeans algorithm with number of clusters = k, where k is determined by the elbow method. (3) Select data points representing frequent patterns : For each cluster i ∈ [k], we select the cluster matrix A i , where A i contains the rows of the entities that belong to cluster i. We then apply matrix sketching algorithm on A i to find a set of frequent directions {v j } in A i . Then, for each frequent direction v j , we select a data point a j ∈ A i along that direction and include it in the representative subset.
(4) Remove redundant points : Once we select points along the frequent directions, we remove points that are redundant to the selected points. For each a j selected in step-3, we collect all the redundant points in A i as In our experiments, we set θ = 0.85. Once we collect the redundant points R j , we remove them from the cluster. i.e.
We repeat Steps 3 and 4 above until we exhaust the data points in the cluster A i . Intuitively, it is easy to observe that step 3 selects points representing the frequent points in the dataset and step 4 avoids selecting points representing redundant patterns in the representative subset.
The results of the sampling algorithm is shown in Figure 4. The dataset has 15762 males, 3901 females and 453 others and the representative subset sampled by the algorithm has 20 males, 13 females and 1 others. We have implemented a pipeline for Personal Knowledge Graph population as shown in Figure 5. This pipeline consists of existing personal data annotators, Stanford Named Entity Recognizer which provide rule based entity and relation extraction. We have then improved two state of the art models for entity classification and relation extraction as described in the previous sections. Finally we use a graph neural network for Link Prediction to infer more relationships between people mentioned in the corpus. The use of a GNN for Link Prediction is to leverage the attributes of the nodes, along with neighboring nodes and edges. We use an entity resolution system similar to the SystemER introduced by [17].
The input to our pipeline are text sentences. The outputs are person entities, their personal data as attributes and semantically rich relations between person entities. These can be used to populate a graph database like the one provided by networkx [12]. We present the results from training two graph neural networks on the Personal Data Entity (PDE) data extracted using our method and a similar DBPedia data which has been annotated by wikipedia users.

Dataset
Model  Table 4, Position Aware Graph Neural Network [22] performs much better than Graph Convolutional Networks on both the UDBMS dataset which is hand curated by wikipedia contributors and the Elected Representatives data extracted by us. Figure 6 and Figure 7 shows the distribution of node attributes and relation types in the populated graph.
The Personal Knowledge Graph populated by us can be used to improve search, natural language based question answering, and reasoning systems. Further the graph data can be exported to other data formats like the RDF and PPI formats, and used as a dataset for Link Prediction experiments.

COLD START KBP EVALUATION
We evaluate the slot filling performance of the models by using a similarly generated evaluation procedure as of the TAC KBP 2015 cold start slot filling task by considering the personal knowledge graph [6]. The entities for hop0 and hop1 are manually selected using random search for particular persons present in the corpus data. Corresponding queries are generated for hop0. The predicted relation data for hop0 in turn serves as the input entity for the hop1 slot, for which a particular query related to it is manually generated. Hop-0 generally involves the person to person relation and hop-1 consists of person to attribute relation. Figure 8 is a sample of the generated query for a particular entity.
Finally the precision, recall and F1-Score (micro) is calculated for hop-0 and hop-1. The error in hop-0 easily propagates to hop-1 as well. To fairly evaluate each relation extraction model on this task, Stanford's 2015 slot filling technique is used. It is the top ranked evaluation baseline specifically tuned for KBP evaluation. Then later the evaluation metrics for hop-all is developed as the combination of both hop-0 and hop-1 slots.    Table 3, we have used the TACRED model for extracting the relations from unstructured text. We have augmented relations from SystemT externally for slot filling and have not made changes to the TACRED model. In [7], we had presented a negative result in improving TACRED model with sentence embedding and hierarchy information.

Knowledge Graph population from redacted text
We have discussed the Cold Start Knowledge Graph population from unstructured text in this work. An extension of this work is to use unstructured text where the personal data entities are redacted using techniques discussed in [5]. We organized a challenge at AMLD 2020 [8] for this task using a sample of the dataset discussed in this paper. This task can perhaps be posed a masked token prediction problem using language models. Data Programming [18] have proposed a rapid way to annotate training data with weak supervision in their Snorkel data programming system. While we follow a similar approach of using labeling functions written using SystemT, we have not compared different labeling functions to choose the optimal ones. We intend to incorporate this in the future.
Fairness Evaluation using AIF360 [3] introduced a AI Fairness toolkit called AIF360 which provides a number of algorithms for detecting and mitigating bias against unprivileged groups in protected attributes like gender, age, ethnicity, location, sexual orientation and few others. We have used the representative set sampling to show the distribution of the attributes. We plan to use the AIF360 toolkit to evaluate the level of bias in the knowledge graph populated by us.

CONCLUSION
We introduced a system to populate a personal knowledge graph with 20116 nodes, 1681747 entities, 34 attributes and 10 relation types from an unstructured text corpus, namely the wikipedia. Our system uses a combination of neural models and rule based annotators known as SystemT. We then showed how data augmentation improves the overall diversity of the populated knowledge graph and also the performance of the neural models. We also presented a representative set sampling method that maintains the diversity among protected attributes like gender, nationality and ethnicity.