{'fraction_non_alphanumeric': 0.030263519077683227, 'fraction_numerical': 0.0067938512215207246, 'mean_word_length': 4.49095704596835, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 0, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 1, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Named Entity Recognition System for Urdu

Named Entity Recognition (NER) is a task which helps in finding out Persons name, Location names, Brand names, Abbreviations, Date, Time etc and classifies the m into predefined different categories. NER plays a major role in various Natural Language Processing (NLP) fields like Information Extraction, Machine Translations and Question Answering. This paper describes the problems of NER in the context of Urdu Language and provides relevant solutions. The system is developed to tag thirteen different Named Entities (NE), twelve NE proposed by IJCNLP-08 and Izaafats. We have used the Rule Based approach and developed the various rules to extract the Named Entities in the given Urdu text.


Introduction
Named Entity Recognition (NER) is a subtask of Information Extraction (IE). NER extracts and classifies the true Named Entities in text. NER system is widely used in different tasks of Natural Language Processing (NLP) and in many commercial applications on internet like Search Engine. Accuracy of NER system is directly reflected in NLP applications. So, accurate working of NER system is very important. NER system can be used for one's personal interest like company manager wants to know all the names involved in specific text document.

S.No:
Tag Name Example 1 Person Name ‫(ارشد‬Arshad) Named Entities mentioned above were proposed at IJCNLP-08 workshop [17]. Named Entities can be domain specifics like NER system to identify entities in scientific data.
The NER system can be developed using three approaches, 'Rule-Based', 'Machine Learning' (HMM, SVM, CRF, Decision Tree) and 'Hybrid' approach. The Rule-Based system is difficult to develop as one should know the language and grammar rules. These kinds of systems are domain specific. Machine learning approach provides different Statistical NLP tools to train NER system. Statistical tools provide fast way to develop NER system but the accuracy of the system is dependent on annotated training data. For greater accuracy we need to train the NER system with large amount of annotated data. Hybrid approach is a combination of both Rule Base and Statistical based.

Related Work
NER system came in focus during the sixth Message Understanding Conference (MUC-6) [6] . After that many NER systems were developed. Most of these systems were developed for European languages and all systems were highly accurate. For south Asian languages, NER systems yet in developing phase. IJCNLP-08 workshop played a major role in development of NER Systems for Indian languages. This Workshop focused on five languages i.e. Hindi, Bengali, Oriya, Telugu and Urdu. All the systems were developed using Statistical approaches or Hybrid approach. Hybrid NER system for five languages was developed by (Sujan Kumar saha et al. 2008) [2]. Rules were developed only for Hindi and Bengali. The system was developed 3. Approaches to NER 3.1 Rule Based approach: Rules are developed to identify NE in text. This approach takes much time in development and one should have good knowledge of target language. Heuristic based rules are used to identify tags and these rules are language specific. Good rules always yield good results. Development of these kinds of systems is always a time consuming task.

Statistical approach:
Statistical approach is also known as Machine Learning approach. This is a fast way to develop a NER system. The system is trained using annotated training data set in specified format. Accuracy of statistical approach is dependent upon the training data. So, we always train the system with a large set of annotated data. Various Machine Learning models like HMM, CRF, MaxEnt, are used for NER system. 3.3 Hybrid system: Hybrid system is combination of Rule Based approach and Statistical approach. To develop the Hybrid system we use Statistical tools as well as linguistic rules. Combinations of both approaches make a system more accurate and efficient.

Issues in Urdu NER System
No capitalization: Urdu and other Asian Languages do not have concept of capitalization. In European language like English this feature is widely used to recognize Named Entity in text because all the names in text always begin with capital letter. Absence of capitalization feature makes the NER task hard for Urdu language.
Ambiguous Name: Urdu language has lots of ambiguous names that can be used as proper noun as well as common noun. Main challenge of any NER system is to separate or extract proper noun in place of common noun. Example: ‫کت‬ (barkat) or ‫ت‬ ‫سا‬ (slaamat) can be the name of a person or it can be used as common noun.
Spelling variations: Lack of standardization in Urdu language can be seen in spelling as well.
There are different spellings that can be used for same word. Like word Hospital can be written in two ways in Urdu ‫ل‬ ‫اسپت‬ / ‫ل‬ ‫ہسپت‬ (hasptaal/asptaal) which makes the task difficult for NER. We are unable to collect the standard spelling of foreign language. Example: ‫سٹیچیوٹ‬ ‫ا‬ / ‫سٹیٹوٹ‬ ‫ا‬

(institutes/instichutes).
Non-availability of resources: Language Resources are must for any approach whether it is Rule Based or Statistical. There is no large gazetteer and annotated data available for Urdu language.

Why Rule Based Approach
Rule Based approach is time consuming task to develop any NER system. Rule based approach is used only when you know the target language well and have sufficient knowledge about the linguistic rules like knowledge of grammar. The system developed using Rule Based approach always yields the good results. On the another hand, Statistical approach which provide us with many Statistical tools, to develop NER system like HMM, CRF, SVM, MaxEnt etc, with the help of these tools development process of the system is rapid as compared to Rule Based approach. We have studied that in IJCNLP-2008 that all the NER systems were developed using different Statistical approaches. But none of the system provides good results for Urdu text because annotated data provided by the workshop is only 36000 Urdu tokens which is not sufficient to train Urdu NER system. New Statistical techniques like CRF not perform well for Urdu. Absence of any large Urdu gazetteers is also one of the reasons for low accuracy. These kinds of gazetteers boost the accuracy of Statistical approaches. Rule Based approach used by (Kashif Riaz 2010 [4]) for Urdu language shows good results. Rules are used to identify six tags. Workshop on NER system by IJCNLP 2008 focused on twelve NER tags. By studying various research papers we concluded that we should follow Rule Based approach though it is a time consuming approach but this approach will give us promising results as we have seen in Kashif Riaz's [4] system. We did not try Hybrid Approach because the absence of large annotated corpus for Statistical part. We have tried to develop different rules for all 12 NE tags which are used in IJCNLP-08 workshop.

Rule Based Model
Following Rules are used to identify different tags in Urdu text.

Algorithm for Urdu NER
We have developed the system on Windows platform using Dot Net framework 4. System is using other available classes of framework to implement all features of our NER system. Like Tokenizer and Linked List class and its functions. We have developed many other modules for different rules used by the system. The system works in linear complexity. Rule is applied to resolve ambiguity in names. 8.5 Rule is applied to find Numbers in non numerals form. 9. Rule is applied to find out Abbreviations which were not found during Gazetteer search. 10. Rules are applied to find out Organizations 10.1 Those Organizations entity which were not found during Gazetteer lookup, rule will try to find out and tag them as organization entities. 11. Show tagged output to user along with untagged data. Algorithm is self explained; still lightening some of its steps regarding Gazetteer look up. Gazetteer lookup is used to find out various Named Entities in text. We have collected Named Entities related to various fields i.e. Politics, Business etc. In algorithm's step 3 gazetteer look up is used for Locations, Terms, Brands, Abbreviations and Organizations Tags. All these tags are less ambiguous so the system tags them without applying any rule. System have gazetteer list related to these tags which are not ambiguous. In Step 4 system tokenized the input text and normalized the tokens for further processing. Step 6, 7 and 8 are used to tag Person Names. In Step 6, algorithm finds out person names based on Title Person, Designation and Surname. In this step system is able find out Person Named Entities without any gazetteer list. In Step 8, system used gazetteer list to find out person names and apply various rules to resolve their ambiguity. Some person names have patterns in its suffix or in prefix of the word. So, Step 6 of algorithm finds out this kind of person name by using suffix stripping.

Evaluation and Results
We have constructed two sets of test data. Test data is collected from different websites [19] of Urdu. Test data mainly include News from different fields like Politics, Sports, Business and Science. The reason of collecting News data is that because News data is always full of Named Entities. So, it gives challenging job to our NER system to identify all different kinds of NE tags accurately. Test data also include ambiguous data, our NER system tried to resolve ambiguities and tag only true entities. Mainly ambiguities are in person names and the system resolves them by applying different rules. For evaluating the system we have tested the system on two different test data sets. Both test data sets have news and articles related to different domain. Test data set 1 have data related to political news, some articles and short stories. Test data set 2 mainly have news related to science and business.

Test Case Number of tokens Domain
Test set 1 12032 tokens News and articles related to politics.

Test set 2 150243 tokens
News data related to science topics, business news.    Table 5 for four tags are not good as compared to other tags because these tags need more time to collect accurate data. If we consider accuracy of all the thirteen tags then accuracy is 74.09%. Accuracy of the system also depends upon the domain of testing data. Test set 2 includes scientific and business terms so that system was not able to perform well. We have considered thirteen tag as compared to twelve tags used in IJCNLP-08. The system tagged Izaafat words because in Urdu, Izaafats are used very frequently and when we translate Urdu to target languages then we need to translate these Izaafats in specific target language words. Like (aab-e-hayat) ِ ‫آ‬ izaafat meaning in English is sacred water. But some time izaafat plays the role as NE for example ‫لصہ‬ ‫خ‬ ‫وراثت‬ (vīrāsat-ē-khālsā) name of a place and it should not be translate in target language but transliterate it.

Conclusion and Future Work
We have developed the system to tag different Named Entities and system is able to find out and tag them all. But system has some limitations too.
1. We have developed the rule to tag person's name having length of three words. If person name has longer string of words like four words in a name then it will tag three words as one Named Entity and fourth one as another Named Entity. For Example in ‫رداری‬ ‫ی‬ ‫ع‬ ‫آصف‬ ‫د‬ (mohmad āsīph alī jardārī) person name, ‫رداری‬ (jardārī)will be tagged separately and ‫ی‬ ‫ع‬ ‫آصف‬ ‫د‬ (mohmad āsīph alī)will be tagged as another Named Entity. Count of Entities will be two in place of one.
2. Same is the case with Number tag. Like: ّ ‫چو‬ ‫سو‬ ‫دو‬ (Two Hundred fifty four) tagged as one named entity but if we have longer string having more than three words like ‫اکھ‬ ّ ‫چو‬ ‫ار‬ ‫دو‬ (Two Thousand Fifty Four Lakh) will be tagged as two separate entities. Where (Lakh) ‫اکھ‬ will be tagged as a separately.
3. Partial tagging problem in Org tag like ( ‫ڈی‬ ‫ا‬ ‫لوجی‬ ‫ٹیک‬ ‫آف‬ ‫سٹیوٹ‬ ‫ا‬ ) Indian Institute of Technology will tagged partially as (Institute of Technology) word Indian will not be tagged with that Org tag. 4. Problem in Date tag, some time Date is written in word form like: ‫ر‬ ‫گی‬ ‫ر‬ ‫دوی‬ ‫وری‬ ‫ج‬ ‫یس‬ (Twenty January Two Thousand Eleven.) This kind of string will not be tagged as Date tag but will be treated as separate tags like January as Time tag and Twenty as Number tag and Two Thousand Eleven as one Number tag. 5. System is not using any kind of technique to resolve the segmentation problem in given Urdu text. Like other Asian languages Urdu has problem of space omission and space insertion. We will try to improve our NER system by using effective segmentation technique in near future. 6. System is not using POS Tagger or POS tagged data. So system always has to do larger number of comparisons to find out entities in given text. It is difficult to make any decisions for system based on rules because system compares only words but not their part of speech. Due to large numbers of comparisons system is little bit slowly when we gave large amount of input text to system. POS tagged data and stemmer is very essential for NER system and we have not used any POS tagger for our NER. So, we will develop POS tagger to include it in our system. 7. There is a problem to resolve ambiguity of person's name. We have collected the data of ambiguous person names and system treats them as special case to resolve its ambiguity but some time all the rules fail to remove its ambiguity. For example ‫دو‬ ‫اوا‬ ‫کو‬ ‫فتح‬ (phatah kō būlāva dō) here word ‫فتح‬ (phatah) can be proper noun or common noun. We do not have any clue to find out whether it is a person name or not. Rule checked Title Person, Designations or surname if all these conditions are not present in given sentence then we need to check out postpositions, but some time these postpositions come along with common name.
We have collected 6000 person names gazetteer. We will collect more number of person names and will try to include surname of other languages. Main problem is gazetteers of terms related to various domains. Collected data is not sufficient. We do not have standard spelling of terms related to various fields. We will try to collect standard spellings of different terms.
To develop the Urdu NER system we have insufficient resources as we discussed earlier but still we are able to get good accuracy. In future we will try to develop other essential tools for Urdu NER.