{'fraction_non_alphanumeric': 0.018907078625778187, 'fraction_numerical': 0.0036891860733225734, 'mean_word_length': 4.873392010832769, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 0, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Outliers Emphasis on Cluster Analysis - The use of squared Euclidean distance and fuzzy clustering to detect outliers in a dataset

Outlier is the term that indicates in statistics an anomalous observation, aberrant, clearly distant from others collected observations. The outliers are the subject to animated discussions in various contexts with regard to be or not to be considered in the average evaluations. Outliers can become a precious source of information, on condition that be able to accurately identify the presence in the reference datasets. The need to identify the presence of clustered outliers in a dataset not previously treated could argue for a fuzzy clustering, emphasized by using the quadratic Euclidean distance as similarity measure. For interesting and useful results, it should be inclined a possibilistic clustering approach, where the term 'possibilistic' means, always in mathematical rigor, a component of interpretation of values that point out anomalous cases. The crisp method does not allow it, the fuzzy method introduce it, the possibilistic one use it. This is a very simple paper with divulgative purposes, addressed especially to students, but not only.


INTRODUCTION.
Outlier is the term that indicates in statistics an anomalous observation, aberrant, clearly distant from others collected observations.
The descriptive statistics from datasets containing outliers, may provide highly misleading keys to understanding phenomenons.
Understand its origin is not always easy, but it is definitely necessary. But it is even more important to know what to do, how to treat them and how to identify them uniquely.
The outliers are the subject to animated discussions in various contexts with regard to be or not to be considered in the average evaluations. For example in the business sector, in which it is not uncommon that the budget achievement is carried out by isolated, and perhaps not repeatable, large-scale contracts. Or in the insurance sector, and particularly in the claim sector in which may exist amounts so-called catastrophic, rare to happen but causing highly distorted statistics. Now think at the context of healthness and epidemiology: a group of cancer patients contains one or two people who respond to treatment in a markedly different and positive way: they could raise the average errors of assessment in inducing doses of therapy, to the detriment of other patients while belonging to the group but that respond to treatment in a more modest way.
In the last two years the study of outliers was directed with great determination to fraud cases searching, especially financial, and to internet intrusion, once and for all acquiring authoritativeness in the field of risk analysis. The study of these cases is able to highlight abnormal behavior characteristics that are not always random, but often hide fraudulent and volunteers behavior. Many internal investigations for companies, but also of the police or tax authorities, are based on the search for important quantitative abnormalities in the enormous mass of available data.
From the examples above is possibile to understand how outliers can become a precious source of information, on condition that be able to accurately identify the presence in the reference datasets. The same outliers become a touchstone in extreme situations for quality control: if the point of breaking of a ceramic tile of the Shuttle had + become a simple value among many, or worse, had been put outside from the database to avoid distortions, almost certainly no Shuttle would never return to earth.
In cluster analysis, especially in the presence of numerically relevant datasets, it can be difficult to identify outlier, especially if you have not done a careful prior examination of the dataset.
The same choise of the clustering method should be carefully considered. It is not always required to use the most hard method (crisp clustering) for the group analysis, or rather in some cases may not detect some particularities of the dataset. In many cases, the fuzzy clustering can be an effective and perhaps better alternative. But, as we shall see later, the need to identify the presence of clustered outliers in a dataset not previously treated could argue for a fuzzy clustering, emphasized by using the quadratic Bezdek), and have a difficulty level that is not much suitable for practical use. From the Possibilistic Theory will be maintained basic assumption regarding cancellation of the membership cluster constraints; therefore here will be proposed a theoretical case and a simple method to get a reliable result by using a simple spreadsheet.

CRISP CLUSTERING.
We use a tipical study dataset in traditional clustering, which is a symmetric dataset.
In crisp clustering data is clearly attributed (rightly or wrongly) to a cluster. Imagine that you want to attribute the data in our possession within two clusters.
The centroids are located respectively in the overlapping elements of Z3 and Z8.
The table of allocation would be as follows.
The case becomes complicated when we insert an additional data that arises in exactly symmetrical and equidistant position from the two clusters calculated.
This is clearly an extreme case, probably not much related to reality, but this explain the difficulty of decision that has to take the analyst. Following the rules described in (2.1x) the element Z11 must be inserted into one of the two clusters.
Two alternatives are possible. The first is to decide to create three clusters The other alternative is to consider a fuzzy clustering, in which the allocation to clusters is not unique, but is represented by the amount of the membership of elements to all clusters calculated. In fuzzy clustering the membership value of each element is not only 0 or 1, but any value that lies between these extremes. The conditions are: With these conditions the amount of allocation of individual elements to a specific cluster, rather than the other, calculated in the cartesian coordinates of the centroids shown in Fig. 5 The grades of membership are calculated by reference to the Euclidean distances from the centroid, as determined by the formula The Z11 element has a factor of 0.5 degree of membership for each cluster. Through fuzzy clustering is no longer required, in extreme cases, to decide in which clusters insert the symmetrical element, because the same element is symmetrical and is exactly half (geometrically speaking) between the two centroids, and the table shows it in an absolutely way. The same table also shows, for example, that the element z3 belongs to cluster 1 significantly more than the elements z1, z2 and z4. At this point it may seem easy to say that these considerations are also largely determined by visual approach to the chart (not much about the correct positioning of the centroids), but try to imagine a much larger and asymmetrical dataset, ie with a much less regular distribution: if there is no evidence of graphic centroid (very probable) the reading of a table like the one described above would be the only source of information for analysis.

OUTLIERS PRESENCE.
Let us see the presence of an outlier. For simplicity of exposition we also consider this element symmetrical. The position of centroids change, of course

POSSIBILISTIC CLUSTERING.
A more general form of fuzzy partition is the possibilistic partition. + You get that reducing the constraint (3.1b), while not eliminate it, and then keeping to the memebership value of the item to a subset at least greater than zero.
The conditions thus become: The result does not necessarily provide membership value to clusters whose sum is equal to 1, but could be lower if the item is less typical  Since the detailed study of all these criteria partition is not the purpose of this document, we will only provide arguments on how to distinguish a possible outlier. The reader will have opportunity to + read documents listed in the bibliography, in case he needs detailed statistical arguments.
A good system for the identification of an outlier from a fuzzy system is to use one of the measures listed above: the squared Euclidean distance.
In this way the weight of the elements becomes progressively greater, and greater distances are emphasized.
The and are subject to the conditions (5.1x): the membership value is not 1 or 0 as the crisp partition, but it is between 1 and 0 as the fuzzy partition, and in particular sum of memberships even greater than 0 must be no longer equal to 1 if the item is not typical compared to clusters. In this case a sum of the attribution of 0.20 denotes a highly atypical element.