{'fraction_non_alphanumeric': 0.04077414505824878, 'fraction_numerical': 0.0050497933107854194, 'mean_word_length': 4.286441519741743, 'pattern_counts': {'":': 1, '<': 9, '<?xml version=': 0, '>': 5, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 11, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Pragmatic Theory of Machine Learning

C.S. Peirce understood pragmatism (pragmaticism) as a method of deriving new knowledge for practical use through explaining observations. But this is what machine learning (ML) is doing, essentially. A solution one infers in ML can be seen as the best explanation of the accumulated facts (the training set) intended for help in decision making. Peirce used the term \textbf{abduction} for this kind of inference. Here I formalize the concept of abduction for real valued hypotheses, and show that 14 of the most popular textbook ML learners (every learner I tested), covering classification, regression and clustering, implement this concept of abduction inference. The approach is proposed as an alternative to Statistical learning theory, which requires an impractical assumption of indefinitely increasing training set for its justification.


Introduction
The only commonly accepted theory of machine learning (ML) is statistical. To understand ML as a statistical problem, one has to implicitly assume that (1) we are learning a dependence generated by a fixed probability distribution and (2) the training set may be increased indefinitely so that "eventually" this distribution will be well represented by the training set. .
For most of practical cases these assumptions are a stretch. Applied ML is about learning a nondeterministic dependence from a given finite sample for some urgent decision making. Limited sample does not allow one to assume existence of probabilities or even existence of infinite populations. Essentially, applied ML and theoretical ML solve different problems. As a result, theoreticians cannot answer the questions practitioners ask [12].
Instead I propose to apply Peirce's [9] pragmatic view on learning from experimental data. Within this paradigm, ML is a search for the best explanation of the observations. This search logically understood by Peirce as abduction inference. In artificial intelligence, problems of diagnostics were already considered to be an example of abduction [4].
Here I adapt the concept of abduction to deal with real-valued hypotheses and formulate the concept of a abduction learner. I conjecture that every (worthy) learner in ML is abduction learner. Fourteen popular textbook learners for classification, regression, clustering are shown to support the conjecture.
To the best of my knowledge, the proposed approach is the first one to explain and logically justify large variety of existing learners from a single point of view. Pragmatic understanding of applied ML opens a new path to solving "how to" questions practitioners ask, to design new useful learners for real life problems.
Here is a brief description of each section: 1. Traditional views on ML: learning to predict, statistical learning.
2. Pragmatic view on learning, informal description.
3. Logic of data explanations: alignments and deviations. 4. Logic of data explanations: aggregation of deviations 5. Logic of data explanations: recursive aggregation. 6. Explanation criteria 7. Abduction learning procedure. The Main Conjecture. 8. Proof that the most popular learners are abduction learners. 9. Conclusions.

Traditional views on ML
Here I describe two understandings of ML: traditional one and one of statistical learning theory.

Givens
Denote Ω the set of real life objects of interest. For example, this may be patients with skin cancer, or bank clients or engine failures. There is a hidden essential quality we would like to find out (may be, a diagnosis or prognosis). Some properties (features) of the objects Ω can be always evaluated and numerically expressed. Some of them are expected to be relevant to the hidden property. Suppose, there are n such features. Denote X ⊆ R n domain of feature vectors for objects in Ω. The hidden essential quality also has numerical expression values in Y ∈ R. The value of the hidden essence in a given object is called "feedback". We assume there is an "underlying dependence" ϕ : X → Y between feature vectors and the feedback. Yet, we can not assume that the dependence is deterministic.
For example, the features may not define completely the feedback we are trying to model, there is uncertainty in measurements, random mis-classification and so on. Objects with the same features may have different feedback, and the same object evaluated twice may have different features or even feedback. This is not a bad luck, but a inevitability. Indeed, ML is needed only when there is no exact theory explaining the phenomenon we are trying to predict. Therefore, we do not know what it depends on. The measurements have intrinsic uncertainty.
The information about the underlying dependence ϕ is given as training set: observations about values of feedback in certain data points set of tuples { x, y }. These tuples will be also called empirical instances.
The sooner we find the proper hypothesis about the dependence, the better for the decision making. So, data shortage is not a bug, it is a feature of ML.

Prediction problem
The goal is assumed to be prediction of future values of the nondeterministic dependence.
For example, here is how the prediction problem is understood in [7]: Given a training set S and data point x of a new observation x, ? predict its feedback y.
The main issue with this problem statement is that to evaluate the decision and to select between the hypotheses we need to know what is not given: the future.
There is no way to solve this problem with the available data.

Statistical Learning Theory Approach
Statistical Learning (SL) theory is the only commonly accepted theoretical approach to ML. This is how the proponents of the SL theory understand the problem: "Intuitively, it seems reasonable to request that a learning algorithm, when presented more and more training examples, should eventually "converge" to an optimal solution." ([6]) The "optimal solution" here is the hypothesis having the ER loss criterion close to minimal for the given class of functions regardless of the distribution.
It does not appear to be intuitive to solve the problem where more and more training examples are expected, if we have only one finite training set. V. Vapnik [14] formulated the justification of the statistical approach in the most direct way Why do we need an asymptotic theory · · · if the goal is to construct algorithms from a limited number of observations? The answer is as follows: To construct any theory one has to use some concepts in terms of which the theory is developed · · · .
In other words, statistical learning theory assumes indefinite increase of the training set, so that the statistical approach can prove some results. Statistics has laws of large numbers, so the problem has to be about ever increasing training sets and convergence.
The apparatus of probability theory does not help to understand and solve the true pragmatic problem with fixed finite data, finite time allocated for decision making and un-quantifiable uncertainty.

Pragmatic view on learning
First, I address the common misconception that ML is induction.

ML is not induction, but abduction
It is a common belief that ML is an induction inference. But is it?
The dictionary says that induction is a method of reasoning from a part to a whole, from particulars to generals, or from an individual to universal. Roughly, induction extends a property of a part on the whole. We start with objects of a certain class and their known common property, then we conclude that all objects of this class have the same property. This procedure is exactly opposite to deduction, when knowing a property of a class and one object of the class, we infer the same property of the object. For both these types of inference, the common property of objects of interest (the hypothesis), is given in the beginning.
This is not how we learn. We start with objects (observations) and no hypothesis, The property -the pattern -which unites objects is not known. It is what needs to be found.
Guessing the hypothesis to explain the facts is what Peirce called abduction inference [8].

Informal description of pragmatic learning problem
From pragmatic point of view, the goal of learning is to find the best explanation of observations, not prediction. Let h be a function used to explain observations. I call it explanation hypothesis, or simply explanation. Notation S is used for training set, H(h) set of all hypothetical instances of the function h. We will be interested in the conglomerate of instances M (h, S) = {S, H(h)}.
Informally, we can formulate the explanation principle: For an explanation h to be any good, the values of feedback on instances in M (h, S) with close data points shall be close.
Closeness of feedback on close data points in the conglomerate M (h, S) can be used to evaluate an explanation quality.
The explanation principle makes it clear that a nondeterministic dependence has to have mostly close feedback on close data points to be "explainable". And to be a good explanation for an explainable dependence, a hypothesis has to have a close feedback on close data points as well.

Data Explanation Logic
Now, I concentrate on developing a formalism for the abduction criterion

Language of Alignment
So far, I considered only an underlying dependence with a single variable. In a general case the restriction is not necessary. For example, using two independent variables may be convenient for formalizing ranking problem.
The first order signature has these 6 + n sorts among others: Some other sorts will be introduced later, when we need them to describe aggregation and regularization.
Usually, domains of the sorts X 1 , . . . , X n and Y are some metric spaces. However, the triangle axiom of a distance on a domain is irrelevant, not required.
The first order symbols are defined in the Table 2.
The function o distinguishes observations from the hypothetical instances. It may be useful, when there are many types of observations.
When there is only one independent variable (n = 1), I will skip the index variable in ρ x and x.

Alignments and Deviations
The formulas are formed as in first order predicate logic.
Let us consider an example. The simplest alignment for the case when n = 1 is called point-wise: The simplest deviation is

Aggregation of deviations
To compare explanations, one has to compare collections of deviations. Usually, it is done by mapping such a collection into a single number for a comparison. Let us assume for simplicity that deviations corresponding to the same alignment are ordered one way or another.
Denote Z an additional domain of finite sequences of real numbers. The elements are interpreted as sequences of deviation for aggregation and will be denoted by capital letters A, B, G. The elements of sequences will be denoted with corresponding small letters with indices.
Let A ′ be an operation of ordering elements of the sequence A from smallest to largest.
Here are the binary relations on this domain.
We will use three functions  We will call aggregation stable, if the next property is also satisfied: . This contradicts stability. The proof for the minimum is similar.

Proof. It is obvious that If
Take any i and denote b ′ j = q(a ′ i ). Then b ′ i > a ′ i in all three possible cases: Here is an example of an aggregation operation.

Theorem 1. Any percentile is a stable aggregation.
Proof. Denote P r (A) r-th percentile of A ∈ Z. Order insensitivity is obvious, because percentile does not take into account order of elements. Let us prove monotony. Let A < B. From the Statement 2 it follows that A ′ < B ′ and for any i a Stability is trivial..

Recursive aggregation
Here I introduce a language to generate a class of aggregation functions called recursive aggregation.

Language of recursive aggregation
We will use additional functions Table 4: Function symbols Symbol Arity Sorts of arguments Semantic

Function [ ] (scaling)
The function [ ] is strictly monotone: Typical examples of the function [] The function max(x − a, 0) cannot be used for scaling since it is not strictly monotone.

Function ⊕ (compounding)
The function ⊕ has three axioms: The next functions satisfy all the axioms:

Function η (normalization)
The function η is strictly isotone by the first variable and antitone by the second variable.

Recursive aggregation function
The function is defined recursively Step 1 : Step i + 1 :

Recursive aggregation
Recursive aggregation is defined by formula

Theorem 2. Recursive aggregation Ω(A) is an aggregation.
Proof. Let us prove order insensitivity. Suppose, a sequence B is a permutations of a sequence A.
Each permutation can be obtained by finite number of transpositions of neighboring elements. Suppose, B can be obtained from A by K transpositions. Let us prove the theorem by induction over K. First, suppose K = 1.
Suppose, the B transposes elements a i , a i+1 . Denote α(l), β(l) values of the recursive aggregation function obtained on the step l with the sequences A, B respectively. Since all the elements prior to i are identical in these orders, Using symmetry and associativity of the function ⊕ we get a(i + 1) = b(i + 1). All the elements starting from i + 2 are identical in both sequences A, B. Therefore Ω(A) = Ω(B).
Suppose, we proved the property for K = k. Let us prove it for K = k +1. Suppose, the transpositions are ordered by the indices of involved elements, and the last transposition involves elements a i , a i+1 . Then, the same considerations apply again.
Let us prove monotony. Suppose, A < B. According to the Statement 2, then A ′ < B ′ and for any i a i < b i .
Let us prove it by induction by n = A = B . For n = 1 .
Then Ω(A) < Ω(B) because both functions η, [ ] are strictly monotone by the first variable. Suppose, the statement is proven for n = k. Denote A k , B k sequences with the first k elements of the A, B respectively. By the inductive hypothesis Since the function η is strictly isotone by the first variable. it means .
Let us prove monotony of operation Ω for n = k + 1.
because both arguments of the operation ⊕ on the left are smaller than corresponding arguments on the right. The function η is strictly monotone by the first argument. This proves the theorem.
Here are the most popular aggregation functions.
Theorem 3. For a sequence A : m = A , Proof. For every operation the functions of the Recursive aggregation language are defined in the next table: This proves stability and the theorem.

Explanation criteria
In general. an explanation criterion evaluates quality of a hypothesis. Informally, the idea of such a criterion is presented in 2.2.

Regularization
As I mentioned in 2.2, "For an explanation h to be any good, the values of feedback on instances in M (h, S) with close data points shall be close." In particular, an explaining hypothesis h itself shall have close feedback on close data points. In means it shall not have high derivatives, when it is differentiable.
To take into account derivatives, the model of Data Explanation Logic shall have yet another base with the only vector: a sequences of parameters of the explaining hypothesis.
Regularization is a real-valued function on this vector. Usually, the function is some evaluation of derivatives.

Badness rule
A badness rule is a triple of alignment criterion π, deviation function δ and an aggregation operation Ω. For a recursive aggregation, the aggregation operation is further determined by the functions ⊕, [ ], η.
Let us consider an example of a badness rule called Point-Wise rule, T pw . The rule includes 1. An alignment relation (see (1)):

A recursive aggregation operation "averaging" (see Theorem 3) with operations
.

Explanation criterion
An explanation criterion consists of • Series of badness rules.
• Combining operation C(A), defined on sequences of real numbers. It maps outputs of all previous rules into a single number, criterion value. Combining operation shall be monotone by all components of the vector A.
Regularization rule is not necessary. If there is only one badness rule then combing operation is not used.

Search for the best explanation
Pragmatism implies a search for the best explanation. Here, I want to narrow a concept of search and describe two basic search procedures.
Let us consider learning algorithm which minimizes an explanation criterion L(h, S) on a class of hypotheses h ∈ F, given a training set S. The next procedures will be considered standard.

Basic training
Basic training takes F, S, L(h, S) and parameter q, and it consists of the next steps Focusing may be a nonlinear transformation of the training set. Yet, typically, it is used to select observations, or features, or emphasize some of them with weights.
The selection of hypotheses may not go over whole class F , but its finite subclass F ′ ⊆ F.

Wrapper strategy
Wrapper strategy takes F, S, L(h, S), the empty set Q and consists of the next steps: Stopping check here checks a specified condition, and if it is true, the procedure exits Wrapper loop. Otherwise, the loop continues with generating new parameters.
Summarizing, we get a definition of an abduction learner

Abduction learner
Abduction learner minimizes an explanation criterion on a model of Data Analysis Logic using Basic training with or without Wrapper strategy.

The main conjecture
Each popular learning algorithm (the procedures of k-NN, Naive Bayes, SVM, hierarchical clustering, for example) is formulated in unique terms, apparently solves its own type of problem. My conjecture is that they all can be explained as abduction learners,

Main Conjecture
In ML every learning algorithm is an abduction learner.

Popular learners support the Main Conjecture
To show that a learner is an abduction learner we need to show that, given a training set, it minimizes an explanation criterion using basic training with or without wrapper loop.

Linkage-based clustering
The algorithm is also popularly known as hierarchical clustering. Clustering can be seen as modeling a nondeterministic dependence, where the cluster number is an independent variable, and data vector is a feedback. It can be said that by finding an association of a data point and its cluster number, we explain the training data as a set of clusters.
In [13], a general concept of linkage-based clustering is introduced this way: These algorithms proceed in a sequence of rounds. They start from trivial clustering that has each data point in a single-point cluster. Then, repeatedly, these algorithms merge "closest" clusters of the previous clustering. . . . Input to a clustering algorithm is between-point distance, d. There are many ways of extending d to a measure of distance between domain subsets (or clusters).
The book proposes three ways to evaluate the cluster-distance by the distances between their members: by minimum distance, average distance and maximum distance.
The last option clearly contradicts declared goal "merge 'closest' clusters". But I will consider it too.
For each round, the training set is a sequence of first order formulas of LDE where c i ∈ N is a cluster number of if i-th observation, and y i is the observed data point of the same observation.
the set of data points of the cluster i. Suppose, there are k clusters. The purpose of a round is to identify two "closest" clusters. The notation h ij , j > i will indicate a hypothesis that clusters C i , C j "belong together", are the best candidates for merging, and H k = {h ij | i < j ≤ k} denotes the class of all the hypotheses.
All the hypothetical instances of the hypothesis h ij make the set The loss criterion L(h ij , S) = ρ(C i , C j ) is the cluster-distance between the clusters C i , C j . Now, the learning procedure can be described by the rule: Proof. To show that the learner is abduction learner, we need to show that, the L(h ij , S) is an explanation criterion for every suggested cluster-distance, and the described procedure is Basic training.

Round of hierarchical Custering
The explanation criterion has a single badness rule with alignment relation the same relation as for the T pw badness rule, The deviation is defined by the rule It means, the deviation function t in this case also coincides with the deviation function for T pw .
The aggregation operation is identified by the type of clustering: minimum, average and maximum. Minimum and maximum are aggregations by the Theorem 1. Averaging is a recursive aggregation as proven in Theorem 4. Therefore, the criterion L(h i,j , S) is a badness rule and an explanation criterion. The learning procedure is Basic training without focusing. This proves that each round of linkage-based clustering works as an abduction learner.
The learner would agree with the main conjecture not only for the aggregation operations mentioned in the book (average, minimum, maximum), but also for any other aggregation operation.

k-NN
This classification method is intended for the observations with binary feedback in Y = {0, 1}. The expectation is that in small neighborhood an underlying binary dependence is, mostly, constant. For binary dependencies this condition is equivalent to being "explainable", as discussed in 2.2. The less is the difference between f and the feedback in the training data points close to x 0 , the better is the explanation. The goal is to find the best explanation out the two hypotheses.
Small neighborhood is defined by the parameter k. Denote d k distance from the point x 0 to the k-th closest to x 0 data point of the observations S. Then d k , x 0 are parameters of the learner.
The alignment relation is The deviation function t(r 1 , r 2 ) = r 2 . So, the deviation δ(α, β) = ρ y (α, β). And the aggregation operation is averaging L(f, S, k, x 0 ) = L 1 (A) (see Theorem 3), where A is sequence of deviations in arbitrary order. Thus, the criterion L(f, S, k, x 0 ) is an explanation criterion.
The procedure of the learner can be described in these steps.

k-NN
• Focusing:Defining parameter d k The procedure is the Basic training. Thus, k-NN is an abduction learner.

Two k-NN learners with adaptive choice of k
The k-NN may work, because we presumed the underlying dependence to have, mostly, small variations of feedback on close data points (section 2.2). For binary underlying dependence it would mean that it is, mostly, constant in small neighborhoods. Optimally, the radius d k shall be small enough to have majority of the points in the neigborhood of the same class, and large enough that random outliers in the finite sample S(ξ, k) do not play much of a role.
Here I discuss two approaches to select k optimally for every new data point. The first is described in [1], the second is my new algorithm. Both learners find prevalent class y in the focus sample, calculate its frequency p k (y) and the error rate r k (y) = 1 − p k (y) the same as k-NN.
Authors [1] propose, given a data point x, start with a small k and gradually increase it while calculating bias t k (y) = p k (y) − 0.5 of the prevalent class with every k. The procedure stops when the bias reaches certain threshold. If the threshold was not ever reached, they don't output any answer. So, they search for the smallest neighborhood where the prevalence of one class is above the threshold they picked beforehand.
The threshold they propose to use is: where n is size of the training sample, δ and c 1 are some user-selected parameters, picked before any data analysis. Thus, instead of one parameter, k, the proposed modification require a user to pick 2 parameters with unclear meaning. The learner uses the same criterion as k-NN. The procedure can be described like this: • Output: If k < n, output f ′ as decision. Otherwise, refuse to output a decision.
Thus, the learner performs the basic training of the original k-NN with wrapper for parameter selection, corroborating the main conjecture.
This learner is developed within the statistical learning paradigm, where the training set is expected to be arbitrary large. As n increases, so does the threshold ∆(n, k, δ, c 1 ). Therefore, the selected value k, the size of the focus training set, will go to infinity with n. And thus, by the law of large numbers, the solution will converge asymptotically to the expectation of the class in the given neighborhood. At the same time, the ratio of k to n is expected to decrease, thus the size the k-neighborhood will tend to 0. If the distribution is continuous in x, then the leaner will likely find the solution as n tends to infinity.
The issue here is that n is not going to infinity or anywhere. For a fixed n, the learner favors smaller k, where the evaluation of prevalent class is subject to random fluctuations caused by small sample.
To alleviate this issue, I propose an alternative approach which uses Hoeffding inequality ( see, for example, [13]) to select k.
The Hoeffding inequality can be written as where p is observed frequency of an event, E is the expected frequency (probability) of the same event, and t is an arbitrary threshold, and k is the sample size. Suppose, p evaluates observed frequency of class 1 (rate of the class 1 among the neighbors), E is the probability of the class 1 in the neighborhood of a given point. If p is above 0.5, then observations of the class 1 prevail, and we pick hypothesis 1 out of two. Otherwise, the we pick hypothesis 0.
Let t = | 0.5 − p |. If | p − E | > t the expected prevalent class is different from the observed prevalent class. If it is the case, we selected the wrong hypothesis. In this case, the right side of the inequality gives us an upper limit of probability that we picked the prevalent class wrong.
For selection of k we use the weight, calculated as the right part of (3) : Obviously, the larger is k, and the further is the frequency p from 0.5, the lower is the weight. The weight will serve well for the selection of the parameters k, because we need to find the neighborhood where p is far from uncertainty, 0.5, yet, the size of the neighborhood is not too small.
Here is the description of the learner's procedure for the given data point x.

Hoeffding k-NN
• Generation of parameter k := k + 1 -Basic training: * Focusing: Select focus training set Q k (x) of k observations with data points closest to x. * Fitting: Evaluate error rate r k (c) of hypotheses c ∈ {0, 1} in Q k (x) * Optimal selection: Select the hypothesis c ′ (k) with minimal error rate r k (c ′ (k)).
-Stopping check: k = n − 1 Thus, this learner is an an abduction learner as well.

Decision trees
For this learner, the features are expected to be "ordinal": every feature has finite number of ordered values; there are no operations on feature values. The feedback of observations is binary. Again, the assumption is that the underlying dependence is mostly constant in small neighborhood. Here, the size of the neighborhood is not set up a priory. The algorithm finds maximal homogeneous neighborhoods for the best explanation of observations. The learner starts with whole domain, splits it in two subdomains by a value of some feature. Then, the procedure is repeated for every of the subdomains until a subdomain called "leaf" is reached. The decision is selected for this subdomain. The navigation over the tree of subdomains continues until some stopping criterion is reached. The algorithm has a precise rule for generating the parameters of the next subdomain based on the previous trajectory and the obtained results.
There are two criteria of a leaf: 1. Number of observations in the subdomain is below a threshold N .
2. Percentage of observations of the prevalent class in the subdomain is above the threshold q (it is homogeneous).
In each subdomain the procedure selects one of two binary hypotheses with minimal error rate. It is easy to see that badness rule for selection of the hypothesis is T pw ??. We can denote L tw (h, S) the explanation criterion of the learner.
The procedure may be described as a Basic training with Wrapper strategy:

Decision Tree
• Generating parameters g of the next subdomain -Basic training: * Focusing: select subdomain G(g) and subset of the training set S(g) with parameters g * Fitting: Generate hypotheses h ∈ {0, 1} on S(g) and evaluate their loss criteria L pw (h, S(g)). * Optimal selection: select the hypothesis d(g) with minimal value of L pw (h, S(g)), h ∈ {0, 1}.
-Stopping check End of tree • Output: For each g : W (g) = 1 output d(g) as decision on G(g).
For the points outside of any leaf the decision is not defined. Therefore, the Decision tree is an abduction learner as well.

Naive Bayes
The underlying dependence has n independent variables and binary feedback. The learner works as if it deals with nominal data: the only relationship between data points is equivalence.
The procedure defines decision function on one data point at the time. For a given data point z = z 1 , . . . , z n the procedure selects n subsets of the training set. Subset S j includes all the observations with j-th variable equal z j . For each subset S j , the learner evaluates error rate e(c, S j ) of each hypothesis c ∈ {0, 1}. Then for each hypothesis it calculates criterion ∆(c, S) = j (1 − e(c, S j )).
The learner selects a hypothesis with the maximal value of the criterion. The explanation criterion for this learner has n badness rules as well as a functional ψ to aggregate values of all n badness criteria.
The i badness rule for a hypothesis c is Aggregation operation in each badness rule is recursive aggregation averaging, L 1 (A) from the Theorem 3.
A combining operation to combine the results of all badness rules is It is easy to see that the operation Ψ is both monotone and order insensitive (see 4) . Thus, the loss criterion of Naive Bayes is an explanation criterion. Now the procedure of the learner with given data point z = z 1 , . . . , z n may be described by the rule
On the Fitting step, the procedure calculates n badness values (corresponding n badness rules) for each hypotheses, then aggregates these values into the abduction criterion for a given hypothesis. Interesting that calculation of each individual badness on this way requires focusing. However, as always, a process involved in calculation of a criterion is not reflected in the scheme of the learner.
This proves that Naive Bayes Naive Bayes is an abduction learner. A product in the aggregation of the badness values is chosen in Naive Bayes because it is sensitive to the low frequencies of a class: if some value 1 − e(c, S j ) is close to 0, the product will be affected much more than the sum of the frequencies, for example. If some feature value almost never happens in a given class c, the hypothesis c will have no chance of being selected, regardless of other feature values of z. It justifies choice of product for aggregation.
The loss criterion of Naive Bayes is traditionally interpreted as evaluation of posterior probabilities with "naive" assumption that the features are independent. There are several issues with this narrative.
First, it works for only one learner: if NB learner is based on naive idea about Bayes rule, other learners would need different foundations.
Another issue is that it creates an impression that the learner needs an improvement, is not sophisticated enough.
I hope, I demonstrated that interpretation of the learner as "naive" and "Bayesian" misses the point. The procedure is driven by its specific data type, and it is explained as performing AGC inference, the same as majority of other learners.

Logistic Regression
This learner assumes the features are continuous, the feedback of the observations is binary, but the feedback of the decision is continuous: so, it is required some rounding up in each data point. The decision is defined on the domain χ. The procedure of generating the hypotheses is not specified.
The class of functions associated with logistic regression is .
The functions have values in the interval (0, 1). The learner minimizes criterion The explanation criterion contains only one badness rule.
The function is isotone by ρ y (α 1 , α 2 ) and does not depend on ρ x (α 1 , α 2 ). The proper aggregation operation is averaging (L 1 (A)). It is obvious that this badness rules defines the criterion ∆(f, S).
So, the logistic regression supports the main conjecture as well.
The issue with this learner is that, the same as ERM, it does not take into account the similarity of feedback in close (not identical) data points, and, therefore, has a tendency of overfitting.

Linear SVM for classification
All the previous learners belong to machine learning "folklore". Their authors are not known, or, at least, not famous.
SVM is one of the first learners associated with a known author: it is invented by V. Vapnik. His earliest English publications on this subject appeared in early nineties [3], [2]. Let The problem is formulated as minimization of the criterion s.t. for all β ∈ S, y(β) · f (x(β)) ≥ 1 − ξ(β) and ξ(β) ≥ 0.
The criterion looks intimidating, but it may be simplified though. For this, we want to switch to narrower class of functions, which shall contain all the same decisions.
The observations β ∈ S satisfying condition y(β)·f (x(β)) > 0. The last consideration implies that if f is the decision of the problem, then the problem has a decision f ′ in the class F ′ (S) with the same set of correctly recognized observations S ⊕ (f ′ ) = S ⊕ (f ). Therefore, we can restrict the search for a decision in the class F ′ (S) only.

Theorem 6. The linear SVM classification problem minimizes the loss criterion
Proof. The conditions (5) can be rewritten as ∀β, β ∈ S : The values ξ(β), β ∈ S do not depend on each other, so the minimum of their sum is achieved when every variable ξ(β) equals its lowest possible value. Let us find these lowest values for ξ(β) depending on if β ∈ S ⊕ (f ) or β ∈ S ⊖ (f ). If By definition of F ′ (S), |f (x(s))| ≥ 1. Then In this case, the lowest possible value for ξ(β) is 0. Then In this case, the lowest possible value for ξ(β) is 1 + |f (x(β))|. So, We still need to prove that for β ∈ S ⊖ (f ) Let us take β ∈ S ⊖ (f ). If y(β) = 1, then f (x(β)) < 0 and |f (x(β))| = −f (x(β)). So, If y(β) = −1, then f (x(β)) > 0 and |f (x(β))| = f (x(β)). So, Now I need to show that to prove that L svm (f, S) is an explanation criterion. The distance between the feedback of observations and the function value is defined here by the rule The only badness rule coincides with point-wise rule T pw . The formula Z(f ) = w(f ) 2 of the criterion is a regularization component: w(f ) is the gradient of the hypothesis f , and w(f ) 2 is the square of its norm. The functional Ψ which combines values of these two criteria is Ψ( This confirms that linear SVM for classification minimizes an explanation criterion.

Linear Support vector regression
The learner ( [5]) minimizes criterion The class of hypothesis is the class of all linear functions. The second component of the loss criterion is regularization, the same as in the SVM. The distance between feedback of an observation and the value of a hypothesis is defined through the function V : for α 1 = (≍ (ϕ(x 1 ) = y 1 )), α 2 = (≈ (ϕ(x 2 ) = y 2 )) ρ y (α 1 , α 2 ) = V (y 1 − y 2 ).
Then the AGC criterion scheme here coincides with the scheme for linear SVM for classification.
So, the linear support vector regression supports the main conjecture as well.

Support Vector Regression with Kernels
The learner is defined as in ( [5]). Suppose we have a set of basis functions H = {h i (x), i = 1, . . . , k}. We are looking for a hypotheses The loss criterion used here is Here the transformation x → h 1 (x), . . . , h k (x) from a n-dimensional space R n into k-dimensional space H(x) may be called focusing. Then the problem is reduced to solving a linear SVM regression in the transformed space. Thus, SVR with kernel supports the main conjecture as well.

Ridge Regression
The learner finds the solution in the class of all linear hyperplanes F = {f : f = wx + b}, and it has the criterion The first component of the loss criterion is regularization component, the same as in SVM, SVR.
Unlike SVR, in this case the distance on Y is y 1 −y 2 = (y 1 −y 2 ) 2 . While SVR does not count small errors, Ridge regression counts all errors, but small errors have low influence because of the square in the norm.
The explanation criterion for Ridge Regression is not any different from the criterion for SVR. The only difference between the learners is in the interpretation of Data Explanation Logic: the distances on Y are different.
Thus, Ridge regression corroborates the main conjecture too.

Neural Network (NN)
Let us consider single hidden layer NN for two class classification as it is described in [5]. First, the learner transforms n− dimensional metric space of inputs R into k-dimensional space Z using non-linear transformation; where δ(r) is delta function and g i are linear functions. Denote z(x) the vector with coordinates Z 1 (x), . . . , Z k (x) .
For each x ∈ R n the class is selected as C(x, G, F ) = arg max c f c (z(x)). The learner uses the loss criterion It is obvious that the loss criterion is a point-wise badness rule T pw .
The learner optimizes simultaneously parameters of the functions G and F. For selection of parameters of these functions the learner uses gradient descent, which is called "back propagation" in this case. The learner uses some additional stopping criterion.
So, the procedure does not have a focusing stage. If calculates loss for a given set of parameters, evaluates gradients by each parameter, and then updates parameters based on the gradients. After the stopping criterion is achieved, the algorithm outputs the decision with the lowest loss criterion.
The procedure has only two types of steps: 1. fitting, which includes • generation of the C(x, G, F ) hypothesis based on previous value of loss criterion and gradients • evaluation of loss criterion L nn (G, F, S). of the current hypothesis 2. optimal selection: selection of the hypothesis with the lowest loss criterion.
Thus, NN is also an abduction learner.

K Means Clustering
The learner is different from hierarchical clustering in that it does not combine clusters, rather, for each observation, it chooses the proper cluster. It is assumed that the distance on the domain of data points is Euclidean.
Here is the description of the learner from [5].
1. Given the current set of means of clusters M = {m 1 , . . . , m K }, each observation is assigned to the cluster with the closest mean.
2. The rounds of assignment of all observations are repeated until clusters do not change.
The learning happens when we search for the cluster for the given observed data point. Denote C(x) the assignment of a cluster to a data point x. Given the set of observed data points S x = {x 1 , . . . , x m }, K clusters with cluster centers M of the sizes {l 1 , . . . , l K } the procedure assigns a new class to an observed data point to minimize sum of all pairwise distances within each cluster where ξ, ζ ∈ S, and x k is mean of the k-th cluster. I use the form (8) to prove that the learner agrees with the main conjecture. Denote x 0 a data point, x 0 ∈ S x , which we need to assign a cluster on this step. As in the case of hierarchical clustering, we consider underlying dependence ϕ as a function from cluster index k to the observed data point x. There are K hypotheses H(x 0 ) = {h 1 , . . . , h K }. Each hypothesis h i has a single hypothetical case ψ(x 0 , i, ≍).
We assume, before current run of the learner, the clusters are already assigned to each observed data point besides x 0 . So, the run starts with the training set having observations S(x 0 ) = {β ∈ S : x(β) = x 0 }.
Let us define an explanation criterion equivalent with 8. There is only one badness rule with the alignment relation π(α 1 , α 2 ) = (x(α 1 ) = x(α 2 )), which says that we evaluate contradiction degree for each pair of formulas with the same cluster, regardless if it is an observation or a hypothetical instance. The alignment relation is symmetrical, therefore for each pair of formulas α 1 , α 2 which satisfies the condition, the pair α 2 , α 1 satisfies the condition as well. In effect, every pair is counted twice.
The deviation function is t(r 1 , r 2 ) = 1 2 r 2 2 and the deviation is calculated by formula δ(α 1 , α 2 ) = t(ρ x (α 1 , α 2 ), ρ y (α 1 , α 2 ) = 1 2 ρ y (α 1 , α 2 ) 2 For the recursive aggregation we use the averaging L 1 (A) again. The learner generates all hypotheses H(x 0 ), evaluates the explanation criterion for each of them and selects the hypothesis with the lowest value of the criterion. This is Basic training. Thus this learner corroborates the main conjecture as well.

Conclusions
Peirce considered learning from experimental data as "data explanation". Pragmatism, which he introduced, informally describes the logic behind this process as abduction inference. ML is an automated learning from data for practical applications. If Peirce is right, ML could be understood as an automated abduction inference. And indeed I demonstrated here that it is the case.
The criterion of data explanation is formalized here within Data Analysis Logic I defined. No statistical concepts were used in this formalization. The process of the criterion minimization is described as two standard procedures: Basic training and Wrapper. Combination of an explanation criterion and the procedure of its minimization makes an abduction learner. My conjecture here is that every ML learners is an abduction learner. This conjecture is corroborated on 13 most popular learners for classification, clustering and regression. Thus, ML may be called an automated abduction indeed.
The approach has important advantages over commonly accepted statistical learning paradigm. It allows to understand 1. real life learners in their variety; their differences and common features; 2. conditions for learning form finite data sets 3. common structure of classification, regression and clustering problems and their algorithms 4. regularization as an aspect of an explanation criterion.
The future development of this approach may include • Logical understanding of testing as necessary part of learning process.
• Logical approach toward data analysis beyond learning • Automated algorithm selection.
• Development of new algorithms based on the proposed ideas. One example of such novel algorithm, adaptive k-NN, is introduced here.