{'fraction_non_alphanumeric': 0.03391088190553927, 'fraction_numerical': 0.021518940377694504, 'mean_word_length': 4.4896130346232175, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 0, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Putting Humans Back in the Loop: A Study in Human-Machine Cooperative Learning

This paper introduces a novel approach to human-machine collaborative learning that allows for the chronically missing human learnability in the context of supervised machine learning. The basic tenet of this approach is the refinement of a human designed software model through the iterative learning loop. Each iteration of the loop consists of two phases: (i) automatic data-driven parameter adjustment, performed by means of stochastic greedy local search, and (ii) human-driven model adjustment based on insights gained in the previous phase. The proposed approach is demonstrated through a real-life study of automatic electricity meter reading in the presence of noise. Thus, a cognitively-inspired non-connectionist approach to digit detection and recognition is introduced, which is subject to refinement through the iterative process of human-machine cooperation. The prototype system is evaluated with respect to the recognition accuracy (with the highest digit recognition accuracy of 94%), and also discussed with respect to the storage requirements, generalizability, utilized contextual information, and efficiency.


Introduction
An important aspect of digital education relates to supporting the learner to acquire software development competencies. The main lines of research in this field include cost-effective simulation of programming environments [8,20], dynamic adaptation of e-training [9,36], human-machine interaction [31,34] and collaborative learning [21,22]. Recently, the research attention has been also devoted to developing specific digital teaching paradigms [5]. This paper makes a novel contribution to the field. It introduces an approach to human-machine -192 -collaborative learning that allows for the chronically missing human learnability in the context of supervised machine learning.
Deep learning has undoubtedly made a significant breakthrough in many scientific domains. One of the main reasons of the enormous popularity of neural networks is that theyat least in the manner usually practiceddo not require considerable domain expertise or human engineering [26]. The very term "learning" is somewhat misused in this context. Deep learning relates to the process of encoding statistical regularities from the training corpora into parameters, which operate by very different principles from those underlying human learning [38]. Modern deep neural networks may contain up to hundreds of millions of automatically adjusted parameters [26], and derive high-dimensional representations that are not interpretable by human. Therefore, although deep learning may result in very useful software artifacts, it does not contribute to human learnability of domain expertise. This paper 1 considers the question of bringing the human back into the learning loop. It proposes an approach to making the process of software development more explanatory to the practitioner, while keeping some of the existing advantages specific to supervised learning. The proposed approach is demonstrated through a real-life study of automatic electricity meter reading in the presence of noise.

Main Idea and Outline
This paper makes contributions along two research lines. First, it introduces a twostage approach to digit detection and recognition (cf. Section 2). The approach is cognitively-inspired to the extent that it integrates the dichotomy between the preattentive processing and the attentive processing that is present in the theories of human attention [7,13,24,31,32,35]. It is also parameterized, and the number of free parameters is small enough that the model is analytically tractable by human. To this extent, this approach is non-connectionist.
However, it cannot be assumed that this approach is per se generalizable to the target domain. This leads to the second research lineiterative refinement of the approach through human-machine cooperation. This research line derives from iterative and incremental software development practices that belong to the folklore of computer science [25]. However, it is particularly focused on supervised learning, and on what we refer to as "iterative learning loop", represented diagrammatically in Fig. 1. Each iteration of the loop consists of two phases (cf. Section 3):  Automatic data-driven parameter adjustment, performed by means of stochastic greedy local search over a training corpus.
 Human-driven model adjustment based on insights gained in the previous phase. The main idea is to allow the practitioner to take advantage of automatic datadriven supervised learningnot only with respect to parameter adjustment (which is the usual case), but also to gain additional qualitative insights into the target domain. Thus, the iterative refinement of the approach to digit detection and recognition is both machine-driven and human-driven. The number of iterations is not predefined, i.e., the model is refined until it satisfies external requirements (in this paper, we describe two iterations).
After the iterative learning loop is completed, the model is validated on a test corpus. The paper ends with the discussion and conclusion (cf. Section 4).

Two-Stage Approach to Digit Recognition
The theories of human attention acknowledge that selective processing of sensory information has an important role in human cognition [2,6]. However, they do not agree on the processing stage in which information are selected [32,35]. The early selection view assumes that incoming sensory information are filtered in a processing stage prior to the stage of semantic interpretation, and that only the selected information is interpreted [7,24]. The late selection view assumes that all sensory information is semantically interpreted, and that selection occurs in a later stage on the level of interpreted information [13]. A significant body of evidence supporting one or other of these views has been presented in this long-standing -194 -debate, which may imply that the early and late selections of sensor information do not necessarily exclude each other [32].
In line with this, we assume the view that selection of sensory information occurs in two processing stages (cf. [31]):  Pre-attentive processing serves as the basis for perceptual grouping,  Attentive processing allows for semantic integration.
In the considered real-life study of automatic electricity meter reading, the preattentive processing stage is devoted to detecting relevant numbers, i.e., rows of digits that represent rates, while ignoring irrelevant digits (e.g., an electricity meter serial number) and symbols. Each number detected in the pre-attentive processing stage separately undergoes the attentive processing stage, devoted to recognition.
Two-stage processing has already been applied in various approaches to the research problem of object detection and recognition. At the methodological level, these approaches usually apply data-driven statistically-based techniques, such as the Markov random field theory and maximum a posteriori principle [30], the AdaBoost algorithm [15,29], the support vector machine [1], neural networks [28], etc. (a more extensive overview is given in [15]). Symbolic approaches are applied significantly less often [17]. However, at the practical level, the production of representative and balanced training corpora is a challenging task, especially in such cases when surface manifestations of noise in image vary significantly. In contrast to the dominant trend, the approach introduced in this section is feature-based, but still nonconnectionist in the sense that it does not require an extensive training corpus.
To extract feature vectors that represent image segments, we refer to the normalized histogram of oriented gradients [10]. To estimate the similarity between the feature vectors, we apply the cosine similarity [12].

Feature Extraction
Let f be an image segment, and let ) , ( y x f be the intensity of segment f at pixel ) , ( y x . To compute the gradient of f at pixel ) , ( y x , we apply the horizontal and vertical Sobel filters: where the magnitude is approximated by: and the gradient direction is calculated as: Gradient vector y) g(x, is further decomposed along n chain code directions, i.e., it is decomposed along a set of n elementary vectors rotated in increments of n / 2 (where n is an input parameter), having its gradient direction ) , ( y x  approximated to the closet chain code direction. More formally, gradient vector y) g(x, is mapped onto a feature vector: in which all elements but one are equal to zero, and the value of the nonzero element is equal to the gradient vector magnitude ) , ( y x g : where M and N are input parameters. Each cell -196 -calculated as the sum of all feature vectors that represent the pixels belonging to the given cell, i.e.: and then each element in vector ) ( f  is normalized with respect to the entire segment, i.e.: where: , ) ( (13) and . An image segment is represented by a feature vector whose size (i.e., N M n   elements) is constant and does not depend on the size of the segment. This enables the comparison of segments of different sizes. Let ) (

Feature Vector Similarity
-197 -Since all elements in feature vectors are nonnegative, ) , ( 2 1 f f sim will always be in range   1 , 0the higher the score, the more similar the feature vectors.

Pre-attentive and Attentive Processing
The electricity meter reading is conducted in the following stages: (i) preparatory stage (i.e., image pre-processing), (ii) pre-attentive processing (i.e., number detection), and (iii) attentive processing (i.e., number recognition). The image preprocessing includes: The other two stages are described in more detail in the following subsections.

Pre-attentive Processing: Number Detection
The introduced approach is feature-based. Each digit In the pre-attentive processing stage, a sliding window is used to search through the image and perform early selection of relevant image segments. For each sliding window segment f , its feature vector ) ( f  is generated and compared to the ground-truth feature vectors in  . If the maximum similarity value of ) ( f  with each one of the ground-truth feature vectors is greater than the predefined threshold value 1  (which is also an input parameter), i.e., segment f is marked as potentially containing a digit, and added to set  .
It should be noted that the mapping of relevant digits in the input image onto segments in set  is not intended to be bijective. The size and step of the window are input parameters. Depending on their values, segments stored in  may overlap (i.e., they may contain the same digit), or a digit in the input image can remain undetected. To illustrate this, we deliberately selected non-optimal values of the size and step of the sliding window (i.e., the window size is smaller than optimal, and the step is greater than optimal). The result is shown in Fig. 2. For the purpose of presentation, the width and height of all images are scaled up by the factor of two, and the rectangles that designate segments are automatically generated by the prototype system. The grayscale input image is given in Fig.  2(a). The segments in  are depicted in Fig. 2(b). Some segments are incorrectly marked as containing a digit, while some digits were not detected at all. However, a segment that contains the entire number can be determined as the minimum rectangular segment that contains all segments in  , as depicted in Fig. 2(c). In addition, since an electricity meter may contain more than one row of relevant digits, after the window slides through the entire image, set  is partitioned so that each subset    i contains digits that belong to a separate number. For the purpose of this contribution, we assume that rows of digits are presented one below the other (which is often the case). We consider that segments i p and j p from set  are related (i.e., they contain digits belonging to the same number) if they overlap along y-axis or if there is another segment   k p such that k p is related both to i p and j p . This relation between segments is an equivalence relation, and it is used to partition set  : where each subset    i relates to a separate number. This is illustrated in Fig.  3(a,b). The grayscale input image given in Fig. 3(a) contains two relevant numbers, i.e., the electricity meter has two rates. The set of segments detected by the sliding window is partitioned into two subsets: 1  and 2  (cf. Fig. 3(b)). The image segment containing the first number is determined as the minimum rectangular segment that encapsulates all segments in 1  . Similarly, the minimum rectangle segment that encapsulates all segments in 2  relates to the second number.
-199 -In general, after the completion of the pre-attentive processing stage, a set of segments containing relevant numbers in the input image is extracted: Each of these segments separately undergoes the attentive processing stage.

Attentive Processing: Number Recognition
In the attentive processing stage, the late selection performed over the segments containing relevant numbers includes the following steps: . This segmentation algorithm is adapted only with respect to the threshold value used to merge segmentswe use a fixed threshold value  passed as an input parameter.
(ii) Segment filtering. We discard subsegments whose dimensions (relative to the size of the containing segment i num f ), black/white pixel ratios or height-to-width ratios are not in the expected ranges for a digit. It should be noted that the segment filtering conditions are not algorithmically-driven but rather based on the authors' qualitative and inherently limited insights into the domain problem. Therefore, they are subject to further refinement, as described in Section 3.
the segment is recognized as containing digit ) E.g., when two segments designated in Fig. 3(b) are passed to the attentive processing stage, the recognition results are indicated in Fig. 3(c). The digits after the decimal points are deliberately ignored, in accordance with the external requirements.

Iterative Learning Loop
The approach to digit detection and recognition introduced in Section 2 is subject to refinement through the iterative process of human-machine cooperative learning, combining automatic data-driven parameter adjustment with humandriven model adjustment (cf. Subsection 1.1).
The image corpora used in the iterative learning loop (and in the subsequent evaluation of the system, cf. Section 5) contain real-life electricity meter images with significant noise and incompleteness from various sources. Electricity meters are inconsistently illuminated, physically damaged (e.g., scratched glass) and obscured by dirt or dust. All the images were captured by naïve operators using standard Android-based phones. The training corpus consists of 955 images containing only one rate (i.e., one row of relevant digits and the surrounding context, similarly as in image given in Fig. 2). This corpus is used for automatic data-driven parameter adjustment. The test corpus, used for the purpose of evaluation, is described in Section 4.

Stochastic Greedy Local Search
The approach to digit detection and recognition is parameterized by a set of ten parameters (cf. Table 1) which are subject to data-driven optimization. The space of possible parameter assignments is too large for an exhaustive search. However, states in the search space are full assignments to all the parameters, which allows for applying a stochastic greedy local search algorithm. Similarly, as in deep learning approaches, the objective function is seen as a hilly landscape in the multidimensional space of parameter values [26], and therefore we apply an adapted hill-climbing algorithm with random restart [11,37]. The algorithm is specified in Fig. 4 as a higher-order function, and its main idea may be described as follows.

Output:
optimized instantiation of parameters for the fitness function:  The hill-climbing: 1. The greedy local search starts from a randomly chosen instantiation of parameters. We refer to it as the current instantiation x . is generated. Let ' x be the most fit instantiation in set ) (x g .

If '
x is more fit than x , then ' x becomes the current instantiation, and the algorithm goes back to Step 2. Otherwise, the current instantiation is selected as a candidate for optimal solution.
Multiple random restart: To reduce the probability of getting stuck in a local extremum, the hill-climbing is restarted from finite number of different, randomly selected instantiations or parameters. Each run of the hill-climbing algorithm generates a candidate for optimal solution. After multiple restarts of the algorithm are completed, the most-fit candidate is selected as optimal solution.

Mutation (local changes in the search space) Input:
instantiation of parameters: ), , , . , , , set of n instantiations of parameters.

Algorithm:
  G return for end We use the following input arguments to the hill-climbing algorithm.
(i) Number of random restarts is set to five.
(ii) Parameters. Not all parameters given in Table 1 are actually optimized. The dimension of the sliding window and the steps of the sliding window along both the axes were predefined in line with the external requirements (cf. the discussion point on efficiency in Section 4). In addition, the threshold value for digit detection in the pre-attentive processing stage is considered to be equal to the threshold value for digit recognition in the attentive processing stage. The complete instantiation of the fitness function is represented as a set of parameters marked with * in Table 1.
-203 -(iii) The fitness h of a given instantiation of parameters is defined as the number of training samples that are recognized correctly and completely.
(iv) The mutation operation is defined in Fig. 5. Mutation of a given instantiation of parameters ) , , , ( generates a set of n new instantiations, each of which differs from initial instantiation x in only one parameter value, i.e., mutation involves only local moves in the search space.

Learning through Human-Machine Cooperation
Each iteration of the learning loop includes automatic data-driven parameter adjustment, followed by human-driven model adjustment. We describe two iterations conducted in the study.

Iteration 1 -Automatic data-driven parameter adjustment:
In the first iteration, the parameters are automatically optimized based on the introduced hillclimbing algorithm and the training corpus. At the digit level, 94% digits are correctly recognized, 3.16% incorrectly recognized, and 2.84% not detected. The confusion matrix is given in Table 2. At the number level, 67.02% images are completely recognized. Table 2  Confusion matrix in the first iteration   0  1  2  3  4  5  6  7  8  9  ND Total  0  628  2  0  0  0  0  4  0  7  1  22  664  1  22 452  0  0  0  0  1  0  6  5  17  503  2  0  0  399  0  8  0  0  7  1  0  11  426  3  0  0  1  434  2  0  0  0  10  3  16  466  4  0  0  0  0  446  0  0  3  2  0  17  468  5  1  0  0  1  0  404  3  0  4  4  14  431  6  8  0  0  0  0  1  430  0  10  0  8  457  7  0  0  3  0  0  0  0  440  0  0  10  453  8  2  0  0  5  0  1  8  0  442  3  10  471  9  1  0  0  4  0  1  2  0  4  422 11  445  INS 78  19  15  15  29  7  23  6  95 34 -321 INSsegment incorrectly recognized as a digit; NDdigit not detected Iteration 1 -Human-driven model evaluation and adjustment: The analysis of the system's performance shows that, out of 4784 digits, 136 digits (i.e., 2.84%) are not detected. On the other hand, the number of segments incorrectly recognized as digits is more significant -321 segments (which equals to 6.71% of all digits)and affects the accuracy of the system more intensively. Thus, a new insight into the domain problem emerges, which would otherwise remain hidden because it is not directly related to the parameter adjustment: the segment filtering (cf. point (ii) in Subsection 2.3.2) is the critical point of the system. It is clear that the segment filtering conditions should be adapted accordingly. However, their modification is a trade-off. By tightening the segment filtering conditions, the number of segments incorrectly recognized as digits will decrease, but it can be also expected that the number of digits that are not detected will increase. Therefore, the conditions were modified by the authors with the intention to achieve a balance between these two confronting factors. In addition, this modification may affect the appropriateness of the parameter values automatically derived in the previous phase. To address these issues, the second iteration of the learning loop is started.

Iteration 2 -Automatic data-driven parameter adjustment:
In the second iteration, the parameters are again automatically optimized. The optimization resulted in the same instantiation of parameters as in the first iteration, and the confusion matrix is given in Table 3. Table 3 Confusion matrix in the second iteration  Tables 2  and 3, it can be derived that the total number of segments that undergone classification in the attentive processing stage was reduced from 4969 in the first iteration to 4753 in the second. I.e., the number of processed segments decreased for 216 ( 4753 4969  ). This set of 216 omitted segments can be divided into two subsets. The first subset contains 172 segments that were incorrectly detected as digits in the first iteration, but correctly rejected in the second iteration. The second subset contains 44 segments that were correctly detected as digits in the first iteration, but not detected in the second iteration (cf. Table 4). The first subset is dominant, which implies that the modification of the segment filtering conditions in the first iteration resulted in a more balanced relationship between the number of segments incorrectly recognized as a digit, and the number of digit that were not detected.
In addition, the recognition accuracy at the digit level slightly decreased: from 94% in the first iteration to 93.79% in the second. However, the recognition accuracy at the number level increased from 67.02% to 72.46%. This is not contradictorythe number recognition accuracy is increased due to the fact that the number of segments that were incorrectly detected as digits was significantly decreased as a result of the modification of the segment filtering conditions. These conditions can be further optimized in subsequent iterations, but the first two described iterations suffice to illustrate the proposed approach.

Evaluation and Discussion
The automatically calculated instantiation of parameters and the human-adjusted segment filtering conditions are evaluated on the test corpus containing 721 images containing only one rate (and the surrounding context). To avoid bias (e.g., training on the test data), the training corpus, described in Section 4, and the test corpus do not overlap. More precisely, it is not only that the training and test corpora do not include images of the same electricity meters, but they also do not include images of the same electricity meter types.  0  1  2  3  4  5  6  7  8  9  ND Total  0  406  6  0  1  0  0  2  0  8  2  10  435  1  1  340  0  0  1  0  2  0  3  0  36  383  2  0  0  320  0  1  0  0  10  1  0  7  339  3  1  0  0  272  0  3  1  0  14  15  13  319  4  0  0  2  0  339  0  0  1  1  0  14  357  5  2  0  1  1  0  316 12  0  1  3  18  354 trade-off between the accuracy and the efficiency of the system. The average processing time per image is 0.57 s (with standard deviation of 0.22 s) in the training phase, and 0.59 s (with standard deviation of 0.17 s) in the testing phase (measured on a standard personal computer).
(iv) Single-frame recognition. In the reported experiment, each electricity meter was represented by one single image. In a practical application of this technology (e.g., using an Android-based phone), multiple frames of an electricity meter would be captured and processed. Since each recognized digit is assigned a similarity score (cf. Eq. (19)), the captured frames can be evaluated, and the most appropriate candidate selectedwhich would additionally increase recognition accuracy.

Conclusions
This paper identified two separate but related contributions. First, we introduced a cognitively-inspired, non-connectionist approach to digit detection and recognition, in the presence of noise. Second, we proposed a novel approach to human-machine collaborative learning. The basic tenet of this approach is the refinement of a human designed software model, through the iterative learning loop, combining automatic data-driven parameter adjustment with human-driven model adjustment. This approach is demonstrated through a real-life study of automatic electricity meter reading in the presence of noise.
In the terminology of cognitive info-communications, automatic object recognition is referred to as an elementary cognitive capability [3,4], in contrast to the higher level cognitive capabilities such as affective computing [37], human augmentation and health monitoring [16]. However, the proposed approach is relevant to the field of cognitive info-communications in two respects. One of the fundamental cognitive capabilities that remained under-investigated in this field is learning from small sets of prior experiences. Our approach to digit detection and recognition tends to meet, although only partially, this desideratumits advantage is that it does not require significant training data, which is demonstrated in [18]. On the other hand, machine learning-based systems are usually developed in an extrinsic manner, i.e., a system is trained as a whole. The automatic adjustment of a large number of parameters leaves the practitioner out of the learning loopit neither allows for human learnability in the training phase, nor does it provide the practitioner with insights into the performance of individual subsystems. In contrast to this, we proposed an approach to human-in-the-loop supervised learning. To illustrate human learnability, in Subsection 3.2, we discussed the idea that the iterative learning loop enables the practitioner to recognize the segment filtering, as critically important and extends understanding. More generally, the iterative learning loop is intended to make the process of software development more explanatory to a practitioner, by enabling them to intrinsically develop and evaluate individual subsystems, while keeping the advantages specific to supervised learning. In the dominant trend of ever more complex systems, based on black-box machine learning techniques, making the underlying computational models more human-interpretable, is an important requirement for computer-aided education in computer science.