Instruction: You are one of the reviewers, your task is to write a review for the article. You will be given the title of the article, the number of the round in which the article is located, and your order among the reviewers.

Input:
Title: SUBSET-BASED STEREO CALIBRATION METHOD OPTIMIZING TRIANGULATION ACCURACY
Review round: 1
Reviewer: 3

Output:
Basic reporting: The author presents an approach to select the best subset of images to calibrate a stereo rig using the well-known library OpenCV for image processing. The author has implemented a routine to try different sets of calibration images and evaluated them in terms of several metrics. By using these metrics the author selects the best subset of images and final calibration.

The article is well written and structured and the literature review is concise and well linked to the publication.

The presented figures are of exceptional quality and help the reader understand the contents fully.
Experimental design: The author validates the approach with a single dataset of 264 stereo images (I expect them to be time-synchronised) and presents several calibration runs together with some metric values in particular cases of the runs.

Please note line 292 Jmdir is said to be an outlier if Jmdir > 25 mm, but figure 8 also suggests an outlier is any run with Jmdir < 15 mm

Looking at Table 2 results, given that the standard deviation of the plane is approx. 0.95 there is no statistical difference between any of the runs. I would suggest the author to validate the results using more precise methods.
In my opinion, there is a lack of information in terms of the geometry used in the experimental design. What is the expected calibration resolution? What is the intended working distance? At which distance was the calibration pattern captured in the images?

There is also another important detail: how does your method ensure that all the image are is covered at some point by a calibration point and avoid skewed or biased distortion models?
Are "good candidates" the points at the centre of the image (less distorted) or are they normally distributed across the image?

Figure 12 mentions a deviation from the nominal. What is the nominal value here? Has the standard deviation also been deviated from the nominal? That is not stated.

I personally do not see the value of the PCA data transformation. Could the author validate that with the reprojection error at some point?
Validity of the findings: I had a look at the images and they do not look sharp. I would encourage the author to use a harder support material for the checkerboard pattern. Cardboard can easily bend and provide a > 1 mm deviation inaccuracy.

The presented results are not statistically sound and controlled. I believe further work needs to be done from data gathering to better analysing it.
Additional comments: No additional comments


Instruction: You are one of the reviewers, your task is to write a review for the article. You will be given the title of the article, the number of the round in which the article is located, and your order among the reviewers.

Input:
Title: SUBSET-BASED STEREO CALIBRATION METHOD OPTIMIZING TRIANGULATION ACCURACY
Review round: 2
Reviewer: 1

Output:
Basic reporting: Approved with the corrections.
Experimental design: no comment
Validity of the findings: no comment
Additional comments: The article is appropriate for publication.


Instruction: You are one of the reviewers, your task is to write a review for the article. You will be given the title of the article, the number of the round in which the article is located, and your order among the reviewers.

Input:
Title: SUBSET-BASED STEREO CALIBRATION METHOD OPTIMIZING TRIANGULATION ACCURACY
Review round: 2
Reviewer: 2

Output:
Basic reporting: no comment
Experimental design: no comment
Validity of the findings: no cemment
Additional comments: Author revised the paper as suggestion and give clearly answer.


Instruction: You are one of the reviewers, your task is to write a review for the article. You will be given the title of the article, the number of the round in which the article is located, and your order among the reviewers.

Input:
Title: EXTRACTING AUTOMATA FROM NEURAL NETWORKS USING ACTIVE LEARNING
Review round: 1
Reviewer: 1

Output:
Basic reporting: Positive:
+ Embedding into existing research landscape well-described.
+ Relevant literature is covered at appropriate depth.
+ Approach idea clearly described.
+ Raw data shared.
+ Self-contained research contribution, including proposal of a new technique and its evaluation.
+ The writing is generally good.

Needs improvement:
- The text contains a certain number of grammar errors. These typically do not impair understanding, but make the reading of the article more tedious. For examples, see my annotated PDF, which also contains a few additional hints.
Experimental design: Positive:
+ Fits into the scope of PeerJ CS
+ Appropriately points out cap in existing knowledge
+ Investigation sufficiently rigorous
+ Sufficient detail for replication
+ Relevant, well-defined research questions

Needs improvement:
- While generally self-contained, the article does not explain the choice of the comparative approach for evaluation (passive learning). Specifically, it does not explain how this approach works, and why it is a suitable comparator for evaluation.
- The evaluation was performed on only one case. The considered case was chosen because it represents a typical "hello world" program for machine learning. There needs to be a discussion why not a more realistic neutral network was considered, and why only one.
Validity of the findings: Positive:
+ Underlying data have been provided
+ Discussion of limitations very welcome

Needs improvement:
- Since the evaluation is based on a single example, chosen due to its use as a "hello world" case, the conclusions about performance in abstract/introduction/conclusion have to be toned down. It's OK to summarize the results as positive or promising, especially with regard to the considered case. But it's somewhat far-fetched to say "we have confirmed that our DFA gives a better performance" without qualifications.
- I am not sure whether I can agree to the following statement: "The results show that all the learned DFAs perform well on the testing dataset, with the F1 score more than 50%." For a binary classifier, a F1 score close to 50% indicates a close-to-random performance. Please either explain this apparent contradiction, or weaken down the statement (note that negative/inconclusive results are accepted).
- In Section 4.4, the text on the comparison to the comparator tools needs to include more details and qualifications (also hard to see in table). How much better was the new approach? It might be a good idea to perform a statistical analysis (although not strictly requirement if performance gain is obvious).
Additional comments: I have some doubts about the general usefulness of the approach, notably this circumstance: "Note that,
when a counterexample is found, it may be not that the hypothesis is incorrect, but rather that the abstract model is not precise enough and needs to be refined." Without a way to tell both sources of errors apart, it might be hard derive meaningful information about whether the produced models are correct. Maybe the authors can comment on their ideas to address this concern.

However, since this issue is more about the usefulness/importance of the results rather than soundness, it did not inform my recommendation.


Instruction: You are one of the reviewers, your task is to write a review for the article. You will be given the title of the article, the number of the round in which the article is located, and your order among the reviewers.

Input:
Title: EXTRACTING AUTOMATA FROM NEURAL NETWORKS USING ACTIVE LEARNING
Review round: 1
Reviewer: 2

Output:
Basic reporting: The article discusses the application of active automata learning for
neural network classifiers. A deterministic finite automaton is
extracted from a trained network using the OSS library LearnLib. The
purpose is to better understand what is going on in the black-box of
the ML component. The evaluation is based on a MNIST classifier for
handwritten digits.  Different possible abstractions in order to apply
the L*-algorithm are proposed and evaluated. At the end visual
representation of the learned automata are presented and discussed.

Regarding presentation, the paper is well structured and easy to
read. I especially appreciated the examples guiding through the
different abstraction steps. The explanations regarding the
experiments should be improved as it is not always clear what data is
presented (see details below).
Experimental design: As the authors state, the idea of extracting an automata from a
trained neural network is not new. However, the field is only at its
beginning and the authors do not assume access to the internals of the
network and evaluate a few new ideas regarding the abstraction of
images. Hence, the work is novel and interesting. The main
contribution is the three-layered abstraction that enables the
processing of the images. Although, the selected images are rather
small.

For equivalence testing, I find the use of an automata, passively
learned from the training data, interesting. However, a weakness of
the approach is the use of testing for comparing the two automata. The
testing algorithms as offered by Learnlib are meant for implementing
equivalence oracles for black-box systems. However, you have both
automata available and can use a precise equivalence check on the
automata which can be done in polynomial time.  As a result of using
testing, you may miss more counterexamples as necessary and your
learned automata may be inaccurate.
Validity of the findings: The techniques and the evaluation are sound. Standard algorithms from
LearnLib are used and the abstractions also make sense. The evaluation
uses a standard benchmark and analyses the data from different
perspectives which gives interesting insights into the application of
L* for classification of images.
Additional comments: Hence, the strength of this work is certainly its novelty and the work on abstraction. A weakness is the
implementation of the equivalence oracle. Since I consider the points in favour as being more essential
I recommend publication.

Details:
L10 and 26: "Deep Learning is a new area of Machine Learning research". You start your article with a false statement. It is not new, but around since a while now.

L35: "and what they eventually learn in actuality" ==> and what they eventually learn actually

L43: There is also a newer survey that should be cited:
        Model learning and model-based testing. BK Aichernig, W Mostowski, MR Mousavi ,
       M Tappler, M Taromirad, Machine Learning for Dynamic Software Analysis: Potentials and Limits, 74-100

L48: symbol data ==> symbolic data

L195: It is the same sigma-safety formula than above. Replace, vm^k with sm^k.

L302: "And we believe 100 is enough for local vibration, so we suggest to set the  interval number n as 2." Is it?
        To me it is not clear how you support this believe. It could as well be that choosing a different interval number could result in more accurate automata. Did you try learning, e.g. with n=3?

L279: a output ==> an output

Table 3: Please, also provide a column with the number of total words, as you discuss how the total number decreases with
         increasing block size.

L363: Please, provide a more explicit explanation of the difference between wAcc, ... and dAcc. How exactly are these
          calculated. I did not immediately understand what "with respect to words" and "with respect to input data" means.
	  However, after some thought I figured it out, but you should help the reader here. Otherwise, the rest of the
	  discussion is lost.

L430-432: this explanation is a bit vague. What bounds are you referring here to? Is it just an implementation issue, i.e.
         incomplete learning or is it due to the abstraction? The equivalence oracle may be another cause. It is only based on
	 the training data and not on the trained network. Please, extend this discusion/conclusion, because it is quite
	 relevant how your approach relates to the true CNN.

L435: a easy task ==> an easy task

L446: for for ==> for


Instruction: You are one of the reviewers, your task is to write a review for the article. You will be given the title of the article, the number of the round in which the article is located, and your order among the reviewers.

Input:
Title: EXTRACTING AUTOMATA FROM NEURAL NETWORKS USING ACTIVE LEARNING
Review round: 2
Reviewer: 1

Output:
Basic reporting: All of my issues pointed out have been addressed satisfactorily.

In addition, beyond my initial check of the provided raw data, I had a closer look at the provided raw data file.

On the positive side, I can verify that the file contains the implementation source code.
Towards a possible improvement, there are two issues:
(1.) Usage instructions for the artifact are missing, I tried compiling the main projects as a maven build, but that lead to an error (see below).
(2.) Evaluation data (measurements) are either missing or not easy to find.
Both issues should be fixed for the final version.

C:\Users\XXX\Downloads\cs-54449-dfa-mnist\dfa-mnist>mvn clean build
[INFO] Scanning for projects...
[ERROR] [ERROR] Some problems were encountered while processing the POMs:
[FATAL] Non-resolvable parent POM for org.deeplearning4j:dfa-mnist:1.0.0-beta: Failure to find org.deeplearning4j:deeplearning4j-examples-parent:pom:1.0.0-beta in https://repo.maven.apache.org/maven2 was cached in the local repository, resolution will not be reattempted until the update interval of central has elapsed or updates are forced and 'parent.relativePath' points at wrong local POM @ line 5, column 13
Experimental design: All of my issues pointed out have been addressed satisfactorily.
Validity of the findings: All of my issues pointed out have been addressed satisfactorily.
Additional comments: The authors have addressed all of my issues thoroughly. With the exception of one issue with the provided raw data file that is probably easy-to-fix (sorry for not pointing it out earlier), I am happy to recommend acceptance of the paper.


Instruction: You are one of the reviewers, your task is to write a review for the article. You will be given the title of the article, the number of the round in which the article is located, and your order among the reviewers.

Input:
Title: FOBSIM: AN EXTENSIBLE OPEN-SOURCE SIMULATION TOOL FOR INTEGRATED FOG-BLOCKCHAIN SYSTEMS
Review round: 1
Reviewer: 1

Output:
Basic reporting: 1)	The writing of the paper should be improved for example
Section 4.4 Them selves.
Experimental design: 2)	It is said that the complex tasks are forwarded to the cloud. However, I don’t see any explanation for these kind of tasks and how are they linked to the blockchain network? It needs explanation in the paper.

3)	“Meanwhile, the communications within the BC network and with the fog layer are configurable”. What exactly this statement signifies. What exactly is the definition of communication here?

4)	It is said that the blockchain can be placed in Fog-layer as well as End user layer. But how do we decide this? Is there a possibility that BC layer act independently in the simulator, i.e., BC should not be placed in Fog-layer and End user layer?

5)	Similarly, it is said that the BC can be implemented in cloud? Is it reasonable to put BC nodes in cloud architecture?
Validity of the findings: 6)	There is an initiate_miner() function. If BC functionality is implemented in End-users, all the nodes will act as a miner or there is a possibility that few nodes act as a miner and few nodes functions normally? If that is the case, can we tune the number of participating nodes in the blockchain network?

7)	Finally, along with the time consumption analysis, authors should also perform the storage cost analysis for the transactions in FoBSim.
Additional comments: Opinion:
Minor Revision
Summary:
To validate an integrated Fog-Blockchain protocol or method implementation, before the deployment phase, a suitable and accurate simulation environment is needed. Such validation should save a great deal of costs and efforts on researchers and companies adopting this integration. Current available simulation environments facilitate Fog simulation, or BC simulation, but not both. In this paper, authors introduce a Fog-Blockchain simulator, namely FoBSim, with the main goal is to ease the experimentation and validation of integrated Fog-Blockchain approaches. According to the proposed workflow of simulation, different Consensus Algorithms (CA), different deployment options of the BC in the FC architecture, and different functionalities of the BC in the simulation are implemented. Furthermore, technical details and algorithms on the simulated integration are provided.
Strong Points
1)	The paper is well presented and organized. Moreover, the novelty of the work is presented correctly.
2)	The paper presents extensive and tabular comparison with state-of-the-art.
3)	The paper is technically sound and authors have also tried to explain the possible problems associated with their proposed design.

Weak Points
1)	The writing of the paper can be improved further.
2)	Few concepts and statements in the paper are confusing. They should be explained further.
3)	The experimental analysis of the simulator can be presented better.

Conclusion
To conclude, the paper is well presented and technically sound. Almost all the concepts are explained well however, the paper can be improved further by addressing the above-mentioned comments.


Instruction: You are one of the reviewers, your task is to write a review for the article. You will be given the title of the article, the number of the round in which the article is located, and your order among the reviewers.

Input:
Title: FOBSIM: AN EXTENSIBLE OPEN-SOURCE SIMULATION TOOL FOR INTEGRATED FOG-BLOCKCHAIN SYSTEMS
Review round: 1
Reviewer: 2

Output:
Basic reporting: Well-written, easy to follow.
Experimental design: Does not evaluate different fog computing infrastructures, geo-distribution, layers, etc.

I think this is the selling point of fog computing, and I would expect experiments that show the effect of fog computing on block chain applications. However, the authors only evaluate block chain applications in a fixed "fog" infrastructure, which may leave the readers a bit disappointed.

I'd suggest to add also experiments where a given blockchain application is tested how it behaves in different fog computing infrastructures. This would shed light on the issue whether a given blockchain application would perform well in different fog settings.
Validity of the findings: Findings seem valid, but are very limited.
Additional comments: While the article is well-written and provides a lot of details of the simulation system proposed, there are some fundamental questions that are not answered.

- Motivation:
Why is fog computing and blockchain a good match

- Blockchain technology:
Important concepts are left unexplained. For instance, I miss an explanation of private vs. public blockchains. The authors seem only to address public blockchains; but they also make references to private blockchains like Hyperledger Fabric.

- Evaluations without real application:
There are no real blockchain applications evaluated, only consensus protocols. It stays unclear how real applications would benefit (if at all) from fog computing. Further, as mentioned above, different fog infrastructures are not compared to each other, which would be interesing.

The authors use too many abbreviations, like BC, FC, CA, SC, ... which make the text sometimes hard to comprehend.


Instruction: You are one of the reviewers, your task is to write a review for the article. You will be given the title of the article, the number of the round in which the article is located, and your order among the reviewers.

Input:
Title: FOBSIM: AN EXTENSIBLE OPEN-SOURCE SIMULATION TOOL FOR INTEGRATED FOG-BLOCKCHAIN SYSTEMS
Review round: 2
Reviewer: 1

Output:
Basic reporting: The paper has addressed all previous comments.
Experimental design: The paper has addressed all previous comments.
Validity of the findings: The paper has addressed all previous comments.
Additional comments: The paper has addressed all previous comments.


Instruction: You are one of the reviewers, your task is to write a review for the article. You will be given the title of the article, the number of the round in which the article is located, and your order among the reviewers.

Input:
Title: FOBSIM: AN EXTENSIBLE OPEN-SOURCE SIMULATION TOOL FOR INTEGRATED FOG-BLOCKCHAIN SYSTEMS
Review round: 2
Reviewer: 2

Output:
Basic reporting: Typo in Figure 1 in the appendix: "Chose" --> "Choose"

In Figure 1, two states are named "Start the simulation". Are there two simulations?

It seems some figures in the appendix / supplemental material are fundamental to the paper. They are heavily referred to in the text. It feels awkward to have them in the appendix. To me, an appendix is something optional for the interested reader, and the reader should be able to fully follow the paper without looking into the appendix.

If you want to save space, there are other possibilities than moving important figures to the supplemental material-
Experimental design: The additional experiment is good (Use case 3).
Validity of the findings: no comment
Additional comments: I'm fine with the revision, most of my comments have been addressed.

However, the question of what material goes to the main paper and what goes to the appendix should be reevaluated.


Instruction: You are one of the reviewers, your task is to write a review for the article. You will be given the title of the article, the number of the round in which the article is located, and your order among the reviewers.

Input:
Title: TO TRUST OR NOT TO TRUST AN EXPLANATION: USING LEAF TO EVALUATE LOCAL LINEAR XAI METHODS
Review round: 1
Reviewer: 1

Output:
Basic reporting: Generally, there is no basic issue. English language is appropriately used throughout. Introduction, context and references are reasonably clear and straightforward. Structure and formatting seem reasonable. Figures are well displayed and clear-cut enough. At least one of the dataset is directly provided in the github link; the others are referenced.
Experimental design: The four metrics chosen appear to be well-motivated and within the scope.
Research question is well-defined. The paper simply provides a set of metrics for users to compare explainability.

The definitions of the metrics are straightforward enough. There is complex processing, transformation or manipulation that will become real impedance towards the rigor of the investigation.
1.	Fidelity is potentially overlapping with local concordance. However, by pointing out the instability of fidelity due to sampling, the author has been able to demonstrate the difference.
2.	Possibly minor issue: 1/2 is used in the definition of prescriptivity. In many contexts, thresholding is not strictly 1/2. The presecriptivity is describe w.r.t model f that outputs a scalar value rather than vector (hence so does the explanation model). This should not be a problem since generalization is probably easy to do.

Replication should be possible, given the link to the code.
Validity of the findings: The results appear valid, as in the graphs showing metrics are plotted as the metrics are defined. No real complex processing to complicate the validity. The main results seem to be direct comparison and observations of the resulting values of respective metrics.

Possibly important insights are present, especially the fact that different explanation methods such as SHAP and LIME do not agree with each other.

Underlying data have been used in previous researches, and likely to be reliable.

Conclusion is concise and clear.
Additional comments: No additional comments


Instruction: You are one of the reviewers, your task is to write a review for the article. You will be given the title of the article, the number of the round in which the article is located, and your order among the reviewers.

Input:
Title: TO TRUST OR NOT TO TRUST AN EXPLANATION: USING LEAF TO EVALUATE LOCAL LINEAR XAI METHODS
Review round: 1
Reviewer: 2

Output:
Basic reporting: The paper is well written, easy to follow and understand, and appropriately structured. The introduction and background sections provide suitable context for the work, and are generally well-referenced, incorporating several review papers in the area in addition to further useful contextual references and the key papers for the methods used. However, some important prior work specifically on metrics for explanations has been missed (details below). The figures are high quality and appropriate for illustrating the ideas and work presented, though some of the captions could be expanded a little (details below). The paper is self-contained and presents a suitable selection of results to demonstrate the utility of the explanation metrics.

Specific comments (line numbers refer to lines as labelled in the supplied paper draft):

- Line 30: the word “symbolic” is not necessary here – it could potentially add confusion for the reader with specific preconceptions about symbolic AI. I would recommend removing it and leaving “human understandable clarification” (and similarly where this term is used elsewhere).

- Line 76: there is a typo – “where” should be “were”

- Line 115: N(x) has not been introduced at this point in the paper, so should either be replaced with text describing it, or the definition of N(x) should be moved into this section

- Line 120-121: in the equation for LIME’s loss, I think the summation should be over z,z’ in N(x) (the neighbourhood of x), whereas the notation for the normal distribution has been used by mistake. Additionally, z’ should be defined explicitly as the “interpretable representation” per the original LIME paper – it’s a bit unclear what z’ means here and so what the pair z,z’ represents for the unfamiliar reader

- Lines 121-122: it is worth pointing out where this detail comes from as it isn’t, as far as I can tell, in the main text of the original LIME paper. The specific kernel I believe is the default in the LIME source code, but this kernel doesn’t have to be used (and indeed the choice of kernel can significantly alter results).

-  Line 131: mu should be defined explicitly in the text

- Overall sections 2.2 and 2.3 would benefit from some additional verbal description to help the reader interpret the equations. It is difficult to form an intuitive understanding of the equations’ meanings without some prior knowledge of these kinds of models, and a small amount of additional text description would help the more general reader. Additionally, some reference to the variants of the SHAP algorithm and mention of which variant was used in the current study should be added, with some explanation about the possible implications for the results.

- Line 149-151: I think this needs a bit of clarification – presumably it is the function f(x) that does not vary much in the region around x (for a feature assigned low importance)? At the moment it reads like x itself does not vary much in the neighbourhood of x, which doesn’t make sense to me… This section is good, though, and makes an important point about the interpretation of these different methods, despite them both being in the same family of techniques!

- Secton 3, starting line 177: I have some concerns about the terminology used to name the metrics. The definition of interpretability as pertaining to human understanding (per Guidotti et al. 2018) is fine, but I don’t think it’s appropriate to use “interpretability” as a name for a metric that only measures one small element of this (the explanation length), especially when the broader field of “interpretable machine learning” studies a whole range of different facets under the term “interpretability.” I strongly suggest renaming this metric to something less broad and more precise, and replacing the use of “interpretability” with a more suitable name throughout the paper. Line 188: similarly, “fidelity” has been used to describe the general property of the correctness of an explanation in other interpretable machine learning literature, rather than as the name of a specific metric assessing one aspect of this (e.g. the “faithfulness” metric in https://arxiv.org/abs/1806.07538 or Area Over Perturbation Curve, AOPC, in https://ieeexplore.ieee.org/abstract/document/7552539 ). Perhaps a more suitable name might be “local fidelity”? Finally, “stability” has been used to refer to explanations that change very little in response to small perturbations to the input – again in, for example, https://arxiv.org/abs/1806.07538 . This is a bit different from the author’s proposed metric, so perhaps a different name is warranted? Apologies if these requests seem a little pedantic – but I think it is important to try to maintain some consistency in vocabulary within the field.

- Line 192: were these instances from the training set or test set? Do the metrics report different distributions of values between training and test set instances?

- Paragraph starting after line 232, before equation 4: there is a superfluous “element of” symbol

- Figure 1: the figure could use a little more explanation in the caption, especially regarding the meaning of the arrows

- Figure 3: make explicit in figure legend that this is for the *same* input data point and same ML model – and add some explanation of why the output value is different (0.72 vs 0.68) for the two explanations (presumably due to using K=4 rather than the full feature set?) This figure also needs a little more description of the numbers in the visuals as, these are the default outputs of the methods and differ a bit in their presentation. This could be added in the main text or in the caption.

- Line 242: the comment on the reason behind LIME and SHAP’s popularity seems like speculation, so should be noted as such (other possibilities include that they both provided user-friendly Python implementations, provide visualisations, and have been incorporated into some companies’ XAI solutions).

- Line 245: Point of clarification - SHAP can be used on image classifiers without using superpixels as the interpretable representation. The mapping to an “interpretable representation” in this case is just the presence/absence of the original pixels, rather than of superpixels.

- Line 315: according to figure 5, the “not at risk” boundary should be 0.49, not 0.51

- Line 381: (also in the abstract) “even explain the wrong label” – what does this mean? I don’t think this is properly explained in the text (apologies if I’ve missed an explanation somewhere).
Experimental design: The experiments chosen to illustrate the utility of the proposed metrics and toolbox are appropriate and quite thorough. There’s always more that could be explored but I think the authors have chosen a useful set of experiments to illustrate the key concepts. They have used freely available datasets and tools so that their experiments can be reproduced, and no private data has been used inappropriately. As far as I am aware, these datasets have been cleared for open use and no ethical issues have been raised with any of them previously. I have a couple of comments and requests for some clarifications:

- Line 241, Section 4: we need more details about the ML models: what was the train/test split? What was the test performance of each model? How likely was each model to over-fit? Were the 100 instances presented in the figures all classified correctly? If not, was there a correlation between incorrect classification and stability/any of the other metrics? Would also be good to briefly mention the tooling used (scikit-learn?), with reference to the github repository)

- Line 260: the same number of samples were used in SHAP and LIME – can authors elaborate on the equivalence here? Given the methods work differently, does this make sense? It’s OK if this is an arbitrary decision, but should be noted as such if the equivalence is not really appropriate.

- Results showing very unstable explanations for NNs/SVMs: this is important and should be expanded on. It seems likely that the models are fitting a highly nonlinear boundary and this is affecting the stability results. However, it’s difficult to generalise the results given only a single NN architecture (MLP with 100 hidden neurons) was tested. A simple experiment that would help illustrate the behaviour of the stability metric with nonlinear models would be to measure it for NNs with increasing numbers of hidden neurons (starting from just 1, also monitoring model performance). I would request the authors perform this experiment for one or two datasets, as this would significantly improve the discussion of the stability metric and aid reader’s intuitions about its behaviour on more complex models.
Validity of the findings: The findings are a demonstration of the behaviour of the proposed metrics using a suitable variety of ML models and datasets. I have asked for a further experiment to expand on the findings for NNs in the previous section of the review. The conclusions are sound and well written. I have a few specific questions/comments:

- Line 279: results indicating prescriptivity > fidelity – are the scales of these metrics comparable like this, so that you can be confident saying the behaviour of the local explanation models is better at the boundary than locally to the data point? How does this change between points classified with high confidence and points classified with low confidence?

- Regarding the local neighbourhood of an input, N(x): the relevance of the local “fidelity” score is going to depend strongly on whether the local sampling method samples data points that are within the distribution of the original training data, or produces samples that are out of distribution. The authors note that the sampling method will affect the fidelity metric (potentially favouring LIME) but do not expand on this sufficiently in the discussion – especially the potential impact of generated samples being out of distribution and what this means for interpreting this metric.

- The authors note the limitation of their proposed stability metric – that it does not take into account feature ranking – but I think this is more serious than implied: it seems a severe limitation, especially for models with fewer features. It is possible that an explanation method could produce explanations with entirely opposite feature importance rankings yet receive a high stability score. This at least requires further discussion and should be highlighted more prominently when the metric is introduced. However, the metric as-is does reveal some issues with the explanation methods tested, so it is at least useful in its current form. A possible improvement to this metric might be to use something like Rank-Biased Overlap: http://dx.doi.org/10.1145/1852102.1852106 rather than Jaccard similarity – perhaps mention this in the discussion?

- The discussion of “prescriptivity” should include some mention of causality and the possibility of mis-interpreting results with predictive rather than causal models. The equation to generate the contrastive input also has the potential to create out-of-distribution data, and this should be noted and the implications discussed.

- Some metrics have already been proposed for XAI methods, for example Area Over the Perturbation Curve (AOPC) and “faithfulness” mentioned above. Though not for LLE methods specifically, these metrics should at least be mentioned as prior work. Even better would be to indicate how these metrics differ, or are similar to, the metrics presented in this paper.
Additional comments: Overall this is a useful and well-written paper addressing an important and under-studied topic in XAI. The provision of a Python toolbox implementing the proposed metrics is especially welcome. One final point: there has been some work in the area of metrics previously that is not mentioned by the authors, perhaps as it has focused on image classifiers/DNNs and saliency map explanations, rather than LLEs specifically (see mentions of AOPC and “faithfulness” previously; also the ROAR method: https://arxiv.org/abs/1806.10758 ). However, these papers make transferable points about metrics so their inclusion in the discussion/prior work seems important. I am aware that it is rather uncouth of me to suggest referencing my own work, but our paper “Sanity Checks for Saliency Metrics” https://arxiv.org/abs/1912.01451v1 seems particularly appropriate here as it proposes methods to assess the statistical validity of explanation metrics. I would not expect the authors to implement these checks in the current paper, but it should certainly be considered as further work (or, alternatively, an explanation of why the proposed sanity checks would not be appropriate should be given).


Instruction: You are one of the reviewers, your task is to write a review for the article. You will be given the title of the article, the number of the round in which the article is located, and your order among the reviewers.

Input:
Title: TO TRUST OR NOT TO TRUST AN EXPLANATION: USING LEAF TO EVALUATE LOCAL LINEAR XAI METHODS
Review round: 1
Reviewer: 3

Output:
Basic reporting: This paper presents a set of metrics for the evaluation of local linear explanations. In principle, evaluating the explainability of local linear models is an important problem. Unfortunately, it is hard to figure out the novelty of the proposed method in the currently submitted form.
Experimental design: - The authors use medical datasets: drug consumption, arrhythmia, breast cancer and so on.
- The authors define problem1 (P1) and problem (P2), and answer the questions using proposed method
- The authors define four metrics to evaluate local linear explanation, but the definition looks heuristic.
- The authors describe detailed information of four proposed metrics.
Validity of the findings: This paper suggests a new framework “LEAF”(Local Explanation evAluation Framework)  to evaluate the local linear explanations. LEAF consists of four metrics so that users can consider both Explaining single decision and Model development. LEAF reveals that either LIME or SHAP cannot ensure its robustness for every scenario, and suggests users apply such frameworks case by case.
Additional comments: Here are comments to authors. It would be good to revise the draft to improve the paper especially on weakness.

Strength
-  The authors define key questions about XAI methods, as well as answer the questions using the proposed method. It looks well organized.
-  This paper shows overall good presentations and its figures are easy to understand.
-  LEAF can evaluate such existing frameworks as LIME, SHAP.  Also, LEAF seems to be flexible and scalable for various models and datasets.

Weakness
-  LEAF looks somewhat heuristic. Suggested four metrics (fidelity, local concordance, stability, prescriptivity) are defined without theoretical proof. The authors only mentioned the reason why the four metrics are reasonable. It would be better to add more theoretical approaches.
-  As the authors mentioned in the Discussion, LEAF is applicable to local linear explanation methods. Though the authors raised universal questions about XAI, the suggested method deals with only two local linear explanation methods.
-  The authors did not use complicated deep learning models in their experiments. Deep learning has been widely used in many areas, but they only used a simple perceptron structure.


Instruction: You are one of the reviewers, your task is to write a review for the article. You will be given the title of the article, the number of the round in which the article is located, and your order among the reviewers.

Input:
Title: TO TRUST OR NOT TO TRUST AN EXPLANATION: USING LEAF TO EVALUATE LOCAL LINEAR XAI METHODS
Review round: 2
Reviewer: 1

Output:
Basic reporting: Many changes have been made to enhance the clarity of the text, addressing the valuable comments from reviewer 2.
Experimental design: As before, there is no complex processing that will need detailed technical attention (apologies for the typos in the previous review).
Validity of the findings: As before, the findings as they are do shed light into the behaviors of some methods tested.
Additional comments: The paper may contribute well towards the XAI community by providing more motivations to define the scope of explainability.


Instruction: You are one of the reviewers, your task is to write a review for the article. You will be given the title of the article, the number of the round in which the article is located, and your order among the reviewers.

Input:
Title: TO TRUST OR NOT TO TRUST AN EXPLANATION: USING LEAF TO EVALUATE LOCAL LINEAR XAI METHODS
Review round: 2
Reviewer: 2

Output:
Basic reporting: All comments addressed
Experimental design: All comments addressed - great to see the new neural network results!
Validity of the findings: All comments addressed
Additional comments: Thanks very much for taking the time to address the points raised in the reviews! All my previous points have been addressed very well.

The only minor point I would raise now concerns the new final paragraphs in Section 5 (Discussion), added to address my point about citing prior metric work in this area. I think the metrics mentioned - "faithfulness" (Alvarez-Melis and Jaakola 2018) and AOPC (Samek et al. 2017) - are not quite as specific to particular settings as the authors suggest. Faithfulness does not have to be used in a self-explaining loop - see e.g. the IBM AIX360 toolbox for a generic implementation: https://aix360.readthedocs.io/en/latest/metrics.html . Similarly AOPC does not necessarily have to be used for images & saliency map explanations - its definition is generic, though it has usually been applied to images/saliency maps. Finally, the reference Tomsett et al. 2020 is about the evaluation of the statistical reliability of explanation metrics, rather than describing any metrics itself.

This section just needs a small edit to clarify these points and ensure the references are incorporated correctly. The point about other metrics is not necessarily that they could be added to the toolbox as further work, but that these represent previous efforts at creating explanation metrics and so should be cited for completeness. The point in Tomsett et al. 2020 is that any proposed metrics for explanations need assessing to evaluate their statistical reliability - though this can certainly be left to future work.


Instruction: You are one of the reviewers, your task is to write a review for the article. You will be given the title of the article, the number of the round in which the article is located, and your order among the reviewers.

Input:
Title: A DUAL MODEL NODE BASED OPTIMIZATION ALGORITHM FOR SIMULTANEOUS ESCAPE ROUTING IN PCBS
Review round: 1
Reviewer: 1

Output:
Basic reporting: no comments
Experimental design: no comments
Validity of the findings: no comments
Additional comments: The manuscript has technical contributions. Moreover, it is improved significantly but I still have some concerns.
1.	There are many grammatical, typographical, and sentence construction errors here and there throughout the paper. In the following, I have listed some such problems. The authors are required to fix the rest of the similar problems and fix those before the resubmission.
•	Please use one convention throughout the documents for abbreviation, i.e., Printed circuit boards (PCBs) or printed circuit boards (PCBs).
•	Abbreviate one time and then use the abbreviation in the rest of the documents.
•	Use one convention for referencing or citing Algorithms 1 or algorithm1
•	There are many complex sentences in the updated version of the manuscript, please make them small and clear to understand the meaning, i.e., First sentence of the last paragraph of page 8.
•	Use the same pattern in the paper, i.e., use comma before every “and” if using at the start.

2.	Please compare your model efficacy with state-of-the-art model, i.e., when simultaneous Escape Routing is escaping of circuit pins simultaneously from inside two or more pin arrays
3.	Please provide or figure in which you compare Gurabi, Minos and Bonmin in terms of time and integrity, which will strengthen your selection.
4.	Please provide figures in the results section that show time consumption and memory consumption.
5.	Authors claim that they achieve 100% routability but statistically it is not the correct statement. You can write 99.9% but not 100%.
6.	Please plot some graphs to check the scalability of your proposed techniques.
7.	I suggest to please add flowchart also so it would be easy to understand your work.

Concluding Remarks:   This paper may be accepted after incorporating minor comments by the author


Instruction: You are one of the reviewers, your task is to write a review for the article. You will be given the title of the article, the number of the round in which the article is located, and your order among the reviewers.

Input:
Title: A DUAL MODEL NODE BASED OPTIMIZATION ALGORITHM FOR SIMULTANEOUS ESCAPE ROUTING IN PCBS
Review round: 1
Reviewer: 2

Output:
Basic reporting: 1.	INTRODUCTION states clearly what are the context, challenges and contributions
2.	Quite sufficient relevant works and their limitations are posed and well-organized for the problem formulation of this paper.
3.	Figures such as Figure 10 and 11 are suggested to use better quality in its resolution.
4.	In RESULTS AND DISCUSSIONS, result statistics of test cases are better to be made into a table for a clear view of comparison with others’ previous works.
5.	In INTRODUCTION and RELATED WORK, cross usage of abbreviations and full names are not preferred, such as PCBs vs. Printed Circuit Boards and SER vs. Simultaneous Escape Routing.
Experimental design: 1.	In DUAL MODEL NODE BASED ROUTING, algorithms are well explained and unambiguous.
2.	The algorithm was designed in a way to ensure 100% routability with its proper constraints.
3.	The lack of evaluation of the global routing model in both Medium Grid and Large Grid cases make the conclusion weak in Line 443-445 and Line 461-462.
4.	In Line 386-391 in RESULTS AND DISCUSSION, lack of provision of statistics from different solvers Bonmin and Minos as a proof of the advantages of utilizing Gurobi solver.
Validity of the findings: 1.	Not all cases provided in Wu and Wong(2013) were tested in this paper, it is hard to conclude that the proposed algorithm outperforms over the one in Wu and Wong(2013) as stated in Line 463-464.
2.	Different solver used than Wu and Wong(2013), the comparison of experiment results don’t have exactly the same base line.
3.	The paper claims significant improvement in both the memory consumption and runtime in comparison to the previous work Wu and Wong(2013), however, the machine specification used on NEOS server was not specified.
4.	The contribution stated in Line 90-91 is not convincing as over half of the cases are not tested for global routing model in Line 436-466.
Additional comments: The paper sounds promising in the beginning in INTRODUCTION and the background study is quite sufficient. The overall algorithm design is proper and unambiguous. However, there is a room for improvement in the experimental design. The experimental design makes the contribution of the paper unconvincing due to the incompletion of case testing on global routing and the test environment difference between the one in others’ work.

If academic version of Gurobi solver has a limitation in constraints, you may need to consider other solvers for a full comparison in both the local and global routing models instead of leaving the global routing model untested for over half of the test cases. Or it would be more convenient if you could borrow the commercial license from some other institutes. The lack of test results makes the conclusion and contribution really weak.
And the machine specification is also a concern when comparing the runtime with others.


Instruction: You are one of the reviewers, your task is to write a review for the article. You will be given the title of the article, the number of the round in which the article is located, and your order among the reviewers.

Input:
Title: A DUAL MODEL NODE BASED OPTIMIZATION ALGORITHM FOR SIMULTANEOUS ESCAPE ROUTING IN PCBS
Review round: 1
Reviewer: 3

Output:
Basic reporting: The research work is interesting and is organized well.
The manuscript require professional language editing.
Experimental design: The design and analysis is sound
Validity of the findings: Abstract and conclusion need to be rewritten for easy and clear understanding.
Additional comments: The research work is interesting and is organized well.
The manuscript require professional language editing.
Abstract and conclusion need to be rewritten for easy and clear understanding.


Instruction: You are one of the reviewers, your task is to write a review for the article. You will be given the title of the article, the number of the round in which the article is located, and your order among the reviewers.

Input:
Title: STANCE DETECTION WITH BERT EMBEDDINGS FOR CREDIBILITY ANALYSIS OF INFORMATION ON SOCIAL MEDIA
Review round: 1
Reviewer: 1

Output:
Basic reporting: The authors propose investigate the performance of LSTM, bi-LSTM and CNN neural models with BERT embeddings and stance feature for credibility analysis on text resources.

The paper is well structured and easy to follow. The challenged problem and the motivation to study this problem are described clearly. However the introduction is too lengthy describing the social aspect of the problem. Previous solutions in the literature are described well, however the difference of the proposed method and contribution over the previous work are not given clearly.

 The figures and the charts are structured appropriately.
Experimental design: The authors focus on the use of BERT embeddings and the similarity between the title and the content of the news text for credibility analysis. In a vast number of recent studies, we see that BERT has been used for different NLP related problems. So using BERT in credibility analysis is  inline with the recent efforts. However the contribution is limited since the BERT pre-trained model is directly used without any further training effort.

The use of similarity between the title and the content of the news text is interesting. However there are several drawbacks about the definition and the analysis of the feature. First of all, this new feature is considered to be about stance of the news, however the measured similarity may not be about the stance. Since the title and body texts are represented through the terms, the similarity may be related with the topic. As another drawback at this point, the details as to how the vector is constructed are not given.  As another limitation, it is not clear why this feature is considered and how effective it is. There is not experiment in the study which is designed to measure the effect of this feature particularly on the credibility analysis. All the experiments involve the BERT embeddings and this feature together.

Another limitation of the study is that a single data set is considered. With a quick search, one can find various data sets on credibility analysis.
Validity of the findings: The results are not obtained under cross validation. Therefore they are open to bias in partitioning of the data set for training, test and validation.

The authors present a performance comparison against the results reported on the same data set, however it is not clear how comparable they are due to lack of cross validation and also since the data partitioning used in the results reported in other studies may be different.
Additional comments: The authors challenge a contemporary problem and present the study clearly. The motivation and the social aspect of the problem are discussed a bit lengthy but It is an enjoyable manuscript to read.

Using BERT embedding is a nice approach following recent developments in the area. Using title/body text similarity is a good idea with potential. However the contribution of the feature to the credibility analysis is not analyzed well.

Unfortunately there are other limitations with the experimental setup. Since the experiments are not conducted with cross validation it is not clear how comparable the results of different settings and the results reported on previous work.  Use of just a single data set is also a limitation to generalize the results.


Instruction: You are one of the reviewers, your task is to write a review for the article. You will be given the title of the article, the number of the round in which the article is located, and your order among the reviewers.

Input:
Title: STANCE DETECTION WITH BERT EMBEDDINGS FOR CREDIBILITY ANALYSIS OF INFORMATION ON SOCIAL MEDIA
Review round: 1
Reviewer: 2

Output:
Basic reporting: no comment
Experimental design: no comment
Validity of the findings: no comment
Additional comments: The authors have proposed an approach to detect fake news using BERT embeddings and reported high accuracy scores. However, there are some issues in the paper that need to be addressed:

- The major issue is the use of the terminology ‘stance detection’. What the authors have done is not stance detection; at best it could be seen as a naive approach for clickbait detection. According to [1], “Stance detection aims to assign a stance label (for or against) to a post toward a specific target.” For more related works refer to [2], [3].

- The title is slightly misleading. Credibility analysis and fake news detection are two related but different tasks. The title needs to be modified to make readers aware of what to expect in the paper.

- “Misinformation or Information Disorder in the form of fake news is usually in the form of completely false information, photographs or videos that are intentionally created and distributed to confuse or misinform…” -  Firstly, this is not exactly true. Disinformation and fake news may not be completely false information. In fact, in most cases, these are statements, claims and narratives with some degree of truth, taken out of context to mislead and deceive readers. Images and videos may not be intentionally created for this purpose - again, genuine images may simply be taken out of context to deceive viewers.
Secondly, the structure of this sentence needs to be changed and made more understandable.

- The division of sections and subsections is very confusing and this needs to be fixed: The ‘Previous Work’’ section is presented as a subsection of the ‘Social and Cognitive Aspects’. I would suggest removing the ‘Social and Cognitive Aspects’ section, since there is a huge body of literature that needs to be explored and explained, if this section is to be included somewhere later on in the manuscript.

- The architecture of the model needs to be described in words as well, alongside the diagram.

- Connections need to be made to the pre-existing works in the credibility literature.

- Change all occurrences of ‘dis-information’ to ‘disinformation’.

- URLs in footnotes 4 to 13 can be shortened, or moved to the references.

- The manuscript needs to be thoroughly proofread to remove grammatical errors.

References:

[1] Sun, Q., Wang, Z., Zhu, Q., & Zhou, G. (2018, August). Stance detection with hierarchical attention network. In Proceedings of the 27th international conference on computational linguistics (pp. 2399-2409).

[2] Augenstein, I., Rocktäschel, T., Vlachos, A., & Bontcheva, K. (2016). Stance detection with bidirectional conditional encoding. arXiv preprint arXiv:1606.05464.

[3] Mohtarami, M., Baly, R., Glass, J., Nakov, P., Màrquez, L., & Moschitti, A. (2018). Automatic stance detection using end-to-end memory networks. arXiv preprint arXiv:1804.07581.


Instruction: You are one of the reviewers, your task is to write a review for the article. You will be given the title of the article, the number of the round in which the article is located, and your order among the reviewers.

Input:
Title: STANCE DETECTION WITH BERT EMBEDDINGS FOR CREDIBILITY ANALYSIS OF INFORMATION ON SOCIAL MEDIA
Review round: 1
Reviewer: 3

Output:
Basic reporting: no comment
Experimental design: (1) Please state whether the parameters of the BERT model used in training was fine-tuned or just fixed.
(2) Regarding the class label ratio of the used dataset, to better simulate the situations in real-world applications (fake news are few), it would be better to conduct experiments trying different settings of the class-label-ratio, such as 1:5 for moderately skewed label distribution and 1:10 for high skewed distribution. This can be done by experimenting with a single model (e.g., the best one).
(3) As the stance signal is crucial to the task, it would be informative to show the differences between the stance scores (cosin) of fake news and those of the true news in the dataset.
Validity of the findings: (1) It would be worthing doing a statistical test on the results, in order to show that the differences between the results of the baseline methods with the proposed method are indeed statistically significant. To be able to do this, the authors may consider applying k-fold cross-validation to the dataset so that it is possible to obtain multiple results from each baseline and the proposed method for performing the statistical test.

(2) It would be also worthing to do an error analysis to discuss and compare the errors made by the baseline methods and the proposed method, e.g., showing test examples that are misclassified by the proposed method and speculate why the errors may occur.
Additional comments: No additional comments


Instruction: You are one of the reviewers, your task is to write a review for the article. You will be given the title of the article, the number of the round in which the article is located, and your order among the reviewers.

Input:
Title: STANCE DETECTION WITH BERT EMBEDDINGS FOR CREDIBILITY ANALYSIS OF INFORMATION ON SOCIAL MEDIA
Review round: 2
Reviewer: 1

Output:
Basic reporting: The revisions in the organization and the writing are satisfactory.
Experimental design: The authors responded to my earlier comments satisfactorily for most of the items. Especially the new experiments under cross validation and analyzing the effect of stance improved the analysis part.

The only comment that have not been attempted is the experiments with additional data sets. The authors consider the results reported on Kaggle's Fake News data set and make a comparison over this data set. However ISOT's Fake news Dataset is also used in the previous studies.

https://www.uvic.ca/engineering/ece/isot/datasets/fake-news/index.php

Analysis on an additional data set is crucial to show that the success of the proposed method is not specific to a certain case. So I'd suggest including analysis and comparison on this data set as well.
Validity of the findings: There are several recent studies on BERT based fake news detection.

Some of them are as follows:

Zhang, Tong, et al. "BDANN: BERT-Based Domain Adaptation Neural Network for Multi-Modal Fake News Detection." 2020 International Joint Conference on Neural Networks (IJCNN). IEEE, 2020.

Kula, Sebastian, Michał Choraś, and Rafał Kozik. "Application of the BERT-Based Architecture in Fake News Detection." Conference on Complex, Intelligent, and Software Intensive Systems. Springer, Cham, 2020.

Kaliyar, Rohit Kumar, Anurag Goswami, and Pratik Narang. "FakeBERT: Fake news detection in social media with a BERT-based deep learning approach." Multimedia Tools and Applications (2021): 1-24.

Ding, Jia, Yongjun Hu, and Huiyou Chang. "BERT-Based Mental Model, a Better Fake News Detector." Proceedings of the 2020 6th International Conference on Computing and Artificial Intelligence. 2020.

In order to present the validity of findings in the paper, I'd suggest summarization such BERT related recent approaches and if possible comparing the reported results.
Additional comments: Hence I'd suggest

- including analysis on ISOT fake news data set

- including comparison with recent BERT based fake news detection approaches in the literature.


Instruction: You are one of the reviewers, your task is to write a review for the article. You will be given the title of the article, the number of the round in which the article is located, and your order among the reviewers.

Input:
Title: STANCE DETECTION WITH BERT EMBEDDINGS FOR CREDIBILITY ANALYSIS OF INFORMATION ON SOCIAL MEDIA
Review round: 2
Reviewer: 2

Output:
Basic reporting: N/A
Experimental design: The authors have largely addressed the comments regarding experimental design, by showing the results using both pre-trained and fine-tuned models and effects of different features used. However, there is still a lack of results on comparing settings with different class-label-ratios, as suggested in the second comment of the "Experimental design".
Validity of the findings: The authors have nicely addressed the comment regarding the statistics test. However, the second comment on an error analysis on the testing results is still missing.
Additional comments: I would suggest the author do an error analysis on the testing results (e.g. a discussion of three misclassified test examples), which can help undertand where and how the proposed method fails.


Instruction: You are one of the reviewers, your task is to write a review for the article. You will be given the title of the article, the number of the round in which the article is located, and your order among the reviewers.

Input:
Title: KNOWLEDGE DISTILLATION IN DEEP LEARNING AND ITS APPLICATIONS
Review round: 1
Reviewer: 1

Output:
Basic reporting: This review studies techniques which transfer the knowledge acquired by large deep learning models to smaller models, which can then be used in embedded and mobile devices.

The level of English is more than adequate, explanations are clear and accessible to a broad range of readers.

It mostly respects the PeerJ standard structure, with a few extra sections that makes sense for the contents of the overview. However, readers may appreciate some more subdivisions in the Survey section, which only has one subheading and one sub-subheading (I believe those should be at the same level instead, this may be a formatting errata). The acknowledgements section includes funders.

The review applies and is accessible to any deep learning practitioner, including those who may not be specialized in the topic but may want to embed a certain level of intelligent behavior in a small device, a situation where knowledge distillation techniques are of interest. This field has been reviewed recently but none of those reviews are published on a peer-reviewed journal as of now, so this would apparently be the first review of the topic in a reliable source, since the topic itself is also very recent.

The introduction of the manuscript introduces the concepts appropriately, but I think it is missing some examples as to what tasks can be achieved with deep learning in embedded/mobile devices (e.g. fitness tracking, sensor data compression?), since the main justification for knowledge distillation is the need of smaller deep learning models, but there is no explanation for what problems these models may solve.
Experimental design: The content of the article is well within the aims and scope of PeerJ Computer Science. The described methodology in order to collect studies and results seems appropriate and rigorous. It is also systematic, since it introduces an objective metric for the fitness of different algorithms to the problem, which takes into account the reduction in size as well as the preservation (or even improvement) of accuracy. The value of this metric is lower as performance in both aspects improves. The metric is relative to the sizes and accuracies of the models, and does not directly depend on the data used, but it is computed using the metrics reported by the original papers, so I am unsure about its ability to compare those models. The authors could justify briefly the level to which this metric is independent of the datasets used.

The survey seems diverse and comprehensive, all methods are sufficiently described and the explanations are put together well, including detailed information about the experiments and results of each study. There is, however, little to no visual aid to complement the textual explanations. I think a simple diagram outlining the main components of a deep learning-based knowledge distillation model (i.e. teacher, student, the flow of data and weights, or how the student is trained) would be very helpful to give the reader an intuition on what all these proposals have in common.

Cited sources are reliable in general, either from reputable journals and conferences or, at least, well-known papers on ArXiv.
Validity of the findings: The discussion of the results is sound, and several guidelines are provided on how to improve works in the topic. Some possible future directions are also mentioned and appropriately cited. The conclusions summarize the manuscript correctly and attempt to guide the novel reader on how to use these models.
Additional comments: In summary, my overall opinion of this paper is very good, but I believe some improvements could be made that would make it easier to read and comprehend. My suggestions are as follows: to extend the introduction with applications of deep learning in embedded devices, better subdivisions of the Survey section, and a diagram or two explaining the common points of the inner workings of these models.


Instruction: You are one of the reviewers, your task is to write a review for the article. You will be given the title of the article, the number of the round in which the article is located, and your order among the reviewers.

Input:
Title: KNOWLEDGE DISTILLATION IN DEEP LEARNING AND ITS APPLICATIONS
Review round: 1
Reviewer: 2

Output:
Basic reporting: The introduction could include some additional sentences to explain the main contributions and findings of the survey.

Section numbers in the last paragraph of the introduction do not appear.

For a survey, the background could be more formal, introducing key concepts and definitions. The authors could also detail the categories or perspectives for the survey analysis, such as inputs, algorithms, distillation/compression approaches, outputs, etc.

The title mentions “applications”, so I would expect a specific section summarizing current applications and others the authors suggest could be explored in the future. Some information is given, e.g., used datasets in each paper, but a section from the application perspective could be more practical for readers interested in particular domains.

Some specific sentences that authors should clarify are:
- Section “survey”, line 136. Did the authors exclude papers not presenting evaluation metrics or they were only discarded from the comparison?
- Section “survey”, line 142. Every neural network -> every deep neural network
Experimental design: The survey methodology to search, select and summarize the papers should be improved. The authors only use one source (Google Scholar), so many relevant papers could be missing. It is not clear if the search strings are independent or not. The number of papers found, filtered out and finally selected should be also indicated. Usually, exclusion and inclusion criteria are established to clearly state the reasons why papers are discarded and selected. Current quality criteria seem pretty subjective, i.e., which are minimum acceptable citation count, or which are the “relevant” journals and conferences. All this information is necessary for replicability.

Reporting of each paper is quite complete, but it is not easy to understand how the authors have organized the paragraphs of each category (soft labels, transformation). Both sections are large to read, so the authors could think if a subdivision would fit, e.g., based on the application, specificity (agnostic or depend on network architecture), purpose of the knowledge distillation process…
Validity of the findings: The authors propose a metric to compare knowledge distillation techniques, but it is not evaluated for any of the surveyed techniques. Having a new metric could be very useful for researchers and adding a short study showing how it is computed and interpreted for a subset of techniques would add value to the paper.

The authors compare and discuss the distillation scores obtained by different techniques as reported in the original publications. However, it is not clear if all these techniques are comparable, i.e, do they comprise the same input deep learning? I guess not, so averaging or comparing achieved reduction and accuracy improvement is a bit risky. The authors could try to extract some common behaviors among techniques depending on the targeted architecture, dataset/application, etc.
Additional comments: None.


Instruction: You are one of the reviewers, your task is to write a review for the article. You will be given the title of the article, the number of the round in which the article is located, and your order among the reviewers.

Input:
Title: KNOWLEDGE DISTILLATION IN DEEP LEARNING AND ITS APPLICATIONS
Review round: 2
Reviewer: 1

Output:
Basic reporting: This overview paper studies techniques which transfer the knowledge acquired by large deep learning models to smaller models, which can then be used in embedded and mobile devices.

The level of English is adequate, apart from some minor grammatical errata that should be edited out: e.g. "The main objectives of this work" -> "The main objective of this work"; "Also, the paper discuss" -> "The paper also discusses"; "deep learning models use to run" -> "deep learning models are used to run" (or just "typically run").

The introduction and background sections have been sufficiently improved.
Experimental design: This study fits within the scope of the journal and there is no recent peer-reviewed review of the topic, to my knowledge. The overall structure of the paper has been improved and is now easier to follow. Diagrams have been added to complement method descriptions.
Validity of the findings: no comment
Additional comments: Overall, the paper has received many improvements and the contents are now well organized and present the whole picture notably better. The only modification I would recommend is a quick revision for grammar mistakes such as those I marked above, for greater clarity. Apart from that, I would consider that this overview reaches publication quality.


Instruction: You are one of the reviewers, your task is to write a review for the article. You will be given the title of the article, the number of the round in which the article is located, and your order among the reviewers.

Input:
Title: KNOWLEDGE DISTILLATION IN DEEP LEARNING AND ITS APPLICATIONS
Review round: 2
Reviewer: 2

Output:
Basic reporting: As suggested, the authors have extended the background to include some of the concepts used along the paper. I would also move the definition of online and offline distillation (including figure 2) that now appears at the beginning of section 5 (lines 173-176) to the background.

The authors have also included a new section summarizing the applications, as requested. This section contributes making the paper more complete.

New figures are quite useful to understand the background concepts and the categories used to classify the papers.

The manuscript still contains some grammar mistakes (a few are listed below), so proof-reading is highly recommended before publication.

- Section 1: The main objectives of this works is => are
- Section1: Also, the paper discuss => discusses
- Section 3: It’s purposes => its purposes
- Section 5: the two sub-category => subcategories
- Section 6: deep learning models use to run => are usually run ?
- Section 6: To be practically in use => To be of practical use
- Section 6: To be low latency => To have low latency
Experimental design: I am not fully satisfied with the answer given by the authors about the survey methodology. Even if they do not want to conduct a systematic literature search, the process followed to find and select the papers should be better explained in the manuscript. It seems that the survey is focused on recent works not included in previous surveys, so the covered period of time should be given. The names of the journals and conferences considered as “relevant”, as well as the minimum citation count, should be reported as well. Even though these criteria might not be valid for a systematic review, the reader has the right to know how the authors choose papers. Otherwise, the “overview” of the area is strongly biased by the authors’ interest on certain papers, but the reader is not aware of it.

The new organization of the survey section has greatly contributed readability.
Validity of the findings: The authors have successfully addressed my comments about the validity of findings.
Additional comments: None.


Instruction: You are one of the reviewers, your task is to write a review for the article. You will be given the title of the article, the number of the round in which the article is located, and your order among the reviewers.

Input:
Title: OBSERVER BASED ROBUST H∞ FUZZY TRACKING CONTROL: APPLICATION TO AN ACTIVATED SLUDGE PROCESS
Review round: 1
Reviewer: 1

Output:
Basic reporting: The paper use a clear English, has a strong technical background, a relevant but not complete literature review, with appropriate paper structure and figures, as well as clear definitions of all terms and theorems, and detailed proofs.

The paper should be considered for publication, after solving some elements/issues detailed in the following review.
1. The most important issue is the lack in explaining the novelty of your proposed method.
2. The next most important item is the positioning of your paper in the recent international context.
3. Some minor mathematical gaps were detected.
4. The least important points are related to the English.
Experimental design: The paper is according to the aims and scope of the journal.
The research question is quite well defined, but needs some further investigation.
The research complies with high technical and ethical standards.
The method is sufficiently described from the reproduction aspect.
Validity of the findings: Impact and novelty not sufficiently assessed, but existing. Some refinement are recommended in the review.
Details on the data used are not explicitly given.
Conclusion is appropriately stated.
Additional comments: Paper Review
As a summary of this review:
The paper uses a clear English, has a strong technical background, a relevant but not complete literature review, with appropriate paper structure and figures, as well as clear definitions of all terms and theorems, and detailed proofs.
The paper should be considered for publication, after solving some elements/issues detailed in the following :
1. Your most important issue is the lack in explaining the novelty of your proposed method.
2. The next most important item is the positioning of your paper in the recent international context.
3. Some minor mathematical gaps were detected.
4. The least important points are related to the English.
please find details in the file attached.


Instruction: You are one of the reviewers, your task is to write a review for the article. You will be given the title of the article, the number of the round in which the article is located, and your order among the reviewers.

Input:
Title: OBSERVER BASED ROBUST H∞ FUZZY TRACKING CONTROL: APPLICATION TO AN ACTIVATED SLUDGE PROCESS
Review round: 1
Reviewer: 2

Output:
Basic reporting: The paper is generally well written.
The authors should improve the list of references with papers like: https://doi.org/10.1016/j.sigpro.2014.09.011
This is needed especially since the problem of using TS fuzzy approach is not new in the case of the wastewater treatment processes.
The paper needs a serious proof reading. There are a lot of typos such as: Takagi Seguno fuzzy model.
The authors should also clearly present which parts are new and which parts are taken from the literature. For example: Lemmas are new or not because there is no citation attached to them.
Experimental design: The authors should test the robust solution in the case of some uncertainties. I recommend to consider some variation of the maximum specific growth rate or other parameters from the model. This is very important since this type of process are highly affected by uncertainties.
Validity of the findings: From my own works I didn't manage to have a observer for this model using only the dissolved oxygen concentration as measured variable (it would interesting to linearize the model in different operating points and test the observability of the linear model. I did that and the results are not promising). Maybe this method can offer this possibility.
Additional comments: In my opinion the work can be improved taken into consideration the above comments.


Instruction: You are one of the reviewers, your task is to write a review for the article. You will be given the title of the article, the number of the round in which the article is located, and your order among the reviewers.

Input:
Title: OBSERVER BASED ROBUST H∞ FUZZY TRACKING CONTROL: APPLICATION TO AN ACTIVATED SLUDGE PROCESS
Review round: 2
Reviewer: 1

Output:
Basic reporting: The authors provided an improved version of the manuscript. I still think that some other papers that were dealing with estimation in the case of wastewater treatment processes (using Kalman, Hinf, Luenberger etc.) should be included in the references section.
Experimental design: OK
Validity of the findings: OK
Additional comments: OK


Instruction: You are one of the reviewers, your task is to write a review for the article. You will be given the title of the article, the number of the round in which the article is located, and your order among the reviewers.

Input:
Title: OBSERVER BASED ROBUST H∞ FUZZY TRACKING CONTROL: APPLICATION TO AN ACTIVATED SLUDGE PROCESS
Review round: 3
Reviewer: 1

Output:
Basic reporting: OK
Experimental design: OK
Validity of the findings: OK
Additional comments: OK


Instruction: You are one of the reviewers, your task is to write a review for the article. You will be given the title of the article, the number of the round in which the article is located, and your order among the reviewers.

Input:
Title: X-ARCHITECTURE STEINER MINIMAL TREE ALGORITHM BASED ON MULTI-STRATEGY OPTIMIZATION DISCRETE DIFFERENTIAL EVOLUTION
Review round: 1
Reviewer: 1

Output:
Basic reporting: This paper proposes an X-architecture Steiner minimal tree algorithm based on multi-strategy optimization discrete differential evolution. The layout of the article conforms to the norms, the language is fluent, and no grammatical errors have been found.
Experimental design: The experimental design of this paper is complete and the experimental comparison is rich, and the experimental results well reflect the optimization degree of average wire length, optimal wire length, and standard deviation. Finally, the experimental part well reflects the effectiveness of the proposed algorithm.
Validity of the findings: The proposed algorithm can effectively reduce the wire length of X-architecture Steiner minimal tree and can obtain the best results currently.
Additional comments: The construction of Steiner tree is a hot topic in related fields, and I give relevant revision opinions after reading this paper.
(1) The flow of XSMT-MoDDE algorithm can be more detailed.
(2) What is the future research work on this work? It is suggested to give future research work in Conclusion Section of the paper.


Instruction: You are one of the reviewers, your task is to write a review for the article. You will be given the title of the article, the number of the round in which the article is located, and your order among the reviewers.

Input:
Title: X-ARCHITECTURE STEINER MINIMAL TREE ALGORITHM BASED ON MULTI-STRATEGY OPTIMIZATION DISCRETE DIFFERENTIAL EVOLUTION
Review round: 1
Reviewer: 2

Output:
Basic reporting: Global routing is an important link in Very Large Scale Integration (VLSI) design. As the best model of global routing, X-architecture Steiner Minimal Tree (XSMT) has a good performance in wire length optimization. For this reason, an X-architecture Steiner Minimal Tree algorithm based on Multi-strategy optimization Discrete Differential Evolution (XSMT-MoDDE) is proposed. This work proposes four strategies to optimize the DDE algorithm. The four strategies are somewhat innovative. As a whole, this paper has fluent grammar and rigorous logic, and has designed a wealth of experiments to verify the results.
After reading the paper, I give some additional revised opinions for reference: Firstly, are these optimization strategies only used for DDE algorithms, and can they be used by other evolutionary algorithms? Secondly, the mutation process in Figure 4 can be further explained. Thirdly, further investigation and citation for this paper are recommended for relevant references including VLSI routing.
Experimental design: This paper has verification experiments and comparative experiments, and the experimental design is more reasonable. Results of the comparative experiments prove that XSMT-MoDDE can get the shortest wire length so far, and achieve the better optimization degree in the larger-scale problem.
Validity of the findings: Under the premise of reasonable time complexity, the constructed XSMT obtains the current optimal wire length which is the most important objective of Steiner Minimal Tree.
Additional comments: none


Instruction: You are one of the reviewers, your task is to write a review for the article. You will be given the title of the article, the number of the round in which the article is located, and your order among the reviewers.

Input:
Title: X-ARCHITECTURE STEINER MINIMAL TREE ALGORITHM BASED ON MULTI-STRATEGY OPTIMIZATION DISCRETE DIFFERENTIAL EVOLUTION
Review round: 2
Reviewer: 1

Output:
Basic reporting: no comment
Experimental design: no comment
Validity of the findings: no comment
Additional comments: This work proposes four strategies to optimize the DDE algorithm. Results of the comparative experiments prove that XSMT-MoDDE can get the shortest wire length. The paper writing is fluent.


Instruction: You are one of the reviewers, your task is to write a review for the article. You will be given the title of the article, the number of the round in which the article is located, and your order among the reviewers.

Input:
Title: X-ARCHITECTURE STEINER MINIMAL TREE ALGORITHM BASED ON MULTI-STRATEGY OPTIMIZATION DISCRETE DIFFERENTIAL EVOLUTION
Review round: 2
Reviewer: 2

Output:
Basic reporting: The authors explained all the problems I have mentioned and revised the paper carefully, so I do not have any more question.
Experimental design: I think it is sufficient and this work is well written.
Validity of the findings: The problem of this work is well presented and make sense.
Additional comments: The authors explained all the problems I have mentioned and revised the paper carefully, so I do not have any more question.


Instruction: You are one of the reviewers, your task is to write a review for the article. You will be given the title of the article, the number of the round in which the article is located, and your order among the reviewers.

Input:
Title: STRATEGY TO IMPROVE THE ACCURACY OF CONVOLUTIONAL NEURAL NETWORK ARCHITECTURES APPLIED TO DIGITAL IMAGE STEGANALYSIS IN THE SPATIAL DOMAIN
Review round: 1
Reviewer: 1

Output:
Basic reporting: The writing of this article is particularly good to get what readers are concerned about. Thus, a positive recommendation would be provided under good expectations.
Experimental design: The authors divide the images using fixed sizes to create two databases. The partition of the database should reference the popular way in the deep learning community as well. For example, creating each corresponding steganographic image for each cover image and structuring it as a complete database, then allowing the designed algorithm to select appropriate proportions. Besides, giving a clear name to illustrate the combination of BOSSBase V1.01 and BOWS2 databases. Better to make a website based on GitHub to introduce the summarized databases.
Validity of the findings: The presented innovation meets the requirements that obtain a pleasant recommendation. Better to convert the current study to a benchmark for other researchers. Also, the strategy enclosing special elements might lack a scientific basis. Most of them might be empirical and there are difficulties to learn systematically. Of course, true actions depend on the authors.
Additional comments: Dear authors, so glad to read your paper regarding investigating the CNN (convolutional neural network) based methods applied to steganalysis and adding special modules to enhance the resultant accuracy.


Instruction: You are one of the reviewers, your task is to write a review for the article. You will be given the title of the article, the number of the round in which the article is located, and your order among the reviewers.

Input:
Title: STRATEGY TO IMPROVE THE ACCURACY OF CONVOLUTIONAL NEURAL NETWORK ARCHITECTURES APPLIED TO DIGITAL IMAGE STEGANALYSIS IN THE SPATIAL DOMAIN
Review round: 1
Reviewer: 2

Output:
Basic reporting: no comment
Experimental design: no comment
Validity of the findings: Novel enough for publication.
Additional comments: The authors propose a general strategy to improve existing deep learning based steganalytic methods. The strategy makes modifications in the stages of pre-processing, feature extraction, and classification using SRM filters, batch normalization, absolute value layer, and spatial dropout. The proposed scheme is novel and interesting. The manuscript is suitable for publication. Some comments are as follows:
- The proposed scheme makes modifications in several stages of deep steganalysis. The effectiveness of modifications in each stage should be discussed. For example, some experimental results should be given.
- It is better to summarize the contributions of the manuscript in the end of Introduction.


Instruction: You are one of the reviewers, your task is to write a review for the article. You will be given the title of the article, the number of the round in which the article is located, and your order among the reviewers.

Input:
Title: OPTIMAL 1-NN PROTOTYPES FOR PATHOLOGICAL GEOMETRIES
Review round: 1
Reviewer: 1

Output:
Basic reporting: The paper is well-written and self-contained. It is easy to read. I didn’t find any obvious grammar or spelling mistakes. All figures are clear and easy to understand. Detailed proofs about the upper-bound and lower-bound have been given.
Experimental design: The primary research is within the aims and scope of the journal.
The research question is well defined and interesting.
The paper is theoretically and technically solid.
The algorithm has been clearly described.
Validity of the findings: The theory and method is novel and interesting.
The code is provided.
Conclusions are well stated.
The related works are sufficiently discussed.
Additional comments: The authors investigated an interesting problem that how to find the optimal 1-NN prototypes for pathological geometries. An analytic method and a near-optimal algorithm are proposed. Theoretical analysis about the upper-bound and lower-bound is given. The results are illustrated in figures.

I didn’t see any major problems. But, I encourage the authors to address the following problems:
1.	As shown in Figure 3, the optimal solution (data points and decision boundaries) has an interesting structure. Could you give an intuitive explanation why the (colored) areas determinate by decision boundaries have such shape? Why they are not symmetric like the left and middle sub-figures? Can we adjust some hyper-parameters and get results with different visualization appearances?
2.	Could you introduce some classic solutions for this problem and discuss why their heuristics failed?
3.	It would be better to provide more and diverse results to prove the effectiveness of the proposed algorithm.

Minor problems:
1.	The equations between Line 61 and 62 exceeded the page length.
2.	Figure 2 shows a linear relationship between the circle t and the number of prototypes for 1st order and 2nd order settings. I think the line should be discrete. Right?


Instruction: You are one of the reviewers, your task is to write a review for the article. You will be given the title of the article, the number of the round in which the article is located, and your order among the reviewers.

Input:
Title: OPTIMAL 1-NN PROTOTYPES FOR PATHOLOGICAL GEOMETRIES
Review round: 1
Reviewer: 2

Output:
Basic reporting: Some notations are not clearly defined, for example the horizontal/vertical axes of Figure 1, Figure 3
Experimental design: no comment
Validity of the findings: The computational results are not sufficient. The optimal prototypes for a given dataset, as well as the comparison between different heuristic algorithms, are not reported.
Additional comments: The implications of the tighter bounds are not very clear. For example how these tighter bounds can be used to find the prototypes more efficiently should be reported. The comparison between the proposed algorithms and different heuristic methods for finding nearly-optimal prototypes should be reported as well.


Instruction: You are one of the reviewers, your task is to write a review for the article. You will be given the title of the article, the number of the round in which the article is located, and your order among the reviewers.

Input:
Title: OPTIMAL 1-NN PROTOTYPES FOR PATHOLOGICAL GEOMETRIES
Review round: 1
Reviewer: 3

Output:
Basic reporting: The paper addresses an important and interesting problem in machine learning and classification, specifically finding so-called ‘prototypes’ which are representatives of data sets, that aim to capture similar (or even better) information and can be used to speed up the computations in k-nearest neighbor. Generally the paper is well-written, self-contained, professional, and clear about what it is saying.
Experimental design: I find that it is somewhat lacking in terms of experimental design in a fashion that is directly relevant to prototypes in machine learning in a way that it claims. The very specific problem of concentric circles was already introduced in a previous paper by the same authors, and the main contribution is to do some exact computations for this one very specific and artificial example.  It seems to assume that all data would sit exactly on one of the concentric circles and with diameters precisely multiples, given by the parameter ‘t’ of a fixed size. This seems very unrealistic; It is more likely there would be concentric rings, with different widths, and probably some distortions. It is not at all clear what lessons might be drawn from the exact solutions, and be likely to have any relevance to real cases. Although the general idea of concentric circles is something that could reasonably appear it  seems very unlikely that it would be in such a regular structure. Hence, the computations are not matching the experimental design that would be needed. Also, if it were so regular than a reasonable learning system - maybe with the aid of kernels - would be likely to pick this up.
Also, my understanding is that the idea of prototypes is that they select a smaller set from a finite sample of values. Whereas in this case the selection seems to be from all of the points on each circle - essentially an infinite size initial sample.  I would have expected to see methods to select from a finite (randomised) selection of points on each circle - then the ‘exact’ solutions given could not be applied directly.

It also does not match the claim in the abstract about it heuristic algorithms failing, as there is no inclusion of experiements to show this.

Hence this example seems to be quite unrealistic, and not very informative, nor relevant to prototypes.  It is possible that the authors have some other motivation or context in mind, but then this should be made clear.
Validity of the findings: The limited findings are valid as far as I can tell.

The abstract claims that heuristic algorithms fail in this case, but gives no data or discussion to support this.
Additional comments: minor point:

Theorem 5: should “each circle must have a different number” be “may have”?
Seems it should be relaxing an equality, not adding an alldifferent constraint.


Instruction: You are one of the reviewers, your task is to write a review for the article. You will be given the title of the article, the number of the round in which the article is located, and your order among the reviewers.

Input:
Title: OPTIMAL 1-NN PROTOTYPES FOR PATHOLOGICAL GEOMETRIES
Review round: 2
Reviewer: 1

Output:
Basic reporting: pass
Experimental design: pass
Validity of the findings: pass
Additional comments: The changes have greatly improved the paper, and it now makes a lot more sense to me. It does seem reasonable that the high-quality answers for these special and artificial cases can help provide useful challenging benchmarks for more generally-applicable usually-heuristic methods.

Minor issues:

page 4 say ‘countable’ but this makes it sound like theory results on countability, and is presumably not meant? Might simply mean ‘finite’ ?


Instruction: You are one of the reviewers, your task is to write a review for the article. You will be given the title of the article, the number of the round in which the article is located, and your order among the reviewers.

Input:
Title: SPAMMER GROUP DETECTION AND DIVERSIFICATION OF CUSTOMERS’ REVIEWS
Review round: 1
Reviewer: 1

Output:
Basic reporting: The article covers an interesting topic, spam detection, with the use of a trending technique such as deep learning. It provides a good background on references and tries to solve the problem of spam in Non-English languages (Urdu). That being said, I must say that I am starting to get into this research area, and I am not a great expert in this field.
Experimental design: This paper shows a novel technique to detect spam using deep learning techniques. This is a technique that is getting lots of focus lately. However, even though I do know it is hard to find a data repository and on top of that significative data repository in urdu which is really hard, the goal of the study is to detect spammer groups and it has become a greater deal during the last five years (elections, pandemic, shopping, social networks, …) and the data is relatively old to infer that the technique is valid nowadays. For instance, and roughly speaking, we can think of an "Amazon Verified Purchase" to simplify certain computations. But, in general, I would appreciate if you can do or, at least, infer how it should work nowadays.

Finally, and this is only an example, Google has presented BERT and I do not know how it overcomes or facilitate a better detection of spammer groups than the latter. I would appreciate if you can tell something about that. For instance, the use of BERT with your technique, from the hyperparameters to the implementation of both techniques.
Validity of the findings: My main concern with the aim of the paper is the two branches of study. You propose two methods that I can see and understand both, but they seem like two different types of work and I don’t see the connection between these two. For instance, at the beginning of Section 4 I would appreciate a connection between both or in the conclusions. Regarding to the latter, I humbly think it should be rewritten to take into account the usefulness of both techniques working as they are and together. Right now, it has a lot of future work and almost an enumeration of Sections 4 and 5.
Additional comments: No additional comments