Instruction: You are one of the reviewers, your task is to write a review for the article. You will be given the title of the article, the number of the round in which the article is located, and your order among the reviewers.

Input:
Title: ACCELERATED IMPLEMENTATION FOR TESTING IID ASSUMPTION OF NIST SP 800-90B USING GPU
Review round: 2
Reviewer: 1

Output:
Basic reporting: Authors have improved the paper and satisfactory answered a part of comments. Still I think
1) There are some problems with readability
2) Motivation for the work is not clear enough
3) "Conclusion" section overstates the results

I think the readers will benefit if authors present a real-world use-case for their application. Particularly the following aspects should be highlighted.
1) How often the sources already determined to be IID and Non-IID should be retested.
2) What is expected proportion between IID and Non-IID tests?
3) What is the estimate of noise sources which would be tested simultaneously?
4) In which environment this tests will be executed? A lab checking for compliance? Super-computing center? End-users?
I.e. a real example proving why the performance is critical.
Experimental design: The work is missing by far the slowest component of the test. In the rebuttal authors tell that the missing bzip2-test takes 40 minutes on CPU while other 18 implemented tests only 9 minutes. Then, authors claim that bzip2-test is not important and can be easily omitted.

This even might be true, but the authors talk about the formal test defined by formal specification. Without bzip2 test the proposed software is not complaint with NIST standard and probably can't be used in practice (or the practical use of incomplete test should be explained in the introduction).

Furthermore, I am not aware about active attempts to make CUDA-based implementation of bzip2 algorithm and I am highly skeptical about prospect of such implementation. So while the work have merits in solving part of the problem, the authors should clearly state in the abstract/conclusion that only part of the test was optimized. And that the slowest part of the test was excluded from the presented study due to complexity. Authors might elaborate if they have some expectations on GPU-version of bzip2. Otherwise, they also must make it clear that fast CUDA version of the bzip2 is not under development and the full version of the test might never materialize.

Furthermore, the official code might be pretty inefficient. Authors report very litle difference between single-threaded and parallel (Open-MP) implementation. If properly parallelized and optimized, the existing C implementation might give a comparable performance to the presented GPU version. Considering the situation with bzip2 test, this might be a better way forward if the performance of IID test is critical.
Validity of the findings: I. The authors added table which indicates number of used registers as reported by NVIDIA Nsight. This, however, doesn't answer the point raised in the original review. In the reviewed paper a full page is used to count registers generated by NVIDIA compiler and make trivial occupancy analysis. The number of the generated registers alone (without further analysis) means very little performance-wise. So, there is no practical value in this analysis.

In other words: Authors propose a different method to distribute work between threads. They have measured performance increase. This is a valid point. The following occupancy analysis doesn't contribute to the work because
1) It might be possible to achieve performance improvement even if more than 32-registers would be required and the occupancy would be lower.
2) The proposed optimization also might result in performance degradation even if occupancy stayed the same.

Registers and occupancy play a significant role, but it is more complex than "higher occupancy is always better". The analysis is misleading, does not contribute anything to the paper, and should be removed or significantly extended.

II. My comment about GPU boost was also not handled properly. NVIDIA GPUs modulates the clocks according to the load and temperature. This modulation can't be treated as random Gaussian noise and unlikely can be compensated by averaging 20 executions. A better approach is required to avoid unpredictable fluctuations of performance which e.g. on Pascal architecture my reach 20%.

If the GPUBoost can't be disabled on particular GPUs, the authors should either properly quantify the effect on the measurements or/and try to mitigate it. If the work-load fully loads GPU, one way is to wait until GPU temperature reaches GPUBoost maximum threshold and only, then, start real performance measurements. Then, the GPU will reset clocks to the minimum to avoid overheating and the clocks would likely not fluctuate further significantly. In any case, it is important to monitor the GPU clock rate during the benchmark to ensure that all tests are executed under the same conditions.

III. The performance presented in Table 7 (formerly Table 4) completely changed in the reviewed paper (also performance of the not optimized parallelization methods 1 & 2 has changed). Also, proportions have changed. E.g. in the original paper parallelization method 2 was 10% slower than method 1. But in the revised paper it is insignificantly faster. This makes me doubt claimed 5% of measurements volatility. And overall, the authors should explain to the reviewers the change in the measurement procedure or why all timings are now different.

IV. In p. 18, ll. 505... the authors try to find optimal number of N (number of test iterations performed in parallel). Generally, the higher amount of parallelism should result in the better performance (unless you can avoid part of processing like in IID case). But authors claim the best performance is achieved with N=2048 and it decreases with larger values of N. I think it requires explanation of causes, not just a mere constatation. I also guess that the sampled values of N were taken not optimally. Authors want to perform 10,000 iterations. So, ideally 10,000 should be multiple of N. Of course, also the number of scheduled blocks ideally should be proportional to the number of SMs on the device. So, it might make a sense to consider holistically the GPU architecture (number of SMs and amount of available memory), N, and blocking/work-allocation problem.

V. In the conclusion the authors still overstate the achieved performance improvements. The stated ratio of "33" is the maximum achieved in one test scenario. In other tests the performance ratio is between 4 and 33.
Additional comments: p. 11, l. 304-308. I don't understand explanation of Step2. It would be nice to reformulate

Overall, to accept for publications, I think authors must
1) Provide a good use-case scenario
2) Remove discussion about the number of registers or provide appropriate ptx analysis to prove it relevant.
3) Either account for GPUBoost in benchmarks or give a valid estimate of performance fluctuations. 5% is a clear underestimate.
4) The absence of bzip2 test should be clearly stated in the abstract and conclusion. The achieved performance should not be exaggerated in the conclusion.


Instruction: You are one of the reviewers, your task is to write a review for the article. You will be given the title of the article, the number of the round in which the article is located, and your order among the reviewers.

Input:
Title: ACCELERATED IMPLEMENTATION FOR TESTING IID ASSUMPTION OF NIST SP 800-90B USING GPU
Review round: 2
Reviewer: 2

Output:
Basic reporting: no comment.
Experimental design: no comment.
Validity of the findings: no comment.
Additional comments: no comment.


Instruction: You are one of the reviewers, your task is to write a review for the article. You will be given the title of the article, the number of the round in which the article is located, and your order among the reviewers.

Input:
Title: ACCELERATED IMPLEMENTATION FOR TESTING IID ASSUMPTION OF NIST SP 800-90B USING GPU
Review round: 3
Reviewer: 1

Output:
Basic reporting: The authors made significant improvements over 2 iterations. They have further optimized their method and added many valuable details to the paper. The structure is clear and figures/explanations are well presented.

Authors addressed my comments and explained their vision on applications of the developed method. I am still have one question. I am not expert on cryptosystems and  I am curious about the following: Lets say we identified some source as non-IID at Windows operating system. Do we really need to recheck it also on Linux and other operating systems? Can it really be IID in Windows and non-IID in Linux? Can authors reference an example?
Experimental design: I am not convinced that the official multi-threaded NIST code is optimal as presented in rebuttal letter. The authors provide the following run-times for 8-bit GetTickCount test:
- Table 10 in the previous revision of the paper:
  638s for single-threaded execution without compression
  347s for multi-threaded execution without compression
- Figure 10 in the current revision of the paper:
  3510s for single-threaded execution with compression
  541s  for multi-threaded execution with compression

From these numbers I conclude that the compression test had significantly benefited from the multi-threaded execution, but less than 2x speed-up was achieved for all other tests. Consequently, I am still not completely sure that the selected approach is optimal for the goal. Instead I would rather recommend to improve CPU version. It should be possible to run compression test only conditionally as they do in the presented GPU-based approach. Considering little difference between single-threaded and parallel (Open-MP) implementations for the rest of the tests, I believe there is a significant space for improvements and a performance comparable to the presented by the authors could be achieved as well without GPUs. This would make adoption of a new method much easier.

Nevertheless, the work definitively have merits and describes an interesting optimization approach which might be used for other tasks. In a minor revision, I would suggest authors to summarize optimization strategy and suggest how this can be adapted to different problems. This is not prerequisite for publication from my point of view, but I believe it might contribute to the scientific value of the paper.
Validity of the findings: I am generally satisfied with with additions authors provided to handle GPU boost technology. However, the motivation in the following statement is not clear and might be incorrect: “It is analyzed that the difference in performance with or without GPU Boost is not significant because there are more integer operations than floating-point operations.”. Similarly to floating-point operations, execution of integer operations depend on the current clock. So, it is unclear why it should make any difference (the actual reason could be actually that the code is mostly memory bound and as I can see from the image attached to rebuttal letter the memory clock is not scaled with GPU boost). Anyway, authors either need to clarify this or remove the last part of the sentence. For the presented performance evaluation, it is important that GPU-boost doesn’t affect results much. The analysis why it is the case might be considered out of the scope of the paper.
Additional comments: I would suggest to improve Figure 10. It seems to use logarithmic Y-scale. However, the equidistant horizontal grid lines make an illusion that the scale is linear. I will suggest to make it more apparent that the Y-axis is logarithmic either by labeling the Y-axis or making grid lines non equidistant.


Instruction: You are one of the reviewers, your task is to write a review for the article. You will be given the title of the article, the number of the round in which the article is located, and your order among the reviewers.

Input:
Title: ACCELERATED IMPLEMENTATION FOR TESTING IID ASSUMPTION OF NIST SP 800-90B USING GPU
Review round: 4
Reviewer: 1

Output:
Basic reporting: My comments were addressed and I think the paper deserves to be published.
Experimental design: -
Validity of the findings: -
Additional comments: -


Instruction: You are one of the reviewers, your task is to write a review for the article. You will be given the title of the article, the number of the round in which the article is located, and your order among the reviewers.

Input:
Title: PERSIAN SENTIMENT ANALYSIS OF AN ONLINE STORE INDEPENDENT OF PRE-PROCESSING USING CONVOLUTIONAL NEURAL NETWORK WITH FASTTEXT EMBEDDINGS
Review round: 1
Reviewer: 1

Output:
Basic reporting: The article lacks the scientific soundness.
The structure, tables and figures must be modified (complemented, removed, added references into their source).
Experimental design: Methods are not clearly presented (pre-processing methods, fastText vectorization methods).
The only method is tested: no investigation on different classifiers (different their architectures, hyper-parameter values) or vectorization types (with different parameters), etc.
Validity of the findings: The novelty is questionable.
Too little information about the used dataset.
Having no details on the used dataset, used pre-processing methods, used fastText embeddings, it is hard to interpret the results.
Additional comments: There are my remarks:
1. The main problem here: the contribution of your work is not clear. You have solved the sentiment analysis problem with only one classifier (CNN) offered by Kim (not testing any other architectures or hyper-parameter values). The classifier was applied on the fastText embeddings (classic neural vectorization technique, that recently is more and more replaced with more sophisticated transformer methods as, e.g., BERT).
2. You emphasize the importance of pre-processing (Section 2.2), but provide no details on how pre-processing was performed in their experiments. Please, write what normalization, tokenization, stemming tools have you used or implemented yourself. Please, provide details on how accurate they are and how much impact the sentiment analysis task, etc.
3. The authorship of Figure 1 is not yours, please, put the reference.
4. Figure 2 is not very informative: please, provide the exact details on each block.
5. How categories in Figure 3 make the impact on your sentiment analysis task? Please, provide the statistics about the used sentiment dataset. How many instances were in the positive and negative classes after balancing? And if you remove some instances from the original dataset does Figure 3 still make sense?
6. Table 1, Figure 4, Figure 5 are not clear. You either must provide translations or remove table and figures from your paper completely.
7. The obtained very high f-score values (over 0.9) seem suspicious. Maybe your positive/negative splits had too little diversity within each class? It even makes the impression that training/evaluation was performed on the imbalanced dataset. Would you get similar results if you would randomly select other instances from the major class?
8. Without having the exact details about the used dataset, I cannot interpret the results and must accept your conclusions with caution.


Instruction: You are one of the reviewers, your task is to write a review for the article. You will be given the title of the article, the number of the round in which the article is located, and your order among the reviewers.

Input:
Title: PERSIAN SENTIMENT ANALYSIS OF AN ONLINE STORE INDEPENDENT OF PRE-PROCESSING USING CONVOLUTIONAL NEURAL NETWORK WITH FASTTEXT EMBEDDINGS
Review round: 1
Reviewer: 2

Output:
Basic reporting: 1. The use of English language is not proper.
2. Related work is not appropriate or missing.
Experimental design: 1. Experimental design is not suitable to evaluate the efficacy of the proposed approach.
Validity of the findings: Results are not evaluated properly. Further experimentation is required.
Additional comments: 1.	‘Related work’ is missing. Split the ‘Introduction’ into introduction and ‘Related Work’ for completeness.
2.	Line 94. Authors assume that the use of fastText would generate better results. In research, such assumptions are not appropriate. So either use multiple algorithms for this task to evaluate their efficacy or support your assumption with the findings from other research works.
3.	The fastText model uses the skip-gram models works in similar fashion to CBOW so why authors prefere fastText over CBOW, TF/IDF, GLoVe, etc. I strongly recommend the authors to use CBOW, TF/IDF and GLoVe and compare their performance with fastText.
4.	The contributions of the study are not highlighted properly. At the end of Introduction, describe the contribuions in a bulleted form for readability.
5.	The choice of using CNN is not justified. LSTM has proven to show better performance for machine learning tasks.
6.	The performance of the proposed approach is not evaluated. The best way to do that is to compare the performance of the proposed approach with SVM, logistic regression, GBM, and at least two state-of-the-art approaches.
7.	The problem of two class classification is simple. I suspect the performance of the proposed approach will degrade if more classes are added. So I ask the authors to perform classification on ‘positive’,’neutral’, and ‘negative’ grounds.
8.	English grammer and typos.
a.	Always use Sentence case for writing. Many sentences in the manuscript start with small alphaets.
b.	When using numerical values in the text, use them properly, e.g., Section 3.3 ‘’10-7’ to ’10-7’.
c.	Extensive English proof reading and correction is required by a native speaker.


Instruction: You are one of the reviewers, your task is to write a review for the article. You will be given the title of the article, the number of the round in which the article is located, and your order among the reviewers.

Input:
Title: PERSIAN SENTIMENT ANALYSIS OF AN ONLINE STORE INDEPENDENT OF PRE-PROCESSING USING CONVOLUTIONAL NEURAL NETWORK WITH FASTTEXT EMBEDDINGS
Review round: 2
Reviewer: 1

Output:
Basic reporting: no comment
Experimental design: no comment
Validity of the findings: no comment
Additional comments: Thank you for considering my previous comments.
My main concern is still about a focus on one specific language. Maybe you could add some recommendations for other languages on how they can benefit from your research.


Instruction: You are one of the reviewers, your task is to write a review for the article. You will be given the title of the article, the number of the round in which the article is located, and your order among the reviewers.

Input:
Title: PERSIAN SENTIMENT ANALYSIS OF AN ONLINE STORE INDEPENDENT OF PRE-PROCESSING USING CONVOLUTIONAL NEURAL NETWORK WITH FASTTEXT EMBEDDINGS
Review round: 2
Reviewer: 2

Output:
Basic reporting: No comment
Experimental design: No comment
Validity of the findings: No comment
Additional comments: Authors resolved all the issues.


Instruction: You are one of the reviewers, your task is to write a review for the article. You will be given the title of the article, the number of the round in which the article is located, and your order among the reviewers.

Input:
Title: CLASSIFIER UNCERTAINTY: EVIDENCE, POTENTIAL IMPACT, AND PROBABILISTIC TREATMENT
Review round: 1
Reviewer: 1

Output:
Basic reporting: The article was clearly written and professionally presented.
References were used properly.
Experimental design: Research questions were well defined, relevant, and meaningful.

Mathematical calculations were mentioned in a perfect way.
Validity of the findings: Conclusions were well stated and linked to the original research.
Additional comments: The manuscript was pretty impressive and they did a great job in explaining in a very beautiful manner.
I'm impressed and loved the content.


Instruction: You are one of the reviewers, your task is to write a review for the article. You will be given the title of the article, the number of the round in which the article is located, and your order among the reviewers.

Input:
Title: CLASSIFIER UNCERTAINTY: EVIDENCE, POTENTIAL IMPACT, AND PROBABILISTIC TREATMENT
Review round: 1
Reviewer: 2

Output:
Basic reporting: No comment
Experimental design: No comment
Validity of the findings: No comment
Additional comments: The authors proposed a metric for classifier uncertainty. I think the idea is important and novel. I have some suggestions to improve the study:
- Literature review are weak. The authors should add a substantial amount of related references to support their hypothesis and findings.
- The authors only tested their methods on a use case from Kaggle competition. It is not enough to convince the generality of the model. Thus I suggest the authors provide more use cases to make the work stronger.
- Evaluation metrics (i.e. accuracy, confusion matrix, ...) have been used in previous biological works with small dataset such as PMID: 33036150, PMID: 32942564, and PMID: 31987913. Therefore, the authors are suggested to refer more works to attract broader readership.
- The authors have not explained well on the classifiers that they used.
- Did the authors have some independent test on the results?


Instruction: You are one of the reviewers, your task is to write a review for the article. You will be given the title of the article, the number of the round in which the article is located, and your order among the reviewers.

Input:
Title: CLASSIFIER UNCERTAINTY: EVIDENCE, POTENTIAL IMPACT, AND PROBABILISTIC TREATMENT
Review round: 1
Reviewer: 3

Output:
Basic reporting: No comment. The manuscript is well written and concise.
Experimental design: Normally, the sample size (24) and the wide spread of sizes (from 8 to 350) might be a concern, but the code being bundled allows for further verification.
Validity of the findings: The model defined for uncertainity quantification has been shown to arise from logical inconsistencies in the existing metrics (Caelen distributions). Furthermore, a full discussion of the prior considerations is also present. The fact that there exists large variation in the uncertainity of published classifier metrics is surprising, however, the analysis is valid and coherently presented.
Additional comments: No additional comments


Instruction: You are one of the reviewers, your task is to write a review for the article. You will be given the title of the article, the number of the round in which the article is located, and your order among the reviewers.

Input:
Title: CLASSIFIER UNCERTAINTY: EVIDENCE, POTENTIAL IMPACT, AND PROBABILISTIC TREATMENT
Review round: 2
Reviewer: 1

Output:
Basic reporting: no comment
Experimental design: no comment
Validity of the findings: no comment
Additional comments: no comment


Instruction: You are one of the reviewers, your task is to write a review for the article. You will be given the title of the article, the number of the round in which the article is located, and your order among the reviewers.

Input:
Title: JOINT EMBEDDING VQA MODEL BASED ON DYNAMIC WORD VECTOR
Review round: 1
Reviewer: 1

Output:
Basic reporting: Literature is ok.
The hypothesis is ok.
The experimental section should be extended to more comparison and deeper statistical analysis.
Experimental design: More comparison and more statistical analysis is needed.
Validity of the findings: Use more public available datasets to conduct more tests and comparisons.
Additional comments: In general, the paper is very good written, but some issues must be improved:
1) Equations should have some interpunction in the end. A word where after equations should be written in small letters.
2) Colors should be standardized for all figures.
3) Charts are too small.


Instruction: You are one of the reviewers, your task is to write a review for the article. You will be given the title of the article, the number of the round in which the article is located, and your order among the reviewers.

Input:
Title: JOINT EMBEDDING VQA MODEL BASED ON DYNAMIC WORD VECTOR
Review round: 1
Reviewer: 2

Output:
Basic reporting: Introduction should also mention application in disabilities, e.g.:https://doi.org/10.1155/2016/3054258; gaming: https://doi.org/10.1109/TG.2019.2896017; chatbots: https://doi.org/10.1007/978-3-030-17705-8_8

Add sample, illustratory outputs to figures 1 and 2

Remove Figure 3 as its also part of Figure 1

Consider adding Figure 4 as part of Figure 1

Basic implementation is explained (generic scheme was provided in figures 1-5), but paper lacks the minimum technical details on the factual implementation. Supplement the text with a UML based use case diagram and an activity diagram (logical sequence of the code) explaining the implementation. You'll find instructions (if needed) here:
https://www.tutorialspoint.com/uml/uml_use_case_diagram.htm
https://www.tutorialspoint.com/uml/uml_activity_diagram.htm
Add all parameters, discuss optimizer parts.
Experimental design: Authors present an approach based on FasterCNN for image classification and Elmo for text processing, which is quite known solution, therefore I suggest the authors highlight or demonstrate other contributions in the algorithm part. Novelty aspect was not reflected in the abstract and introduction too.
The experiments should be contextualized better (the reader should not be left to assume that they will get their own conclusions). The experiments should be described more clearly (e.g. set up and carry out process, results in raw format, etc.). Add a statistical reliability analysis. Discuss the impact on precision vs resource intensiveness.
Validity of the findings: Results are comparable to those of other authors: https://allennlp.org/elmo
More vigorous discussion should be added clarifying what has been achieved and whats the difference (if at all)... Note, that a different application or a combination of standard methods is still not considered as satisfactory merit for basis of novelty required for scientific publication, so I'd suggest carefully revising such parts in the paper.
Additional comments: No additional comments


Instruction: You are one of the reviewers, your task is to write a review for the article. You will be given the title of the article, the number of the round in which the article is located, and your order among the reviewers.

Input:
Title: JOINT EMBEDDING VQA MODEL BASED ON DYNAMIC WORD VECTOR
Review round: 1
Reviewer: 3

Output:
Basic reporting: This paper provides a framework for VQA model using dynamic word vector.
The framework consists of several well-explored components.
The paper is well written.
The font of the equations needs to be double checked.
Experimental design: Extensive experiments are conducted, although the noverty is limited.
Validity of the findings: The research appears to be sound, the results are exciting for their potential as a research tool, the study will be of interest to PeerJ readers.
Additional comments: I believe that the paper merits publication in PeerJ. However, I would suggest the author to use Latex for the manuscript editing, as it is quite difficult to follow the equations in current status.


Instruction: You are one of the reviewers, your task is to write a review for the article. You will be given the title of the article, the number of the round in which the article is located, and your order among the reviewers.

Input:
Title: JOINT EMBEDDING VQA MODEL BASED ON DYNAMIC WORD VECTOR
Review round: 2
Reviewer: 1

Output:
Basic reporting: UMLs must be added to provide technical explanation of the working principle as was originally requested.
Experimental design: Requested explanation was provided.
Validity of the findings: Statistical analysis still needs to be more detailed. Reliability of results is still not clear.

Attached code does not work. Add a full working project! Were some parts cut from the model?
Additional comments: No additional comments


Instruction: You are one of the reviewers, your task is to write a review for the article. You will be given the title of the article, the number of the round in which the article is located, and your order among the reviewers.

Input:
Title: JOINT EMBEDDING VQA MODEL BASED ON DYNAMIC WORD VECTOR
Review round: 2
Reviewer: 2

Output:
Basic reporting: English has to be improved (the text contains spelling errors).
Literature review is ok. References to formulas, Figures (not invented by authors must be referenced).
The article structure is ok.
The contribution (the novelty) of the paper is not emphasized.
Experimental design: Research is within Aims and Scope of the journal.
Research questions are well defined.
Methods are not invented by the authors. They are formally (mathematically) described.
Validity of the findings: Novelty must be emphasized.
The results are close to the baseline. The authors must evaluate if differences are statistically significant before making the conclusions that their method is better compared to the baseline models.
Additional comments: You have considered remarks of the other reviewers and already improved your description. The work is interesting, the topic is important. Unfortunately, I still cannot say that the description is ready for publishing. There are my comments:
1.	“…but all existing models use static word vectors for text characterization. However, in the real language environment, the same word may represent different meanings in different contexts, and may also be used as different grammatical components.” (line 18-21). E.g., Elmo, BERT embeddings successfully tackle this problem: they solve ambiguity problems by vectorizing each word depending on its context. Thus, not all existing models use only “static word vectors”.
2.	You state that your model based on ELMo vectorization outperforms glove. However, it is not very surprising. ELMo considers the word order and uses LSTMs. It is based on more sophisticated technique compared to glove.
3.	You are using Faster R-CNN, ELMo embeddings and multi-head attention mechanism. All these techniques are not novel. You must emphasize the contribution of your work.
4.	You make the overview of different research works in the field; however, it is not clear how your work differs from the other.
5.	The formal description takes 5-6 pages. I do not have anything against math, but these formulas are not invented by you. You must add references, where needed; explain what parameter values have been set in your experiments and why.
6.	What is the point of presenting Figure 1-Figure 5? Aren’t they presenting the basic architectures? If I am wrong, you must emphasize the differences; otherwise, please, add references, where needed.
7.	The results in Figure 6 of the baseline and N-KBSN are very similar (~1% difference) it arises the doubts if differences are statistically significant. This also questions the validity of your Conclusions.
8.	The discussion section is vague, especially explaining the results related with the image data (Section 5.3).
9.	The grammatical errors (e.g., table8, secction); not unified writing style (ELMo, Elmo; Bert, BERT; biLSTM, bilstm) must be considered as well.


Instruction: You are one of the reviewers, your task is to write a review for the article. You will be given the title of the article, the number of the round in which the article is located, and your order among the reviewers.

Input:
Title: JOINT EMBEDDING VQA MODEL BASED ON DYNAMIC WORD VECTOR
Review round: 3
Reviewer: 1

Output:
Basic reporting: no comment
Experimental design: no comment
Validity of the findings: no comment
Additional comments: The authors partially considered my previous comments.

However, I still have not got the answer about if differences between their results are statistically significant. Your statement "Because the training data used for experiment is limited, the differences are statistically significant" is false: usually having the small dataset is a risk that differences even in several percentage points do not guarantee the statistical significance.


Instruction: You are one of the reviewers, your task is to write a review for the article. You will be given the title of the article, the number of the round in which the article is located, and your order among the reviewers.

Input:
Title: GAIT RECOGNITION USING A FEW GAIT FRAMES
Review round: 1
Reviewer: 1

Output:
Basic reporting: There is a difference between the title and the content of the text. The main content is described about the augmentation method, and the number of images is only expressed as a result. In the case that the point of claim is the number of images, the proposed network must have a structure that allows for time expansion and contraction.

For a limited number of gait frames, the author should refer to the following:
N. Akae, A. Mansur, Y. Makihara, Y. Yagi, ``Gait Recognition at One Frame per Second,'' In The 7th Int. Workshop on Robust Computer Vision (IWRCV2013), Jan. 2013.
Experimental design: It is quite possible that extracting skeleton features may be fail, but it is unclear how much failure occurred in the experiment.
Validity of the findings: no comment
Additional comments: No additional comments


Instruction: You are one of the reviewers, your task is to write a review for the article. You will be given the title of the article, the number of the round in which the article is located, and your order among the reviewers.

Input:
Title: GAIT RECOGNITION USING A FEW GAIT FRAMES
Review round: 1
Reviewer: 2

Output:
Basic reporting: This paper describes a method of gait recognition from a limited number of frames. For this purpose, the authors employ skeleton models in addition to silhouettes which are often used in gait recognition community as inputs for GaitSet network structure with horizontal pyramid mapping (HPM). Experimental results with CASIA-B show the effectiveness of the proposed method over horizontal flipping and CycleGAN.

Strength:
- Motivation of gait recognition from a limited number frames is well introduced and hence convincing.

Weaknesses:
- The authors misunderstand the notion of data augmentation.

- Important references are missing.

- Insufficient experimental validation

- The skeleton model is not well defined.

- Technical novelty is limited.

More details of the above-mentioned concern is as follows.

1. The authors misunderstand the notion of data augmentation.

The data augmentation usually refers to a technique to increase the diversity of your training set by applying random (but realistic) transformations such as image rotation. Hence, we cannot associate addition of a new branch with skeleton model with the data augmentation.

In addition, the reviewer is afraid that the general readers come up with an idea of increasing the number of input frames for gait recognition from a limited number of input frames regardless of training or test stage. This is because the problem setting makes the number of input frames limited and hence direct solution to this seems to be increase of the number of input frames, although this is different from the notion of data augmentation.

Therefore, the authors should clarify the contribution claims including but not limited to the above-mentioned points.

2. Important references are missing.

Because this study aims at solving gait recognition from a limited number of frames, the authors should cite more closely relevant work and also differentiate them from them.

For example, the following study tries reconstructing a gait energy image (GEI) from a limited number of frames and use the reconstructed GEI for recognition.

@article{Babaee_Neurocomp2019,
title = "Person identification from partial gait cycle using fully convolutional neural networks",
journal = "Neurocomputing",
volume = "338",
pages = "116 - 125",
year = "2019",
issn = "0925-2312",
doi = "https://doi.org/10.1016/j.neucom.2019.01.091",
url = "http://www.sciencedirect.com/science/article/pii/S0925231219301845",
author = "Maryam Babaee and Linwei Li and Gerhard Rigoll",
keywords = "Gait recognition, Gait energy image, Deep learning, Fully convolutional neural network",
}

In addition, the following study aims at the most extreme case, i.e., gait recognition from a single image by reconstructing a gait cycle.

C. Xu, Y. Makihara, X. Li, Y. Yagi, J. Lu, ``Gait Recognition from a Single Image using a Phase-Aware Gait Cycle Reconstruction Network,'' Prof. of the 16th European Conf. on Computer Vision (ECCV 2020), Glasgow, UK (online), Aug. 2020.

Moreover, the following family of gait recognition from low frame-rate video is also closely relevant with this work, since they also handle a limited number of input frames.

@inproceedings{Akae_CVPR2012,
		author =	{N. Akae and A. Mansur and Y. Makihara and Y. Yagi},
		title =	{Video from Nearly Still: an Application to Low Frame-rate Gait Recognition},
		booktitle =	{Proc. of the 25th IEEE Conf. on Computer Vision and Pattern Recognition (CVPR 2012)},
		year =	{2012},
		pages =	{1537-1543},
		address =	{Providence, RI, USA},
		month =	{Jun.}
	}

@INPROCEEDINGS{Guan_IWBF2013,
	author={Yu Guan and Chang-Tsun Li and Choudhury, S.D.},
	booktitle={Biometrics and Forensics (IWBF), 2013 International Workshop on},
	title={Robust gait recognition from extremely low frame-rate videos},
	year={2013},
	month={April},
	pages={1-4},
	doi={10.1109/IWBF.2013.6547319},
}

- Insufficient experimental validation

The authors made experimental validation only with a single gait database, i.e., CASIA-B. It would be better to do more comprehensive experimental validation., since there are many other larger-scale gait databases including but not limited to the followings.

@article{Hofmann_JVCIR2014,
	 author = {Hofmann, Martin and Geiger, J\"{u}rgen and Bachmann, Sebastian and Schuller, Bj\"{o}rn and Rigoll, Gerhard},
	 title = {The TUM Gait from Audio, Image and Depth (GAID) Database: Multimodal Recognition of Subjects and Traits},
	 journal = {J. Vis. Comun. Image Represent.},
	 issue_date = {January, 2014},
	 volume = {25},
	 number = {1},
	 month = jan,
	 year = {2014},
	 issn = {1047-3203},
	 pages = {195--206},
	 numpages = {12},
	 url = {http://dx.doi.org/10.1016/j.jvcir.2013.02.006},
	 doi = {10.1016/j.jvcir.2013.02.006},
	 acmid = {2565863},
	 publisher = {Academic Press, Inc.},
	 address = {Orlando, FL, USA},
	 keywords = {Acoustic gait recognition, Depth gradient histogram energy image, Gait energy image, Gait recognition, Multimodal fusion, Soft biometrics},
}

@Article{Takemura_CVA2018,
author="Takemura, Noriko
and Makihara, Yasushi
and Muramatsu, Daigo
and Echigo, Tomio
and Yagi, Yasushi",
title="Multi-view large population gait dataset and its performance evaluation for cross-view gait recognition",
journal="IPSJ Transactions on Computer Vision and Applications",
year="2018",
month="Feb",
day="20",
volume="10",
number="1",
pages="4",
abstract="This paper describes the world's largest gait database with wide view variation, the ``OU-ISIR gait database, multi-view large population dataset (OU-MVLP)'', and its application to a statistically reliable performance evaluation of vision-based cross-view gait recognition. Specifically, we construct a gait dataset that includes 10,307 subjects (5114 males and 5193 females) from 14 view angles ranging 0{\textdegree} -90{\textdegree}, 180{\textdegree} -270{\textdegree}.",
issn="1882-6695",
doi="10.1186/s41074-018-0039-6",
url="https://doi.org/10.1186/s41074-018-0039-6"
}

@Article{Xu_CVA2017,
author="Xu, Chi
and Makihara, Yasushi
and Ogi, Gakuto
and Li, Xiang
and Yagi, Yasushi
and Lu, Jianfeng",
title="The OU-ISIR Gait Database comprising the Large Population Dataset with Age and performance evaluation of age estimation",
journal="IPSJ Transactions on Computer Vision and Applications",
year="2017",
month="Dec",
day="21",
volume="9",
number="1",
pages="24",
abstract="In this paper, we describe the world's largest gait database, the ``OU-ISIR Gait Database, Large Population Dataset with Age (OULP-Age)'' and its application to a statistically reliable performance evaluation of gait-based age estimation. Whereas existing gait databases include only 4016 subjects at most, we constructed an extremely large-scale gait database that includes 63,846 subjects (31,093 males and 32,753 females) with ages ranging from 2 to 90 years old. Benchmark algorithms of gait-based age estimation were then implemented to evaluate statistically significant performance differences. Additionally, the dependence of gait-based age estimation performance on gender and age group, in addition to the number of training subjects, was investigated to provide several insights for future research on the topic.",
issn="1882-6695",
doi="10.1186/s41074-017-0035-2",
url="https://doi.org/10.1186/s41074-017-0035-2"
}

@Article{Uddin_CVA2018,
author="Uddin, Md. Zasim
and Ngo, Thanh Trung
and Makihara, Yasushi
and Takemura, Noriko
and Li, Xiang
and Muramatsu, Daigo
and Yagi, Yasushi",
title="The OU-ISIR Large Population Gait Database with real-life carried object and its performance evaluation",
journal="IPSJ Transactions on Computer Vision and Applications",
year="2018",
month="May",
day="30",
volume="10",
number="1",
pages="5",
issn="1882-6695",
doi="10.1186/s41074-018-0041-z",
url="https://doi.org/10.1186/s41074-018-0041-z"
}

3. Technical soundness of use of the skeleton model under the problem setting is unclear.

The authors introduce the skeleton model as a countermeasure for a limited number of input frames. The reviewer understands that inclusion of the skeleton model as inputs may increase representation capability and hence improves gait recognition accuracy. On the other hand, the reviewer does not find clear connection between the inclusion of the skeleton model and a limited number of frames. For example, the reviewer imagines that the inclusion of the skeleton model is just generally beneficial for various settings, e.g., clothing invariant gait recognition with a full gait cycle. So, the proposed method does not seem to be a specific countermeasure for this problem setting, and hence this reduces the technical soundness of this method.

4. The skeleton model is not well defined.

The authors just illustrate the skeleton model in Fig. 2 without detailed explanations. The reviewer actually does not understand what the skeleton model indicates and how the skeleton model is created. Since the addition of the skeleton model seems to be a key component of this paper, the not well-defined skeleton model is critical.

5. Technical novelty is limited.

The proposed network architecture is an extension of GaitSet for two-stream networks of silhouette and the skeleton model. The GaitSet and two-stream networks themselves are existing and well known modules, and the way of combination is also straightforward, and hence the technical novelty of this paper seems to be limited.

6. Minor editorial errors and suggestions for improvement.

- l. 32: Work (We et al., 2016) is irrelevant with criminal investigation. More suitable citation could be the followings.

@article{Bouchrika_FS2011,
	author = {Bouchrika, I. and Goffredo, M. and Carter, J. and Nixon, M.},
	title = {On Using Gait in Forensic Biometrics},
	journal = {Journal of Forensic Sciences},
	volume = {56},
	number = {4},
	pages = {882-889},
	year = {2011},
}

@inproceedings{Larsen_SPIE2007,
author = {Peter K. Larsen and Erik B. Simonsen and Niels Lynnerup},
title = {{Gait analysis in forensic medicine}},
volume = {6491},
booktitle = {Videometrics IX},
editor = {J.-Angelo Beraldin and Fabio Remondino and Mark R. Shortis},
organization = {International Society for Optics and Photonics},
publisher = {SPIE},
pages = {195 -- 202},
keywords = {Gait analysis, gait recognition, forensic medicine, biometric, recognition, joint angles, posture},
year = {2007},
doi = {10.1117/12.698512},
URL = {https://doi.org/10.1117/12.698512}
}

- Figure 1: This seems to be reuse from GaitSet paper. If so, the authors need to get reuse permission from publisher (e.g., through RightsLink).

- A term human appearance would be better to be replaced with a body shape, since appearance may be associated not only with the body shape but also with cloths color and textures.

- l. 61: learning based -> learning-based

- l. 78: all 11 probe views. This seems to refer to CASIA-B database. If so, it would be better to clarify it.

- figure -> Figure or Fig.
Experimental design: See the above basic reporting.
Validity of the findings: See the above basic reporting.
Additional comments: See the above basic reporting.


Instruction: You are one of the reviewers, your task is to write a review for the article. You will be given the title of the article, the number of the round in which the article is located, and your order among the reviewers.

Input:
Title: GAIT RECOGNITION USING A FEW GAIT FRAMES
Review round: 1
Reviewer: 3

Output:
Basic reporting: This paper proposes a robust method to improve the performance of gait recognition when only a small number of frames can be offered. Two most widely used image augmentation methods and combining people’s silhouettes and skeleton models as input are compared in this paper. It is an important problem for gait recognition since in the real world it is impossible to acquire sufficient gait frames for each person due to various factors. This paper is well organised. Literature references and sufficient field background/context are provided. Experiment results are convincing, and the conclusions are well stated. Throughout the whole paper, professional English is well used.
Experimental design: The research problem in this paper is well defined, relevant and meaningful. It is innovative because few methods have been proposed for gait recognition to investigate the problem of insufficient input data. Methods are described with sufficient details. The experiments are well designed, and the experiment results provided in this paper are convincing. Moreover, the authors provide the implementation codes that make the proposed methods reproducible.
Validity of the findings: This paper is innovative, since for gait recognition few methods have been proposed to investigate the problem of insufficient input data. Two most widely used image augmentation methods and combining people’s silhouettes and skeleton models as input are compared in this paper. One conclusion has been made that using human silhouettes and their skeleton models as the network input is the most effective way to improve the gait recognition performance when the input frame number is limited. Literature references and sufficient field background/context are clearly stated. All underlying data have been provided. They are robust, statistically sound, and well controlled. The experiment results support the effectiveness of the proposed method. Conclusions are well stated and connected with the proposed research problem.
Additional comments: There are some minor errors and typos, for example,
1) F_1, F_2, …, and F_3^’ in Line 177 should be marked on Fig.3.
2) In Line 203, it should be Section 1.1.


Instruction: You are one of the reviewers, your task is to write a review for the article. You will be given the title of the article, the number of the round in which the article is located, and your order among the reviewers.

Input:
Title: GAIT RECOGNITION USING A FEW GAIT FRAMES
Review round: 2
Reviewer: 1

Output:
Basic reporting: Although the manuscript is updated and get a better shape, the following concerns remain unsolved.

3. Technical soundness of use of the skeleton model under the problem setting is unclear.

Explanation provided in the response letter and the revised manuscript is still unconvincing for a countermeasure for a limited number of intput frames. The authors seem to explain the performance drops for a limited number of input frames and improve the performance by skeleton information. But the inclusion of skeleton information is not relevant with the shortage of input frames. So, it is still strange to me to include skeleton information in the context of gait recognition from a limited number of frames.

4. The skeleton model is not well defined.

The motivation of the skeleton model is described in the revised manuscript, but its actual contents are still not described (e.g., the number of joints, representation, etc.).

5. Technical novelty is limited.

Although the authors list up the contributions in the introduction section, the technical novelty concerns raised up in the first-round review are not addressed at all.
Experimental design: N/A
Validity of the findings: N/A
Additional comments: N/A


Instruction: You are one of the reviewers, your task is to write a review for the article. You will be given the title of the article, the number of the round in which the article is located, and your order among the reviewers.

Input:
Title: GAIT RECOGNITION USING A FEW GAIT FRAMES
Review round: 2
Reviewer: 2

Output:
Basic reporting: This paper proposes a robust method to improve the performance of gait recognition when only a small number of frames are available. It is a significant problem for gait recognition because in the real world it is impossible to obtain sufficient gait frames for each person due to various factors. Three strategies are proposed in this paper to approach this problem, including horizontally flipping silhouettes, generating more silhouettes using CycleGAN, and combining human silhouettes and skeleton models. This paper is well organized, and professional English is well. Literature references and field background/context are offered. Experiment results are convincing, and the conclusions are well stated.
Experimental design: In this paper, the research problem in this paper is well defined. It is innovative since few methods have been proposed for gait recognition to investigate the problem of insufficient input data. In this paper, methods are explained with sufficient details, and the experiments are well designed. The experiment results provided in this paper are convincing. Moreover, the authors provide the implementation codes that make the proposed methods reproducible.
Validity of the findings: This paper is innovative, since for gait recognition few methods have been proposed to handle the problem of insufficient input data. Three strategies are proposed in this paper, including horizontally flipping silhouettes, generating more silhouettes using GAN-based networks, and combining human silhouettes and skeleton models as the network input. One conclusion is made that utilizing human silhouettes and skeleton models as input is the most efficient way to enhance the gait recognition performance when the input frame number is limited. Literature references and field backgrounds are clearly stated. All provided data are statistically sound, and well controlled. The experiment results support the effectiveness of the proposed method. Conclusions are well stated and related with the proposed research problem.
Additional comments: The authors have eased my questions. The paper can be accepted now.


Instruction: You are one of the reviewers, your task is to write a review for the article. You will be given the title of the article, the number of the round in which the article is located, and your order among the reviewers.

Input:
Title: EVALUATION OF RUST CODE VERBOSITY, UNDERSTANDABILITY AND COMPLEXITY
Review round: 1
Reviewer: 1

Output:
Basic reporting: The structure of the paper is appropriate, and the paper is easy to read and follow. The English language used in the paper is on par with only minor spelling mistakes, which can be mitigated with another round of proof-reading by the authors. The main contributions are appropriately highlighted in the introduction, making it easy for readers to determine an overview of the paper quickly.

-Please provide the reference for the statement in line 79 "...for example, software like Firefox, Dropbox, and Cloudflare use Rust."

-The manuscript is left-justified, not per the manuscript preparation instructions: "Left justify all text to the left margin. Do not 'full width' justify."

-The analysis for RQ1 should be supplemented with the reference that various lines of code metrics are the ones that are the most appropriate metrics for the programming language verbosity.

-On a similar note, what about the analysis classes for RQ2? Why only limit yourself to the methods?

-During the pipeline of the evaluation framework (Figure 1), please describe the differences of .json results after each step. It would be helpful to show an example of each JSON.

-It is described that compare.py is the main entry point of the source code files and results in the final .json with results. This is not evident from the pipeline in Figure 1 - here, compare.py is only one part of the whole pipeline, not the main script that runs other ones.

-How does tool rust-code-analysis analyze the C, C++, JavaScript, and other code? Is it not only meant for Rust code, as its name suggests? The quick review of the source code indicates that it analyses other languages, especially emphasizing this ability, so the readers do not get confused by its name.

-Be consistent with naming JSON (sometimes it appears as Json).
Experimental design: The experiment's design is appropriate, where various software metrics of the same software methods implemented in different languages are compared among themselves. The data used in the experiment and the source code used in the analysis are both publicly available. This makes the experiment repeatable.

-The main issue with the experiment is the lack of any statistical analysis of the results. There is enough data (enough analyzed software files) to make at least a basic statistical comparison. PeerJ CS is a high impacting journal, and thus, the methodology should be suitable. One could use ANOVA for repeated measurements or Friedman's ANOVA on every metric to find if the differences shown in the charts and table are due to chance (source of software or programmer making them) or they are statistically significant. If there are differences, use posthoc tests (i.e., Wilcoxon signed-rank with correction for multiple comparisons Holm-Bonferroni) to determine the real answers for posed research questions.

-Provide reasoning on why some of the programming languages were used in the experiment and others were not. Yes, it is mentioned that the implemented module only supports some of them, but why those. Are the chosen programming languages valid alternatives in some information systems (web assembly programming, etc.)? With this, you will introduce readers to Rust's typical applications and show what its main alternatives are.
Validity of the findings: The main issue with the findings was already mentioned in part about the experimental design - the lack of any in depath analysis of the results. Additionally, several other issues have to be addressed.

-The discussion on the differences of LLOC is a bit too narrow. Authors only explain the differences in this metric as the product of more types of logical statements available in Rust. In this case, Python would have the lowest LLOC count, which is not evident from the results. Also, the sheer number of available types does probably not correlate with the higher usage of those. If you argue that it does, please provide a reference or at least a viable justification. I would argue that there could be other reasons for more LLOC, which are fundamentally simple. For example, it could indicate that the logical statements are more elementary (do less in one call) than in other languages, so more are needed. Does this make code more verbose? Probably, but it's open for discussion. What if this is the key reason why the cyclomatic complexity and cognitive complexity are the highest with Rust?

-On a similar note, the number of methods and sum of arguments discrepancies could probably further explain Rust's method arguments' lack of default values.

-Also, is the higher count of methods the sign of better structure of the code? The answer to RQ2 suggests this. Again, this implies that using the one method without its variants (for different argument count) is superior. Please elaborate on this point.
Additional comments: The paper presents the results of an analysis of maintainability metrics and other software metrics software written in the programming language Rust. The authors took a publicly available repository of the different procedures written in various programming languages and compared them. The paper's main findings are lenient to the Rust programming language, as Rust results as the language in which software is written without too much complexity, is verbose, and is not too hard to maintain. The paper is derived from the final thesis, which (after my review and search) has never been published before.


Instruction: You are one of the reviewers, your task is to write a review for the article. You will be given the title of the article, the number of the round in which the article is located, and your order among the reviewers.

Input:
Title: EVALUATION OF RUST CODE VERBOSITY, UNDERSTANDABILITY AND COMPLEXITY
Review round: 1
Reviewer: 2

Output:
Basic reporting: - In section Introduction, the maintainability characteristic should be linked to well-known software quality standards such as ISO 9126, ISO 25010; they also provide the sub-characteristics for maintainability, allowing for a finer grained approach
- Introduction section could also use more recent references, as there exists a lot of post-2017 research on the topic.
- Rephrase line 46-47 to eliminate repetition
- Revise reference on line 101
- Line 125: "open-source algorithms" - perhaps this needs a bit of clarification; are they open-source algorithms or are the algorithms implemented in open-source code?
- Table 4 - the meaning for some of the formula terms (column Formula, N1 and N2 for example) remains unclear.
- Perhaps a reorganisation of the Tables on pages 4 - 6 would improve the paper's readability, as currently there is a 1.5 page gap in the article text.
- "NARGS and NEXITS are two software metrics defined by Mozilla and have no equivalent in the literature about maintainability metrics". In that case, what makes the authors employ these metrics for studying the maintainability characteristic?
- Lines 155-156 please recheck
- Figure 1, first two boxes (labeled 'Source code' and the one with file extensions) should be merged, as they make the idea clear; Also, perhaps it would be better to eliminate the .json boxes and represent the entire process on a single line; perhaps use 'JSON' as an annotation over the arrows, to show that was the selected format for data transfer.
- Lines 199-201 - it's not clear to me what this paragraph refers to; perhaps its intent could be further clarified by the authors
- Listing 1 does not improve the quality or understandability of the article; perhaps it would be best to include in the repository's GitHub readme and direct the reader to that using a suitable footnote.
- Lines 403-404, 435-436 refer the wrong Tables/Figures.
- Lines 437-438 - the temporal characteristic of the MI is not clear; changes in its value could be interpreted as a modification of maintainability, but the metric itself reports a singular value.
Experimental design: - The Paper should be structured according to existing best practices regarding case study research (e.g: Runeson and Host - Guidelines for conducting and reporting case study research in software engineering)
- The Maintainability Index was first elaborated for a number of C systems, and has come under strong criticism recently for not being able to adequately express the maintainability characteristic in newer paradigms (such as object oriented) and newer programming languages. While there is still merit in using it, authors should address the existence of relevant concerns. In addition, further explanation is required regarding the different forms employed for the MI. This, together with the selection of rather simple metrics to assess maintainability raises issues regarding the accuracy of the authors' measurements and their validity.
- I am not convinced that RQ1 - RQ3 are related to software maintainability, as it is understood from a software engineering perspective.
- I believe authors should drill down and present a comparative evaluation at target application level; do the descriptive statistics presented hold at each application, or are there more interesting findings?
Validity of the findings: - The selection of the 9 algorithms is arbitrary, and introduces an important threat to the external validity of the study. In addition, it is usually the case that algorithm implementations are but a small part of most large-scale systems, so it is not at all clear how the maintainability characteristic that was evaluated using these algorithm implementations will scale upwards.
- A further threat is represented by the fact that the studied algorithms were implemented as part of a software suite to study the performance of different programming languages/runtimes. This could have a further effect on the representativeness of these code bases for larger scale applications developed using those languages.
- With regards to RQ1, authors did not detail the relation between code verbosity and maintainability. Existing methodologies to determine maintainability, and at a higher level than the MI, such as technical debt are concerned with existing best practices, detection of code smells and other weaknesses; as such, it is unclear how the innate verbosity of a language will translate to the maintainability characteristic.
- Regarding the authors' answer to RQ2, the discussion should be based on the implementation of larger-scale software; it should also include a discussion on the source code author(s) programming style, as that can have an impact on these complexity metrics, especially when considering such a limited code base. This is true especially in the case of the NARGS and NEXITS metrics that are not extensively studied in the literature.
- The application of the Halstead time and bugs metrics to a new programming language/construct introduces further threats to validity; these proposed values (division by 18 and 3000, respectively) should most likely be evaluated empirically first. This is partly addressed by the authors in the Threats to Validity section.
Additional comments: The paper is competently written and approaches a subject of current interest in research. However, I believe that the title is out of sync with the paper's contents. The selection of target applications is severely limited, and suitable for an introductory, or position paper on the subject, and not a full journal publication. Furthermore, the selection of metrics to assess maintainability is limited to simplistic measurements. Recent research into maintainability generally employs more complex measures such as technical debt or the impact of code smells (such as measured using SonarQube or Ptidej). Of course, their application assumes a larger target code base to provide meaningful results. Relating to the selection of target applications, the 9 implementations are part of a benchmarking suite, and as such introduce an important threat related to the validity of the conclusions, when these are externalized to other kinds of software (e.g. open-source world or proprietary implementations of large-scale systems).

I believe that in order to work well, the paper should be re-targeted towards examining source code verbosity and understandability across different languages. In this way, the selection of target applications gains relevance, and well-known metrics  such as the Halstead suite (that are no longer used to evaluate maintainability) can be more successfully employed.


Instruction: You are one of the reviewers, your task is to write a review for the article. You will be given the title of the article, the number of the round in which the article is located, and your order among the reviewers.

Input:
Title: EVALUATION OF RUST CODE VERBOSITY, UNDERSTANDABILITY AND COMPLEXITY
Review round: 1
Reviewer: 3

Output:
Basic reporting: Authors have followed the professional article structure and shared the raw data. I commend the authors for their work but certain issues need to be resolved before acceptance.

1. Mapping of figures/tables should be thoroughly cross checked with the places they are referenced in the manuscript. Authors need to correct the table and figure referencing
For example,.
 In 358, “In the table, we report the mean and…….” Which table authors are referring to?
Line 403 “The boxplots in Figure 4 and Table 9 report the distributions, mean, and median of the Halstead….” Authors have cited wrong table and figure reference. It should be Figure 5 and Table 10.
Similar observation is made at line 435-436. “The boxplots in Figure 4 and Table 9 report the distributions, mean, and median of the Maintainability Indexes computed for the six different programming languages.”. This is repeated line with wrong references.

2. he authors use "we" too much in the paper, while I suggest to use "the paper".

3. Different notations are used for same object. It is recommended that authors should use single term. Some examples are;
Json, json, .json;
line 321 SLOC line 322 souce loc. In line 323 and 324 lines of code.  C
OGNITIVE complexity (line 375, 376) or Cognitive complexity (line 390, 395, 396), or cognitive complexity (line 393) or Cognitive Complexity(396).
Program difficulty ( line 406, 409) and Difficulty(line 408)

4. As authors has mentioned in line 438, “Halstead Volume (V), the Cyclomatic Complexity (CC),…”, they must mention the acronyms for all other terms  when first used in paper.

5. The paper is well organized. But at some points, restructuring of sentences is required. Few examples are:

Line 269-271: Multiple use of and in one sentence. “Concerning the original implementation of the rust-code-analysis tool, we have forked the project and performed modifications on it by adding metrics computations (e.g., the COGNITIVE metric) and changes to the possible output format provided by the tool.”

Line 440-441: “By using all the formulas for the Maintainability Index, we computed for the source files written in Rust an average MI that placed the fourth among all considered programming languages."
“This very low value of the cognitive per method for Rust is related……..” should be “This very low value of the cognitive complexity per method for Rust is related……..”

6. Minor grammatical errors were exposed. For example,
In 335 “….with the second-highest, mean being 59 for the…..” should be “….with the second-highest mean being 59 for the…..”

In captions of Figure 1, Figure 2  :” Distributions of the metrics about…..” should be “Distribution of the metrics about…”.
Line 95: Systematic Literature review should be systematic literature review.

7. Please check the PeerJ reference format and references should be consistent with that format.
The references in manuscript do not follow a commat format. For example,
Alqadi, B. S. and Maletic, J. I. (2020). Slice-based cognitive complexity metrics for defect prediction. In 2020 IEEE 27th International Conference on Software Analysis, Evolution and Reengineering (SANER), pages 411–422. IEEE.
Astrauskas, V., Mu¨ller, P., Poli, F., and Summers, A. J. (2019). Leveraging rust types for modular specification and verification. Proceedings of the ACM on Programming Languages, 3(OOPSLA):1– 30.
Experimental design: The experiments were well implemented, and the results are consistent. Work is novel. A tool is constructed to extract metrics of Rust and object-oriented languages. Metrics are collected for 9 program codes written in 6 programming languages. The paper is well written, the structure makes it easy to follow. Research questions are well formulated.

1. I would request authors to comment on their selection of metrics to be extracted from code. Why they did not extract object-oriented metrics?

2. Algorithms are language independent. Authors have use codes of different languages to do comparative analysis. In Table 6 title, algorithms should be replaced by code. Similarly, in complete text, whenever referring to code, replace ‘algorithm’ by ‘code’.

3. Authors mentioned and analyzed maintainability index in subsection 4.4. I would suggest authors to include some range of maintainability index (for example: bad, average, good, acceptable). This will give more clarity to readers about its relevance.

4. In Table 5, authors have scribed the three variants of MI metric. It is suggested to add reference and little detail for each definition in corresponding section.

5. Table 1 shows that CKJM extracts JAVA and C metrics.
But CKJM collects metrics only for compiled JAVA classes. CKJM stands for Chidamber and Kemerer Java Metrics. It does not work for C code. Authors need to rectify it.
Spinellis D. Tool writing: a forgotten art?(software tools). IEEE Software. 2005 Jul 11;22(4):9-11
Validity of the findings: I appreciate authors to provide all underlying data supporting the replication of the work.

1. In results section, conclusions are well stated for each RQ. But comparative analysis need to be further strengthened by using statistical tests. Authors must include statistical validation of their results. Depending on the nature of data, they can use either parametric or non-parametric tests to statistically validate the results.

2. Conclusion section need to be elaborated. Authors should include main contributions in it.
Additional comments: No additional comments


Instruction: You are one of the reviewers, your task is to write a review for the article. You will be given the title of the article, the number of the round in which the article is located, and your order among the reviewers.

Input:
Title: EVALUATION OF RUST CODE VERBOSITY, UNDERSTANDABILITY AND COMPLEXITY
Review round: 2
Reviewer: 1

Output:
Basic reporting: The authors significantly improved the paper in accordance with my (and other reviewer's) comments. As I see no other issues left uncovered, I have no further objections.
Experimental design: The authors included the statistical analysis, as per my suggestion, and addressed all of the raised issues.
Validity of the findings: The additional explanations make the newest version of the paper higher in quality, thus the validity of the findings is not questionable anymore.
Additional comments: The authors significantly improved the paper in accordance with my (and other reviewer's) comments. As I see no other issues left uncovered, I have no further objections.


Instruction: You are one of the reviewers, your task is to write a review for the article. You will be given the title of the article, the number of the round in which the article is located, and your order among the reviewers.

Input:
Title: EVALUATION OF RUST CODE VERBOSITY, UNDERSTANDABILITY AND COMPLEXITY
Review round: 2
Reviewer: 2

Output:
Basic reporting: • Line 127 - MetricsReloaded seems to be doing fine as an IntelliJ plugin, last release was December 2020 (https://plugins.jetbrains.com/plugin/93-metricsreloaded)
	• In Section 3.1, there is the possibility of detailing each RQ immediately after it was stated; this might further help readers maintain the current context and better understand the scope of the research; however, this is just a suggestion and deferred to the authors' judgement
	• Line 218 - "speed of programming languages" might not best express the authors' intention; what is actually measured is how quickly code written in a given language can be executed after being translated/interpreted by the compiler/interpreter target architecture, as a matter of optimization.
	• Line 239 - capitalize 'Figure'
	• Lines 249 and 266 look a weird, as they start with a non-capital letter, and it's unclear whether by intention or the result of a formatting error.
	• Line 272 should mention that Listing 1 is in the annex, as it's many pages away. Also, the listings might be put into a table with 2 columns and only take up 2 pages for brevity, or included in a data replication package on figshare/zenodo with their own ISBN.
* Line 493 'Whe' (typo)
Experimental design: no comment
Validity of the findings: no comment
Additional comments: The paper was revised according to the findings and suggestions resulting from the first review. The current version provides much more detail regarding the paper's main objective, the means employed to fulfil the stated objective as well as the methodology and threats to validity. Observations resulting from a careful reading can be found in the section 'Basic Reporting'.


Instruction: You are one of the reviewers, your task is to write a review for the article. You will be given the title of the article, the number of the round in which the article is located, and your order among the reviewers.

Input:
Title: EVALUATION OF RUST CODE VERBOSITY, UNDERSTANDABILITY AND COMPLEXITY
Review round: 2
Reviewer: 3

Output:
Basic reporting: I appreciate authors for incorporating the suggested changes in the manuscript.

1. Minor grammatical mistakes are still in the manuscript. Please check use of commas, articles, correct preposition usage. I have listed only few examples for your reference. Kindly crosscheck complete manuscript for correct grammar usage.

Line 48: "metrics under the categories of Size, Coupling, Complexity and Inheritance [7]." should be "metrics under the categories of Size, Coupling, Complexity, and Inheritance [7]."

Line 88: "ownership model to guarantee memory-safety and thread-safety; productivity, with integrated package" should be "ownership model to guarantee memory-safety and thread-safety; productivity, with an integrated package".

Line 109: "review, it is found that the following set of open-source tools is able to cover most of quality metrics" should be "review, it is found that the following set of open-source tools can cover most of quality metrics".

Line 144: "interpreting the results from developers and researchers standpoint." I think you missed to add apostrophe's in developers and researchers.

Line: 171 "lines or comments; the count, however, depends on the physical format of the statements and on programming".  Omit "on" before programming. Repetitive use of On.

Line 223: "Table 7 lists the code artifacts used (sorted out alphabetically) and provides a brief description for each" should be "Table 7 lists the code artifacts used (sorted out alphabetically) and provides a brief description of each"

2. More than one references can be clubbed together.
for example:
Line 52: "code metrics to predict or infer the maintainability of a project [9], [10], [11]." may be rewritten as "code metrics to predict or infer the maintainability of a project [9-11]."

3. Earlier also, I suggested to go through all the references. Kindly comply with PeerJ reference format. Below is only one example, you need to recheck all references.

Line 855 17] Abhiram Balasubramanian, Marek S Baranowski, Anton Burtsev, Aurojit Panda, Zvonimir Rakamaric´, and Leonid Ryzhyk. System programming in rust: Beyond safety. In Proceedings of the 16th Workshop on Hot Topics in Operating Systems, pages 156–161, 2017.
Line 858 [18] Vytautas Astrauskas, Peter Mu¨ller, Federico Poli, and Alexander J Summers. Leveraging rust types for modular specification and verification. Proceedings of the ACM on Programming Languages, 3(OOPSLA):1–30, 2019.
Experimental design: Authors have justified the work by revising the manuscript.

1. Statistical tests are now included . My concern is that
a) please state the hypothesis set for conducting the statistical tests.
b) if Wilcoxon test is used to compare multiple pairs, you need to use a p-value correction like a Bonferroni correction for the results to be reliable.
Validity of the findings: Work is novel. Good work done by the authors. Conclusion and future work is now in better shape. Supporting data is provided and conclusions are linked with original research questions.

Kindly modify Wilcoxon test with Bonferroni correction for reliability of the results, as I stated in experimental design.
Additional comments: No additional comments


Instruction: You are one of the reviewers, your task is to write a review for the article. You will be given the title of the article, the number of the round in which the article is located, and your order among the reviewers.

Input:
Title: RENYI ENTROPY DRIVEN HIERARCHICAL GRAPH CLUSTERING
Review round: 1
Reviewer: 1

Output:
Basic reporting: It is necessary to clearly state the purpose of the work. For example, the purpose of this paper is to apply the quadratic Renyi entropy to graph clustering. This type of entropy makes it possible to perform cluster analysis better in comparison with.... This hypothesis is proved by a series of computer experiments, which are described in the section.....
Experimental design: You need to add a separate section that describes all the experiments performed.
Validity of the findings: The conclusion does not fully reflect the results of the work.
Additional comments: In this paper, author consider the application of the quadratic Renyi entropy to estimate the intracluster and intercluster distance. The work is interesting and deserves publication. However, there are a number of unclear points in the work that need to be clarified.
1.	the concept of ‘embeddings’ is actively used in the work (for example see row 54). But actually, embeddings’ usually associated with ‘word embeddings’ in machine learning. Author have to clarify that word or replace it by another word. The using of that word can leads to misunderstanding.
2.	In General, it is not clear from the article what experiments were conducted. Therefore, you need to add a section that describes all the experiments. For example, in line 191-192, which describes a computational experiment on the dolphin network, the authors say that they used Sigma squared for 0.001 and 1. These were extreme values, or only two values were used? In other words, the description of all experiments must be clearly described, for example, to make a table of experiments.
3.	in line 205, the authors say that the following ‘the clustering algorithm visually agrees: nodes which are drawn together tend to be clustered together..’. This is not clear. It is recommended to avoid such evaluation expressions, or explain them using the results of calculations.
4.	in line 208, where there is a description of ' Jaccard distance’, you must provide a link, since this metric was not proposed by the authors of this work. However, in this paper it looks exactly like the metric proposed in this paper.
5.	it is not clear from the paper whether the proposed approach was tested on datasets with a pre-known number of clusters. This needs to be clarified. If such experiments have not been performed, I recommend adding test results on datasets with a known number of clusters.
6.	It is necessary to reformulate a conclusion. Finally, you need to add a summary of the results obtained for directed and undirected graphs.


Instruction: You are one of the reviewers, your task is to write a review for the article. You will be given the title of the article, the number of the round in which the article is located, and your order among the reviewers.

Input:
Title: RENYI ENTROPY DRIVEN HIERARCHICAL GRAPH CLUSTERING
Review round: 1
Reviewer: 2

Output:
Basic reporting: Your introduction needs more detail. I suggest that you add a short literature review on the existing graph clustering techniques and their corresponding limitations. Thus, you could indicate the existing knowledge gap in the field. Moreover, I suggest that you add some comments on the explanation of your choice in favor of Renyi entropy among many types of entropy. Please add some comments on why you consider two approaches for clustering, namely, the reason to consider graph embeddings (some background knowledge and references are needed here).
Formal results include clear definitions of all terms and detailed proofs except lines 113-114, where the notations of X, C, A_i, b are not explained. Moreover, I suggest adding a reference to the result formulated in those lines (113-114).
Experimental design: I suggest that you formulate more precisely your research question, considering that you already explored the applicability of Renyi entropy for graph clustering in your previous work (Oggier, F., Phetsouvanh, S., and Datta, A. (2018)). The text from your Conclusion section could be used for the formulation of the research question.
Validity of the findings: no comment
Additional comments: The article is written in good, professional English and is well structured. I thank you for providing the source codes of the proposed clustering algorithms. Moreover, I commend the authors for a very detailed description of their theoretical methods and experiments. However, there are some typos and confusing phrases and notations in the work:

1) The phrase in lines 44-45 “we first embed the graph to be clustered into R^d using a semidefinite programming approach for a suitable dimension d that minimizes the distortion of the embedding” seems to be inconsistent with lines 115-116 where embeddings of graph vertices have dimension N. So, actually you do not vary the dimensionality d, you set it to be the number of vertices N. Also, the change of notations in line 117 concerning the dimensionality of embedding is confusing (everywhere above you write “D-embedding into R^d, but in line 117 you suddenly write D-embedding into R^N”). I suggest that you provide some comments or explanations about it.

2) I suggest implicitly write that D is the distortion, where notation D first appears (lines 101-102). In the same manner, I suggest to write what |B| means where it is introduced for the first time (namely, line 140).

3) Euclidean norm in line 100 is written incorrectly. Please, correct it.

4) Typo in the last line of the proof of triangle inequality (between lines 214 and 215): instead of N_{out}(v \union {w}) there must be N_{out}(w \union {w}). Please, correct it.


Instruction: You are one of the reviewers, your task is to write a review for the article. You will be given the title of the article, the number of the round in which the article is located, and your order among the reviewers.

Input:
Title: RENYI ENTROPY DRIVEN HIERARCHICAL GRAPH CLUSTERING
Review round: 2
Reviewer: 1

Output:
Basic reporting: The article is written clearly and does not require revision in this version. Literature review sufficient for the purposes of the article.
Experimental design: The article matches the scope of journal
Validity of the findings: The basic idea is that the density within a cluster is estimated using a Gaussian distribution. In general, it can be assumed that there may be datasets that do not have a Gaussian distribution. The authors showed dates with just such distributions. This is a limitation of both the theoretical and practical parts of this article.
However, exploring different types of densities is outside the scope of this article.
Additional comments: Now the articles become better. Therefore it can be published.
However, I would like to advise you to explore the possibility of constructing an estimate of the entropy not only using the Gaussian distribution but also other types of distributions.


Instruction: You are one of the reviewers, your task is to write a review for the article. You will be given the title of the article, the number of the round in which the article is located, and your order among the reviewers.

Input:
Title: THE CLASSIFICATION OF MOVEMENT INTENTION THROUGH MACHINE LEARNING MODELS: THE IDENTIFICATION OF SIGNIFICANT TIME-DOMAIN EMG FEATURES
Review round: 1
Reviewer: 1

Output:
Basic reporting: Title
Suggested to be revised as “The Classification of Movement Intention through
Machine Learning Models: The Identification of Significant EMG Time-Domain features”
Abstract
Line 28-29 – “the upper limb are voluntary elbow flexion and extension along the sagittal plane” Please rephrase for clarity.
Please add a sentence at the end of the abstract to reflect the importance of the findings
Introduction
L 52-53 “There are numerous controllers have been implemented to the robot-platform for assisting the subject”
Please rephrase as “Several controllers have thus far been applied to the robot-platform to aid the patient in the movement recovery process”
L 71- “In order to used” » In order to use.
A few sentences could be added to summarize the gap that the current investigation is intended to fill before stating the objecting of the study.
Experimental design: L 93- “Fig 1. illustrates the proposed framework that been utilised in this study for detecting…” Please rephrase for clarity
The methodological section should be re-organized into “Participants, Intention recognition, Data Acquisition and Processing, Feature Extraction, Feature Selection and Classification”. Some of the information, particularly with regards to the participants, intention recognition and data processing, are somewhat mixed.
Please add more detail information on the process of data acquisition during the intended and non-intended movement phases.

Add some basic information of the participants such as weight and height under the participants ‘section.

Classification
Please state how many datasets are used for training, testing as well as independent testing
Validity of the findings: The authors investigated the efficacy of electromyogram (EMG) signals in detecting as well as predicting the pre-intention and intention motion of a human subject via the employment of several machine learning models. It was reported from the finding of the study the Decision Tree-based model was able to provide a good classification accuracy of the investigated motions through considering certain features obtained from the EMG signal. Overall, the manuscript is an interesting read, and the findings could be beneficial to the stakeholders in this domain. However, there is lack of detail information particularly in the methodology section to permit replication of the study. The authors are advised to consider the following suggestion to improve the quality as well as the wider readability of the manuscript.

Discussion
Please discuss the major findings in relation to the findings reported in the literature i.e. compare the findings of the current study with the findings from the other studies to claerly highlight the contribution of the study.
Additional comments: No comment


Instruction: You are one of the reviewers, your task is to write a review for the article. You will be given the title of the article, the number of the round in which the article is located, and your order among the reviewers.

Input:
Title: THE CLASSIFICATION OF MOVEMENT INTENTION THROUGH MACHINE LEARNING MODELS: THE IDENTIFICATION OF SIGNIFICANT TIME-DOMAIN EMG FEATURES
Review round: 1
Reviewer: 2

Output:
Basic reporting: No comment.
Experimental design: No comment.
Validity of the findings: No comment.
Additional comments: Some minor revisions as follows:
(1) There are many English verb tense errors throughout the paper. For example, “swab (was) applied to the skin surface in order to remove dirt and dried skin, before placing the electrode. Meanwhile, the torque and angle from the patient (are) captured by a torque sensor and potentiometer. The signal obtained by the torque and potentiometer (were) fed via…”
(2) What's the meaning of “bandpass signal which is 5 Hz for low filter and 500 Hz for the high pass signal”? Does that mean “EMG signal was band-pass filtered at 10–400 Hz”?


Instruction: You are one of the reviewers, your task is to write a review for the article. You will be given the title of the article, the number of the round in which the article is located, and your order among the reviewers.

Input:
Title: THE CLASSIFICATION OF MOVEMENT INTENTION THROUGH MACHINE LEARNING MODELS: THE IDENTIFICATION OF SIGNIFICANT TIME-DOMAIN EMG FEATURES
Review round: 2
Reviewer: 1

Output:
Basic reporting: No comment
Experimental design: No comment
Validity of the findings: No comment
Additional comments: I thank the authors for revising the manuscript as suggested


Instruction: You are one of the reviewers, your task is to write a review for the article. You will be given the title of the article, the number of the round in which the article is located, and your order among the reviewers.

Input:
Title: THE CLASSIFICATION OF MOVEMENT INTENTION THROUGH MACHINE LEARNING MODELS: THE IDENTIFICATION OF SIGNIFICANT TIME-DOMAIN EMG FEATURES
Review round: 2
Reviewer: 2

Output:
Basic reporting: No comment.
Experimental design: No comment.
Validity of the findings: No comment.
Additional comments: The authors have modified their work and satisfied my previous comments. Therefore, from my point of view, this paper should be accepted to be published.


Instruction: You are one of the reviewers, your task is to write a review for the article. You will be given the title of the article, the number of the round in which the article is located, and your order among the reviewers.

Input:
Title: EFFICIENT VIDEO FACE RECOGNITION BASED ON FRAME SELECTION AND QUALITY ASSESSMENT
Review round: 1
Reviewer: 1

Output:
Basic reporting: 1. The writing of the paper (e.g. grammar, tense) should be improved to make it more professional.
2. Some references on related works including gait recognition and quality assessment are suggested to be added for sufficient field background.
[1] Xianye Ben, Peng Zhang, Zhihui Lai, Rui Yan, Xinliang Zhai, Weixiao Meng. A general tensor representation framework for cross-view gait recognition, Pattern Recognition, vol. 90, pp. 87-98, 2019.
[2] Lei Chen and Jiying Zhao, No-reference perceptual quality assessment of stereoscopic images based on binocular visual characteristics, Signal Processing: Image Communication, vol. 76, pp. 1-10, 2019.
3. The format of references should be indentical, e.g. Line 332 and Line 340.
Experimental design: Many important details are missing in the experimental design. For instance, how many dataset (or proportion) is used for training or testing? Some parameters of Adam optimizer are also not provided.
Validity of the findings: 1. For the compared methods, Are the results obtained from the corresponding papers or conducted in your experiments, which need to be clarified.
2. More explanations or analysis should be added for performance comparion, instead of just listing the results.
Additional comments: The paper proposed an efficient video face recognition based on frame selection and
quality assessment. There are several issues to be addressed based on the comments.


Instruction: You are one of the reviewers, your task is to write a review for the article. You will be given the title of the article, the number of the round in which the article is located, and your order among the reviewers.

Input:
Title: EFFICIENT VIDEO FACE RECOGNITION BASED ON FRAME SELECTION AND QUALITY ASSESSMENT
Review round: 1
Reviewer: 2

Output:
Basic reporting: Your method needs more detail. I suggest that you improve the description of the proposed method to your research more clear to readers
Experimental design: Experiment is Ok.
Validity of the findings: This article has certain innovation.
Additional comments: The authors train the lightweight CNN (“FaceQNet mobile”) for face quality analysis by distilling the knowledge of the FaceQNet ResNet-50 model. The reported experiments demonstrate the performance of the proposed method. My comments are as follows:
1. The innovation of this paper is not clear and it is difficult for readers to understand the main contributions of this paper. This part should be added in Introduction section.
2. In the proposed framework (Figure 3), which part is the method proposed by the authors?
3. In Figure 1, what is teacher model? and what is Student model?
4. Some details of the proposed method are not clearly described in the paper.


Instruction: You are one of the reviewers, your task is to write a review for the article. You will be given the title of the article, the number of the round in which the article is located, and your order among the reviewers.

Input:
Title: EFFICIENT VIDEO FACE RECOGNITION BASED ON FRAME SELECTION AND QUALITY ASSESSMENT
Review round: 2
Reviewer: 1

Output:
Basic reporting: no comment
Experimental design: no comment
Validity of the findings: no comment
Additional comments: This reviewer would like to thank the authors for their effort in clarifying and modifying this paper. I have no other questions. I suggest that the paper is accepted for publication.


Instruction: You are one of the reviewers, your task is to write a review for the article. You will be given the title of the article, the number of the round in which the article is located, and your order among the reviewers.

Input:
Title: LOW-COST INTELLIGENT SURVEILLANCE SYSTEM BASED ON FAST CNN
Review round: 1
Reviewer: 1

Output:
Basic reporting: •	The word "low cost" should be updated or more defined in the Title article.
•	Paper written in clear English language.
•	The majority of the text is technically accurate.

==============================================
•	There is no literature survey in this paper.

•	This Paragraph need to  be more clear.
”Line: 84 The efficiency of the existing surveillance systems depends on the effectiveness of human, “
“Line: 85 operators, who are required to monitor the camera footage.”
I suggest that you improve the description at lines 84 and 85 to provide more justification.
In general, Introduction is very good.

•	There is Perfect and Enough theoretical background.
===============================================
•	Paper in standard format as a structure.
•	Some of Figures need to be more quality such as Figure 4, and Figure 8.
•	Figure 9, I Think above title is incorrect. In addition, X-axis and Y-axis not clear.
•	Dataset is clear and good for this track.
===============================================
•	There are relevant results to all hypotheses in this paper.
•	Most of the results mentioned in this paper are clearly defined.
•	Line 274 “Software” this term need to be change I suggest “Implementation”
===============================================
Experimental design: •	With most paper goals the results are described in detail.
•	Rigorous work carried out to a high scientific & ethical level.
•	Methods described with adequate detail & and no replicates are available.
•	Line: 333 Step 2: Train samples: 2514. Val Sample: 486, I think this not clear.
Validity of the findings: • More clarification for the novelty in this paper is needed.
• Data on the results are appropriate and acceptable.
• The data is statistically accurate and reliable.
• The conclusions are well stated. However, I recommend being as points or bulleted list.
Additional comments: I would like to thank the researcher for his handling of this important topic and I think he addressed the topic in a very good theoretical and practical way.


Instruction: You are one of the reviewers, your task is to write a review for the article. You will be given the title of the article, the number of the round in which the article is located, and your order among the reviewers.

Input:
Title: LOW-COST INTELLIGENT SURVEILLANCE SYSTEM BASED ON FAST CNN
Review round: 1
Reviewer: 2

Output:
Basic reporting: 1. General concepts in computer vision are very well explained, like ReLU (lines 132-142) and pooling (lines 143-151). On the other hand, the explanation of saliency is hard to follow (lines 165-215). The scheme in figure 2 helps to understand saliency, though. Good job on including it.

2. Your references are sufficient and properly displayed at the end of the article. I suggest you cite them in the text. For instance, claims like the ones you make in lines 89-90 and 105-106 would greatly benefit from pointing out which reference backs them up.

3. Table 1 needs to specify what metric is following (relative error?) .

4. Figure 9 has an errata in its title: change “recognation” to “recognition”.

5. Please consider adding some of the following definitions – or cite a source that explains them: artificial immune system (line 110), the integration structure “o” (line 213), BMS and LG algorithms (lines 315-316), superpixels (line 317).

6. The image in figure 1 is watermarked. You should ensure you have the rights to use the image and credit the original source.

7. The word “realize” is repeated several times throughout the article, with the intended meaning of “perform” (i.e. lines 23, 31, 40). I recommend you avoid using “realize” in those context as its meaning can easily be missunderstood as “notice”.
Experimental design: 1. The methods you describe are not enough to reproduce the CNN model. Please include the base model you used for transfer learning. You list some good ones as examples (line 303), but forget to specify which one(s) you used.

2. I strongly suggest you detail a bit more the training process: which layers you freezed, how many epochs you allowed the model to train, if you applied early stopping, the type of regularization used, etc.

3. I commend your idea of including the steps to follow in order to build the RCNN (lines 331-341). However, steps 3, 8, 9 and 10 does not pertain to the algorithm process. Rather, they are observations on the data and the model results.
Validity of the findings: To my knowledge, both the training, validation and test data came from the same source. Consider testing your model against images from other sources. Andrei Barbu et al. (ObjectNet: A large-scale bias-controlled dataset for pushing the limits of object recognition models) pointed out that computer vision models tend to learn the bias of a dataset when training (i.e. similar objects always being in front of a similar background). Showing your CNN gets good results against new datasets will prove it is robust and trustworthy.
Additional comments: I thank for providing the model and the script to run and test it. I must note, however, that your code looks very similar to the one that Adrian Rosebrock writes in his blog Pyimagesearch. If you have used his work for reference, I strongly encourage you to include Pyimagesearch between your references.