Instruction: You are one of the reviewers, your task is to write a review for the article. You will be given the title of the article, the number of the round in which the article is located, and your order among the reviewers.

Input:
Title: HIGH THROUGHPUT RESOURCE EFFICIENT RECONFIGURABLE INTERLEAVER FOR MIMO WLAN APPLICATION
Review round: 1
Reviewer: 1

Output:
Basic reporting: In this work, the authors propose a novel algorithm to efficiently model the address generation circuitry of MIMO WLAN interleaver. The interleaver used in MIMO WLAN transceiver has three steps of permutation involving floor function whose hardware implementation is considered to be most challenging due to the absence of corresponding digital hardware. Experimental results are promising, comparisons with few recent similar works including the conventional Look-Up Table (LUT) based technique demonstrated the superiority of the proposed design in terms of maximum improvement in operating frequency by 196.83%, maximum reduction in power consumption by 74.27% and reduction of memory occupancy by 88.9%.
Experimental design: Without comments
Validity of the findings: 1 - More comparisons with previous works could be a nice complement for the current manuscript. Then, the authors could demonstrate how the proposed approach outperform the current state of the art.
2 - A post-synthesis simulation could be a nice complement for the current manuscript.
Additional comments: In general, the manuscript is interesting, well organized and deserves to be published. There are a little grammatical/style error. In my opinion, a grammar/style revision has to be carried out before the manuscript can be considered for publication.


Instruction: You are one of the reviewers, your task is to write a review for the article. You will be given the title of the article, the number of the round in which the article is located, and your order among the reviewers.

Input:
Title: HIGH THROUGHPUT RESOURCE EFFICIENT RECONFIGURABLE INTERLEAVER FOR MIMO WLAN APPLICATION
Review round: 1
Reviewer: 2

Output:
Basic reporting: In this work, according to the authors, they propose an algorithm with the mathematical background for the address generator eliminating the need for ﬂoor function. The algorithm is converted into digital hardware for implementation on the reconﬁgurable FPGA platform. Hardware structure for the complete interleaver including the reading address generator and the memory module is designed and modeled in VHDL using Xilinx Integrated Software Environment (ISE) utilizing embedded memory and DSP blocks of the target FPGA. in the end, the functionality of the proposed algorithm is veriﬁed through exhaustive software simulation.  The authors has done work for high-speed communication which is an appreciate able work domain.

I have a concern that can be helpful to improve the quality of the manuscript.
Experimental design: 1- Experimental approach of the study is not clear, the author should summarize it in a clear pattern so it can be easy for the reader to understand it.

2- In the abstract, the author mentions that they veriﬁed the functionality of the proposed algorithm through exhaustive software simulation, please elaborate it why? is there any other method to verify? if yes then why not others?
Validity of the findings: no comment
Additional comments: 1- The major issue is the presentation of content it should be better and understandable.
2- English grammar and typos should be checked thoroughly.
3- Some keypoint should be add in the introduction section for the ease of the reader to understand the article's contribution.


Instruction: You are one of the reviewers, your task is to write a review for the article. You will be given the title of the article, the number of the round in which the article is located, and your order among the reviewers.

Input:
Title: HIGH THROUGHPUT RESOURCE EFFICIENT RECONFIGURABLE INTERLEAVER FOR MIMO WLAN APPLICATION
Review round: 1
Reviewer: 3

Output:
Basic reporting: .
Experimental design: .
Validity of the findings: .
Additional comments: Highlights the main scientific contributions in section 1.

Section 5 and 6 are the main section of the paper, however they are extremely poor. Authors must
insert more information and results.

Table 8:
If the maximum throughput requirement of IEEE 802.11n is 600 Mbps which is advantage in work
with more? The authors must clarify this information.

Table 9:
The power consumption increases non-linearly with frequency cube (P ≈ F3). How, does the proposal have small consumption then literature? The authors must clarify this information.
Is this information dynamic or static power?


Instruction: You are one of the reviewers, your task is to write a review for the article. You will be given the title of the article, the number of the round in which the article is located, and your order among the reviewers.

Input:
Title: HIGH THROUGHPUT RESOURCE EFFICIENT RECONFIGURABLE INTERLEAVER FOR MIMO WLAN APPLICATION
Review round: 2
Reviewer: 1

Output:
Basic reporting: The author proposes an algorithm to efficiently model the address generation circuitry of the MIMO WLAN interleaved. The interleaved used in the MIMO WLAN transceiver has three permutation steps involving floor function whose hardware implementation is most challenging due to the absence of corresponding digital hardware. They proposed an algorithm with a mathematical background for the address generator, eliminating the need for floor function. The algorithm is converted into digital hardware for implementation on the reconfigurable FPGA platform. Hardware structure for the complete interleaved, including the read address generator and memory module, is designed and modeled in VHDL using Xilinx Integrated Software Environment (ISE) utilizing embedded memory and DSP blocks Spartan 6 FPGA.

Author work in a good manner in revision on raised point and tackle the things very well.
Experimental design: The experimental explanation and flow are cleared in revision.
Validity of the findings: The author presents results well which improves the quality of the manuscript.
Additional comments: The manuscript is much strong after revision but typos are still a problem in the manuscript.


Instruction: You are one of the reviewers, your task is to write a review for the article. You will be given the title of the article, the number of the round in which the article is located, and your order among the reviewers.

Input:
Title: CONTRAST OPTIMIZATION OF MASS SPECTROMETRY IMAGING (MSI) DATA VISUALIZATION BY THRESHOLD INTENSITY QUANTIZATION (TRIQ)
Review round: 1
Reviewer: 1

Output:
Basic reporting: No remarks on the basic reporting, only a suggestion for figure 9.
Figure 9:
It is unfortunate that the majority of the information related to (LAESI, LTP, DESI, MALDI) shown in this graph is crowded in the figure, making it difficult to interpret this information. Perhaps it would help to adapt the scale to improve readability of this figure?

A small correction:

Line 232: Figure 7 -> Figure 8
Experimental design: Figure 2 and lines 165 - 166 ‘The mass image processed with the TrIQ algorithm reassembles best the anatomical details which are recognizable in the optical image’

The contrast enhancement difference between using the natural logarithmic versus a linear equalizing approach comes out clearly. It could be due to the quality of the images but currently it is not easy to see where TriQ improves over the Linear Equalizing approach. Which anatomical details are you referring to? It would be good to clarify or highlight the obtained improvements and in particular the anatomical details you are referring to in the images.

Figure 3 – background optimization
How does the TriQ approach compare to the state-of-the art for background optimization in addition to the raw results shown? It would also be good to add the histology section for comparison as well.

Figure 4 – lines 193 – 198 Global TriQ
 How do the results shown in figure 4 compare to the slices without global TriQ being applied?
Validity of the findings: I like the approach of the global TriQ to improve comparability of ion images in MSI data, which is an important problem to solve. However, I feel the appropriate controls are missing making it difficult to properly assess the validity of the findings.
Additional comments: No additional comments


Instruction: You are one of the reviewers, your task is to write a review for the article. You will be given the title of the article, the number of the round in which the article is located, and your order among the reviewers.

Input:
Title: CONTRAST OPTIMIZATION OF MASS SPECTROMETRY IMAGING (MSI) DATA VISUALIZATION BY THRESHOLD INTENSITY QUANTIZATION (TRIQ)
Review round: 1
Reviewer: 2

Output:
Basic reporting: The manuscript descirbes the implementation of a Threshold Intensity Quantization algorithm for augmenting the contrast in Mass Spectrometry Imaging (MSI) data visualizations.
The authors method provides an improvement of MSI data through increasing the contrast and homogenizing the background.

The language used throughout the manuscript is clear.
Experimental design: The method was implemented as an open-source software RmsiGUI with an R script for post-processing.
They validated the developed method using different dataset acquired from different techniques. The results show an improvement in the contrast and in the detection of ‘region of interest’.

I compliment the authors on their vast data set used to validate the method. However, the mathematical part in the section 2.2 need to be improved by providing more details.
The authors claimed that their algorithm is computationally fast (line 266) but no strong information is provided to justify this. Please provide a quantifiable evidence (running time, memory expenses, …) to justify this, since figure 9 is not clear and does not provide sufficient details.
Validity of the findings: The research question is well defined.
The authors stated that several programs for MSI data analysis exists and employ the statistical language R but did not compare their developed algorithm with these existing ones. Authors are suggested to include one or more of these existing methods as a comparison to prove that their method is working fine.
Authors are also suggested to provide proper applications of this tools as the novelty and the impact is not well assessed.
Additional comments: The introduction shows the context but needs more details. I suggest that you improve the description by some up-to-date references.
The figures are relevant and well labeled, however some are not of high quality (Figure 1 and Figure 9).


Instruction: You are one of the reviewers, your task is to write a review for the article. You will be given the title of the article, the number of the round in which the article is located, and your order among the reviewers.

Input:
Title: CONTRAST OPTIMIZATION OF MASS SPECTROMETRY IMAGING (MSI) DATA VISUALIZATION BY THRESHOLD INTENSITY QUANTIZATION (TRIQ)
Review round: 2
Reviewer: 1

Output:
Basic reporting: No further comments.
Experimental design: No further comments.
Validity of the findings: No further comments.
Additional comments: Nice work!


Instruction: You are one of the reviewers, your task is to write a review for the article. You will be given the title of the article, the number of the round in which the article is located, and your order among the reviewers.

Input:
Title: CONTRAST OPTIMIZATION OF MASS SPECTROMETRY IMAGING (MSI) DATA VISUALIZATION BY THRESHOLD INTENSITY QUANTIZATION (TRIQ)
Review round: 2
Reviewer: 2

Output:
Basic reporting: The manuscript descirbes the implementation of a Threshold Intensity Quantization algorithm for augmenting the contrast in Mass Spectrometry Imaging (MSI) data visualizations.
The authors method provides an improvement of MSI data through increasing the contrast and homogenizing the background.
The method was implemented as an open-source software RmsiGUI with an R script for post-processing.
They validated the developed method using different dataset acquired from different techniques.
The results show an improvement in the contrast and in the detection of ‘region of interest’.
The language used throughout the manuscript is clear.
Experimental design: The introduction shows the context but needs more details. I suggest that you improve the
description by some up-to-date references.
The figures are relevant and well labeled, however some are not of high quality (Figure 1 and Figure 9).
The research question is well defined.
I compliment the authors on their vast data set used to validate the method. However, the mathematical part in the section 2.2 need to be improved by providing more details.
The authors claimed that their algorithm is computationally fast (line 266) but no strong information is provided to justify this. Please provide a quantifiable evidence (running time, memory expenses, …) to justify this, since figure 9 is not clear and does not provide sufficient details.
The authors stated that several programs for MSI data analysis exists and employ the statistical language R but did not compare their developed algorithm with these existing ones.
Validity of the findings: Authors are suggested to include one or more of these existing methods as a comparison to prove that their method is working fine. Authors are also suggested to provide proper applications of this tools as the novelty and the impact
is not well assessed.
Additional comments: No additional comments


Instruction: You are one of the reviewers, your task is to write a review for the article. You will be given the title of the article, the number of the round in which the article is located, and your order among the reviewers.

Input:
Title: REACTIVE NAVIGATION UNDER A FUZZY RULES-BASED SCHEME AND REINFORCEMENT LEARNING FOR MOBILE ROBOTS
Review round: 1
Reviewer: 1

Output:
Basic reporting: The structure of this paper is not consistent to the objective of the work. In the Introduction section, there should be more focus on the decision-making techniques rather than path-finding.
Experimental design: No comment
Validity of the findings: No comment
Additional comments: Authors published good results. There are many grammatical errors. Focus on the writing on what you want to achieve in the results.


Instruction: You are one of the reviewers, your task is to write a review for the article. You will be given the title of the article, the number of the round in which the article is located, and your order among the reviewers.

Input:
Title: REACTIVE NAVIGATION UNDER A FUZZY RULES-BASED SCHEME AND REINFORCEMENT LEARNING FOR MOBILE ROBOTS
Review round: 1
Reviewer: 2

Output:
Basic reporting: 1.	This paper has a lot of typing and grammatical errors. The English should be properly reviewed
2.	Abstract is very fuzzy, and poor written
3.	Paper main contribution is not clear
4.	Authors should give more details about implementation, validation procedure, validation time, computational complexity…
5.	Authors should justify their improvements against previous results
6.	Authors should include technical information in order the reader can replicate the proposed results
7.	Authors should include a detailed analysis about energy consumption improvements
8.	Authors should include a detailed comparative analysis
9.	References should be updated and improved
10.	This paper does not present any novelty that deserves to be published
Experimental design: This paper only includes simulation results, it is not clear the importance of selected navigation profiles, and there is no comparative analysis, therefore the proposed research is not rigorous.
Validity of the findings: There is no novelty reported in this paper that deserves to be published, in its actual form this paper appears to be an application of well-known results without any analysis. Authors do not include enough information in order that the proposed results can be replicated. Finally, there is not clear the importance of proposed results for mobile robotics literature.
Additional comments: This paper deals with an interesting topic, however in its actual form research appears to be ambiguous.


Instruction: You are one of the reviewers, your task is to write a review for the article. You will be given the title of the article, the number of the round in which the article is located, and your order among the reviewers.

Input:
Title: REACTIVE NAVIGATION UNDER A FUZZY RULES-BASED SCHEME AND REINFORCEMENT LEARNING FOR MOBILE ROBOTS
Review round: 1
Reviewer: 3

Output:
Basic reporting: No comment.
Experimental design: No comment.
Validity of the findings: No comment.
Additional comments: The paper sounds good, it is well written and structured. Figures have good quality. Simulation results validate their proposal and gives an insight about its performance in contrast with other approaches. Conclusions are coherent and well linked with the results shown. Anyway, this reviewer thinks that it would be interesting if the authors can give an insight about considering another platform instead of the King Spider robot. I mean, what could be the main differences to identify when considering to work with another robot and which criteria must be considered to modify the algorithm in order to adapt it to the new robot platform. In what way the physical/kinematic parameters will impact for the path generation? How this last will affect the path generation? And, if this is the case, how it will affect to the other approaches used? I know that the input variables and fuzzy sets are given in terms of battery level and distances. So, how will impact the use/modification of hardware in your proposal? And with respect to the others approaches? All of these show the same disadvantage/advantage? What about if, instead to go to the charging station, the use of some of the King Spider’s servomotors could be clamped and just those that permit to follow, in some way, the path are limited to use? Maybe another premise variable can be included. This reviewer suggests to include at references section some books about basics on fuzzy logic.


Instruction: You are one of the reviewers, your task is to write a review for the article. You will be given the title of the article, the number of the round in which the article is located, and your order among the reviewers.

Input:
Title: REACTIVE NAVIGATION UNDER A FUZZY RULES-BASED SCHEME AND REINFORCEMENT LEARNING FOR MOBILE ROBOTS
Review round: 2
Reviewer: 1

Output:
Basic reporting: Clear and unambiguous, professional English used throughout. YES

Literature references, sufficient field background/context provided.  YES

Professional article structure, figures, tables. Raw data shared. YES

Self-contained with relevant results to hypotheses. YES
Experimental design: This paper only includes simulation results, however methods described with sufficient detail & information to replicate
Validity of the findings: Conclusions are well stated, linked to original research question & limited to supporting results.
Additional comments: All my comments have been addessed, then, I have no more comments


Instruction: You are one of the reviewers, your task is to write a review for the article. You will be given the title of the article, the number of the round in which the article is located, and your order among the reviewers.

Input:
Title: REACTIVE NAVIGATION UNDER A FUZZY RULES-BASED SCHEME AND REINFORCEMENT LEARNING FOR MOBILE ROBOTS
Review round: 2
Reviewer: 2

Output:
Basic reporting: No comment.
Experimental design: This work is limited to simulation results. Anyway, authors present the hardware that can be used for implement their proposal in a future work.
Validity of the findings: No comment.
Additional comments: The authors have addressed all the concerns and comments raised by this reviewer, and most of them have been included in the manuscript. This reviewer has no more comments.


Instruction: You are one of the reviewers, your task is to write a review for the article. You will be given the title of the article, the number of the round in which the article is located, and your order among the reviewers.

Input:
Title: EXCLUSIVE USE AND EVALUATION OF INHERITANCE METRICS VIABILITY IN SOFTWARE FAULT PREDICTION—AN EXPERIMENTAL STUDY
Review round: 1
Reviewer: 1

Output:
Basic reporting: No Comment.
Experimental design: The experimental part needs improvement. Lots of effort have been devoted to the pre-processing step. However, details about model building using SVM are limited. Such as which kernel has been used. How the control parameters of SVM have been optimized. How dataset split into training and testing parts. Which error measure has been used. Why this particular measure was selected.

Further, using only one ML algorithm (SVM) is not enough. Authors should consider some state-of-the-art ML algorithms for SFP modeling.
Validity of the findings: In the context of Inheritence metrics, findings are sufficient, but in my view, comparison with the other OO metrics is limited and authors need to improve the paper in this part.
Additional comments: No additional comments


Instruction: You are one of the reviewers, your task is to write a review for the article. You will be given the title of the article, the number of the round in which the article is located, and your order among the reviewers.

Input:
Title: EXCLUSIVE USE AND EVALUATION OF INHERITANCE METRICS VIABILITY IN SOFTWARE FAULT PREDICTION—AN EXPERIMENTAL STUDY
Review round: 1
Reviewer: 2

Output:
Basic reporting: In this work, Authors have used and viability of inheritance metrics in SFP is evaluated through experiment. They have conducted a survey of inheritance metrics whose data is publicly available. They have collected about 40 data sets having inheritance metrics. Based on two criteria: these inheritance metrics have been cleaned and filtered; resulting in nine inheritance metrics captured. After preprocessing, the selected data sets have been divided into all possible combinations of inheritance metrics followed by merging similar metrics.

-The paper topic is good but the readability of this paper is very less.
Experimental design: 1.	Authors have not done detailed literatures survey on defect prediction. Authors should put the details of available techniques proposed by different researchers in the field on software defect predictions.
2.	Presently, researchers in the filed of software engineering are using advance machine learning techniques to develop software fault prediction.
3.	The authors have not given any idea about the reason of using only inheritance metrics i.e., why not cohesion or coupling
Validity of the findings: 1. Authors are working on classification problem not on regression problem. Authors should compare the models using accuracy, F-measure, and AUC value
2.	The authors should also prove that the developed models using inheritance metrics are significantly same of different from models developed using cohesion or coupling metrics.
3.	Paper need detail threat to validity section.
4.	Figure 2 is not easy to read. I recommend to use box-plot diagram.
Additional comments: In this work, Authors have used and viability of inheritance metrics in SFP is evaluated through experiment. They have conducted a survey of inheritance metrics whose data is publicly available. They have collected about 40 data sets having inheritance metrics. Based on two criteria: these inheritance metrics have been cleaned and filtered; resulting in nine inheritance metrics captured. After preprocessing, the selected data sets have been divided into all possible combinations of inheritance metrics followed by merging similar metrics.

-The paper topic is good but the readability of this paper is very less.

1.	Authors have not done detailed literatures survey on defect prediction. Authors should put the details of available techniques proposed by different researchers in the field on software defect predictions.
2.	Presently, researchers in the filed of software engineering are using advance machine learning techniques to develop software fault prediction.
3.	The authors have not given any idea about the reason of using only inheritance metrics i.e., why not cohesion or coupling
4.	Authors are working on classification problem not on regression problem. Authors should compare the models using accuracy, F-measure, and AUC value
5.	The authors should also prove that the developed models using inheritance metrics are significantly same of different from models developed using cohesion or coupling metrics.
6.	Paper need detail threat to validity section.
7.	Figure 2 is not easy to read. I recommend to use box-plot diagram.

The paper is interesting, but needs *major* revision. The paper may be accepted, if the authors pay more attention to readability, clarity, and organization.


Instruction: You are one of the reviewers, your task is to write a review for the article. You will be given the title of the article, the number of the round in which the article is located, and your order among the reviewers.

Input:
Title: EXCLUSIVE USE AND EVALUATION OF INHERITANCE METRICS VIABILITY IN SOFTWARE FAULT PREDICTION—AN EXPERIMENTAL STUDY
Review round: 2
Reviewer: 1

Output:
Basic reporting: A thorough proof-read of the paper is required. I advise the authors to use a language expert or language correction tool for the same.
Experimental design: No comment.
Validity of the findings: No comment.
Additional comments: The authors have addressed all my comments. There are a few comments about the figures and tables.
1. Many tables such as Table 1, Table 2, Table 5 are not clear. It seems that the authors have used images of the table. Please change them and use clear tables.
2. Similarly, Figures 2 and 3 are blurred. Change them.

A thorough proof-read of the paper is required. For example, in Abstract, sentences such as "Software Fault Prediction (SFP) is a way to spot faulty classes. It is helpful in the development and testing process. Software metrics are utilized in SFP." seem very immature. I advise the authors to use a language expert or language correction tool for the same.


Instruction: You are one of the reviewers, your task is to write a review for the article. You will be given the title of the article, the number of the round in which the article is located, and your order among the reviewers.

Input:
Title: MULTI SKIN LESIONS CLASSIFICATION USING FINE-TUNING AND DATA-AUGMENTATION APPLYING NASNET
Review round: 1
Reviewer: 1

Output:
Basic reporting: The goal of the work is to propose an approach using NASNet architecture to recognize eight skin diseases. The paper is in the scope of PeerJ Computer Science. The authors ignore the use of segmentation methods and applied transfer learning and data augmentation. The introduction is extensive literature supported. The authors used a widely adopted data source such as The International Skin Imaging Collaboration . The seven images in the manuscript are low quality.

There is a mistake in the caption and in the header of Table 4.
Experimental design: The authors used 10-fold cross validation, however, Table 4 shows that the authors compared its proposal versus other CNN architectures using different No. classes and evaluation methods. The above comparison is not feasible because all methods have to reproduce the same configuration. The authors should have evaluated the state-of-the-art methods following the same steps, e.g. InceptionV4 needs to be evaluated using the same dataset, the same No. classes and the same 10-fold cross validation. In summary, the comparison performed versus other methods based on Table 4 is not valid and it needs to be performed accordingly the configuration chosen by the authors.

On the other hand, the authors mentioned several metrics to evaluate the performance, but finally they based the conclusions only on accuracy. It should be notice that accuracy is not recommended when there is a high imbalance factor, like the authors recognized in line 208. Also, Figure 6 shows that more than 34,100 images were analyzed during testing, however, the authors already stated that 23,488 are the total number of images, which is inconsistent regarding the evaluation method. In 10-fold cross-validation, for each fold the test set should be composed by approximately 2,348 images. Data augmentation should be applied in such manner that augmented images only compute to find the prediction of the original images, e.g. let us say that test image X is augmented by a factor of 10, then the final prediction of X can be computed using a soft-voting strategy using the 11 images. It should be notice that in this manner the number of test values who computes the final result remains as 2,348.
Validity of the findings: The authors should support the Conclusions based on the evaluation of the methods using the same configuration. In this regard, the experimental study needs to be repeated in some phases in order to sustain the validity of the work.
Additional comments: No additional comments


Instruction: You are one of the reviewers, your task is to write a review for the article. You will be given the title of the article, the number of the round in which the article is located, and your order among the reviewers.

Input:
Title: MULTI SKIN LESIONS CLASSIFICATION USING FINE-TUNING AND DATA-AUGMENTATION APPLYING NASNET
Review round: 1
Reviewer: 2

Output:
Basic reporting: There are some recommendations and mathematical improvements that I suggest in this paper:
- As a recommendation for a better reading of Section 2, main mathematical expressions and equations should be numbered.

- Equation at the end of line #150: The super-index "l" in the denominator must be "j", to match the index with the summation.

- Beginning of line #151: sigma(z^l) is the probability distribution, not exp(z^l).

- As a recommendation at the end of line #155, "after one-hot" should be substituted by "one-hot encoding".

- Line #156: the numbers of "H1,...,H8" must be subindexes.

- Equation in line #159: Summation indexes must be from k=1 to m.

- English language could be improved in the second part of line #159, "with these notations; the vector...etc". The current phrasing makes comprehension difficult.

- Lines #172 and 173: I suggest justify the selection of mini batches n=30. In fact, the recommendation of your reference [37] says that should be a number between 50 and 256.

- At the end of line #186: In the last equation, "theta" must be a subindex of "nabla".
Experimental design: no comment.
Validity of the findings: no comment.
Additional comments: no comment.


Instruction: You are one of the reviewers, your task is to write a review for the article. You will be given the title of the article, the number of the round in which the article is located, and your order among the reviewers.

Input:
Title: MULTI SKIN LESIONS CLASSIFICATION USING FINE-TUNING AND DATA-AUGMENTATION APPLYING NASNET
Review round: 2
Reviewer: 1

Output:
Basic reporting: The authors have fixed the minor corrections, as well the manuscript in overall.
Experimental design: The authors improved the experimental settings and now the manuscript shows extensive results.
Validity of the findings: The results showed Xception achieved 2.4% better validation performance compared to NASNetMobile, however, the authors have a point when comparing by trainable parameters. In this case the best average CNN model was NASNetMobile, as the authors already stated.

There is a concern regarding Table 4. The model achieved the lowest recall performance in Melanoma category. Please notice predicting Melanoma is probably the most important task regarding skin diseases. It will be useful if the authors provide an explanation about the above result.
Additional comments: No additional comments


Instruction: You are one of the reviewers, your task is to write a review for the article. You will be given the title of the article, the number of the round in which the article is located, and your order among the reviewers.

Input:
Title: MULTI SKIN LESIONS CLASSIFICATION USING FINE-TUNING AND DATA-AUGMENTATION APPLYING NASNET
Review round: 2
Reviewer: 2

Output:
Basic reporting: Correction of English grammar is needed in the whole document.

The conclusions given in Section 3 are not consistent or clearly justified in the rest of the document.
Experimental design: There is a lack of clarity and high technical descriptions.

As a recommendation for a better reading of Section 2, the mathematical expressions and equations should be numbered.
Validity of the findings: The conclusions given in Section 3 are not consistent or clearly justified in the rest of the document.
Additional comments: - Correction of English grammar is needed in the whole document.

- Line #44: ID of melanoma is missing.

- As a recommendation for a better reading of Section 2, the mathematical expressions and equations should be numbered.

- Line #150: Equation at the end of this line: The super-index "l" in the denominator must be "j", to match the index with the summation.

- Beginning of line #151: sigma(z^l) is the probability distribution, not exp(z^l).

- Line #154: "x8" must be "m" or another literal because it represents the number of training data, not the number of classes.

- Line #155, "after one-hot" should be substituted by "one-hot encoding".

- Line #156: the numbers of "H1,...,H8" must be subindexes.

- Equation in line #159: Subindex of summation must be "k", not "m".

- Correction of English grammar in lines #159 to 161.

- Beginning of the line #161: subindexes "j" and "k" are missed in the weight "wl".

- Equation in line #162: the sigma/summation letter is not needed in the compact vectorized form.

- Lines #172 and 173: I suggest justifying the selection of mini batches n=30. In fact, the recommendation of your reference [37] says that it should be a number between 50 and 256. So, why did you use 30?

- Line #174: "975 steps: You need to mention the size of your dataset: training, validation and test sets.

- Line #174: "respited"? Did you mean "repeated"?

- Line #185: "Where gi to denote the gradient i at time step t" must be:"where gt denotes the gradient at time step t".

- At the end of line #186: In the last equation, "theta" must be a subindex of "nabla".

- Lines #195 to 202: A detailed description of the dataset must be included: the size of the dataset and a detailed description of each variable, the number of classes of the dataset: training, validation and testing.

- Line #207: an extra line was included.

- Line #218: Correction of English grammar is needed.

- Line #219: What is the "small number of images" that you are talking about?

- Line #273: Table 2 is wrong. For example, at least the values of the main diagonal must be TP. Include who are the actual values and the predictions. And denote the explicit number of classes, 8, not just as "j".

- Line 288: "aF1 Score" must be "F1 Score".

- Line #312: What are the sizes of training, validation, and testing?

- Line #313: What is the original dimension of each image?

- Line #315: A detailed description of the classes imbalanced must be included.

- Line #326: The size of the partitions must be clarified: in Line #312 you are talking of 23,488 images; in Line #326 you are talking of 44,669 augmented images; and the confusion matrix of Figure 6 has around 41,000.

- Line #330: Not only Melanoma and Nevus had significant errors, from the confusion matrix of Figure 6, we can observe that AK, BCC, and BKL have equal or higher errors than ML and NV.

- Line #334: The number of classes/diseases is 8, but the text in Figure 6 is talking of 9 classes. Why is this?

- Line #334: The notation for the skin classes used in Line #155 and the text of Figure #6 must be the same.

- Line #334, Figure 6: Confusion Matrix: To have a better idea of the magnitude of the errors is recommended to include another confusion matrix, but with percentage values.

- Line #338. 83% for all the metrics used? That is very odd. And at least from the values that we can see in the confusion matrix of Figure 6, the accuracy is approximately 84.3%.

- Line #349. It must be Table 4.

- Line #351: The direct comparison of your method with [30] in Table 4 does not have too much sense, because [30] is using more classes, 10, and your method is using 8 classes.

- In Line #330 you are saying that Melanoma and Nevus have the higher errors (as we can observe from the confusion matrix of Figure 6) because the visual similarity, but now your conclusion in Line #359 is that you can discriminate them? How is that?

- Lines #359. And what about Vascular (VASC) and Dermafibroma (DF)? Confusion matrix of Figure 6, shows them with the best results.

- Line #361: These values, of accuracies and losses, are not clearly justified in the document.

- The conclusions given in Section 3 are not consistent or clearly justified in the rest of the document.


Instruction: You are one of the reviewers, your task is to write a review for the article. You will be given the title of the article, the number of the round in which the article is located, and your order among the reviewers.

Input:
Title: MULTI SKIN LESIONS CLASSIFICATION USING FINE-TUNING AND DATA-AUGMENTATION APPLYING NASNET
Review round: 3
Reviewer: 1

Output:
Basic reporting: The authors have improved the manuscript in overall.
Experimental design: The experimental study was already extended in the previous revision.
Validity of the findings: The authors provided an explanation about the performance by comparing to dermatologists and achieving similar accuracy.
Additional comments: No additional comments


Instruction: You are one of the reviewers, your task is to write a review for the article. You will be given the title of the article, the number of the round in which the article is located, and your order among the reviewers.

Input:
Title: MODEL INDEPENDENT FEATURE ATTRIBUTIONS: SHAPLEY VALUES THAT UNCOVER NON-LINEAR DEPENDENCIES
Review round: 1
Reviewer: 1

Output:
Basic reporting: + Couple of typos it seems.
Line 250: Delete X3.
Line 296: Change Age to SBP.

+ Paragraph starting at line 63 can be improved.
We know computational complexity is a big issue using Shapley values. Discussed at lines 63 and 343.
If proposed Sunnies can use any of the ideas proposed for efficient computation, state it. If not, also state it explicitly, saying future work is needed for larger data sets.
Experimental design: + Equation (5) with d=4 and A=diag(0,2,4,6) is used for figure 2. But there are five features in figure 2. I think there is a typo somewhere.

+ Concept drift experiment (line 228) can use some more context.
Comparison methods might help, e.g. XGBoost feature importance (which you can say is model-dependent), or pairwise correlations (linear or non-linear).
Or maybe state that clearly a linear (or non-linear) pairwise correlation would do here, but the point here is that your method is general. I might be overthinking this!
Validity of the findings: Please bear with me while I elaborate more on complexity issue again.
Line 343 you mention it is useful to partition features to independent blocks which is technically true.
But a big question arises, if researchers in a domain can partition features to (non-interacting) independent blocks (of size of about five features), why not just study non-linear independence relations (using DC, AIDC, or HSIC), or even more traditional, why not try interaction terms in regression models and check likelihood scores? After all, there are only handful of features in each independent block.
Let me put it another way. If we only have small number of features
why not look at each term C(SU{v})-C(S) separately, (which probably you've saved when calculating non-linear Shapley values), rather than adding these up to get Shapley value and lose the local information? i.e. simply using non-linear correlations you listed, but not doing the summation over subsets S and reporting single value per feature.
Additional comments: + Example 2 (line 175). I think it'd be informative to also list DC, AIDC, HSIC among {X1}, {X2},
{X1, X2} and Y.

+ To convince the audience that non-linear Shapley values are helpful, you kind of have to show them at least one example that the number of features is high and simply looking at a list of DC, AIDC, or HSIC is not practical.

+ The misspecified model (line 241) is a very clear and essential example! Here one has to move away from (pairwise or linear) and consider ("more global" and non-linear) hence your non-linear Shapley.
But again, this example is also too small! One can look at the list of nonlinear correlations rather than adding them up into Shapley.

+ (Plug alert!) In our paper (S. Ma, -- 2020), we showed that Shapley values can be misleading for learning local causal neighborhoods, using two simple examples of Markovian graphs (i.e. faithfullness assumption were met).
The general statement (which is our current belief) is related to the issues above. Basically, adding up all C(SU{v})-C(S) terms, one loses the important local information about v and certain S. At least for causal discovery, Shapley's way of adding C(SU{v})-C(S) terms, though mathematically attractive with additvity on C and all, seems arbitrary, or I shall say, counterproductive.


Instruction: You are one of the reviewers, your task is to write a review for the article. You will be given the title of the article, the number of the round in which the article is located, and your order among the reviewers.

Input:
Title: MODEL INDEPENDENT FEATURE ATTRIBUTIONS: SHAPLEY VALUES THAT UNCOVER NON-LINEAR DEPENDENCIES
Review round: 1
Reviewer: 2

Output:
Basic reporting: No comment.
Experimental design: I like the proposed method but believe it needs a much more rigorous investigation ("Rigorous investigation performed to a high technical & ethical standard). There are several experiments where Shapley value based, model-dependent methods exist for the chosen problems and should be compared against. It is not necessary, especially given PeerJ's explicit willingness to publish negative or inconclusive results, to outperform these other methods, but it is necessary to include them in experiments.

3.0.1 - Here a decision tree perfectly learns the XOR function but the built-in feature importances do not correctly assign importance. The take-away message seems to be that model-dependent explanations don't correctly handle interactions like XOR or at least that they are flawed in scenarios like this one, and I don't think this is true. The authors cite TreeExplainer and SHAP, and the SHAP repository actually uses XOR as an example where XGBoost's built-in importance does not correctly assign credit but TreeSHAP/TreeExplainer does.  So I think it's necessary to include a comparison to TreeSHAP/TreeExplainer here (which should correctly assign credit).
https://shap.readthedocs.io/en/latest/example_notebooks/tree_explainer/Understanding%20Tree%20SHAP%20for%20Simple%20Models.html
Minor comment - I don't think it's ever explained what the "X1" and "X2" columns of Table 1 are -- presumably the importances of X1 and X2 under various methods.

4.1.1 - Model monitoring and explaining the loss are great experiments; however, this is also done in TreeExplainer, which should be referenced and compared against. This comparison is especially important because TreeExplainer is cited earlier in the paper and tree models are used for this experiment. TreeExplainer demonstrates how to (1) calculate Shapley values of the model's loss and (2) use loss explanations to perform model monitoring in the presence of feature drift. It's hard to justify publishing a section on explaining model mistakes with feature drift in trees without referencing existing experiments that do exactly that. I think the results will be informative and useful regardless of their exact outcome. It is also possible to calculate Shapley values with standard model-agnostic SHAP methods by treating the function to be explained as the model loss rather than the model output, and I think it would be useful (1) try doing so and (2)  discuss how the ADR method differs from that approach.

4.1.2 - Some minor comments: Point (iii) should not include X3. Also, I'm not sure lines 261-263 contribute to the main points of the section. Which of the linear models is both accurate and parsimonious? I think this is meant to be the one that includes the correct interaction term. However, the paper doesn't include R^2 or other metrics to compare accuracy of these models, and even if such metrics were present I'm not sure there's an important broader point to be made, since the section isn't really about how to train sparse/accurate ML models.

Overall, the experimental results are interesting, but feel incomplete without including comparisons to some existing methods for answering similar questions.
Validity of the findings: No comment.
Additional comments: I like this paper -- I think the method is a great idea and the paper contributes to important discussions in ML about explaining the model vs explaining the dataset. I do believe there are large experimental gaps in the current version of the paper, and that the paper should not be published without addressing these gaps and acknowledging other important methods (e.g., Shapley values of the model loss) for answering the questions the paper poses. However, if these gaps are adequately addressed and sufficient additional experiments are run, the paper would be a valuable resource for the ML community. My most important concerns are the experimental concerns noted under (2) Experimental Design. Several other questions and concerns:

1 - I am not sure there is enough discussion of why we should consider HSIC, AIDC, etc to be non-parametric and not consider explanations based on a complex model (gradient boosting, deep models, etc) to be non-parametric.
a) Sufficiently complex models are often themselves described as non-parametric.
b) No free lunch theorems, etc, in ML make me think that there is probably not a perfect solution to feature dependence just as there is no perfect solution to prediction. I would probably be more comfortable with an explicit discussion of the pros and cons of model-dependent versus model-independent Shapley methods than with an overly broad claim that model-independent methods make no assumptions about the DGP.
c) Some specific pros and cons that come to mind: a strength of the model-independent methods is the guarantee in 2.2 that dependencies will always be detected. A strength of model-dependent methods is that they may be better suited to measure the *strength* of interaction -- for example, from 2.2.1, distance correlation should counterintuitively be higher for a small linear association than for a large nonlinear one, which would likely not hold for an XGBoost model explanation. More discussion of the value of the chosen properties would be very useful -- maybe there are certain cases where it's particularly important that we never accidentally miss an important feature, in which the guarantee from 2.2 is particularly important. In other cases maybe other properties are more important and other methods should be used.

Overall I am not concerned about whether model-independent explanations are useful -- they definitely seem to be! But I think more discussion of the the assumptions made in this approach and how they differ from the assumptions in model-dependent explanations would be helpful.

2 - I think it would help to explain why we need to use Shapley values with HSIC or AIDC as characteristic functions, rather than simply applying them out of the box. For example, a raw pairwise correlation vector is often used to understand relationships between features and labels "in the data". Why not simply calculate, e.g., AIDC between all features and labels? Why is the Shapley value helpful? Personally, I have an intuition for why marginal effects differ from Shapley values for ML models but don't have a similar intuition in the case of HSIC and AIDC.

3 - In Section 4, ADL is frequently discussed and plotted but it is not clear to me how it is calculated (i.e., with HSIC or AIDC, etc). I may have missed these details but I think it would be helpful to clarify exactly what method is being used in the plots. I think it is essential to clarify this before publication.


Instruction: You are one of the reviewers, your task is to write a review for the article. You will be given the title of the article, the number of the round in which the article is located, and your order among the reviewers.

Input:
Title: MODEL INDEPENDENT FEATURE ATTRIBUTIONS: SHAPLEY VALUES THAT UNCOVER NON-LINEAR DEPENDENCIES
Review round: 2
Reviewer: 1

Output:
Basic reporting: + Typos,
Equation 1: S_k under summation, the font should be calligraphic.
Line 173: d=5, A=(0,2,4,6,8)
Experimental design: + Lines 301 and 391. If you have 5 features, wouldn't you have 5*2^4=80 marginal contributions? Maybe put the formula in terms of d.
I think line 426 is correctly 240, because there are 3 independent blocks with size 5 each.
Validity of the findings: No comment.
Additional comments: + I understand the argument why it is much easier to look at per feature Shapley values. But let me say a few words.
Usually the list of the marginal contributions is sparse or consists of few distinct values. For example, if C({1,2, 3}) - C({1,2})=0 then any S where {1,2} \in S would give zero marginal contribution for 3. Similarly if features are not interacting then you get distinct values. Maybe you can state that after quick diagnostics using only 5 Shapley values one can always go back and look into the marginal terms for further diagnostics. But I understand expanding on this point would probably need another paper.

+ I think lines 139-142 (Section 2.2) can use some clarification. One can use non-linear correlations to figure out visible dependence before summing it up into Shapley values.
For XOR, it's true that even non-linear correlations give zero for X1 and X2, but S={X1,X2} has non-zero non-linear correlation. This is why Shapley is non-zero to begin with. Following from previous point above, having same Shapley value would not tell you if relation is X1+X2, X1*X2, X1 XOR X2, .... But going back and checking where Shapley value came from gives you this information.
...
Just wanna mention that seeing Cy(S) = 0 but Cy(S') nonzero, when S \in S' can happen even in a linear additive causal graphs when observing collider child makes the corresponding spouse[s] dependent.


Instruction: You are one of the reviewers, your task is to write a review for the article. You will be given the title of the article, the number of the round in which the article is located, and your order among the reviewers.

Input:
Title: MODEL INDEPENDENT FEATURE ATTRIBUTIONS: SHAPLEY VALUES THAT UNCOVER NON-LINEAR DEPENDENCIES
Review round: 2
Reviewer: 2

Output:
Basic reporting: no comment
Experimental design: Overall I think the changes have addressed almost all of my concerns and there is only one major point I’d like to see addressed before I’d recommend publication. I checked "major revisions" because I would ideally like to see the revisions in response to this point, but I do not think it will be a huge amount of work to address.

Major point:
In section 4.1.1, I need to understand better why SHAPloss more closely matches the ADL plot rather than ADR. It’s very counterintuitive to me that a feature with substantial drift could have a negative contribution to loss, especially a more negative contribution than features with no drift. I feel uncomfortable accepting the paper without a bit more understanding of going on here.

The necessary revision here could consist of extra experiments or simply a thoughtful discussion of why the phenomenon might be occurring, to show the authors have covered their bases and this isn’t an implementation problem. One way I thought this could be addressed is by training a linear model rather than a tree model; since the DGP is linear, there’s always the possibility something weird is happening with the learned tree model. Running SAGE with a trained linear model could be a helpful point of comparison. The counterintuitive findings may persist, and that's OK -- even good -- but overall, I'd just like to see a little more investigation and/or discussion of this point.

Minor point:
It’s a little unintuitive in Figure 2 that feature 1 attribution isn’t 0. This is likely because of normalization, but maybe we could have a second figure to show unnormalized Shapley values? Or some other explanation of why feature 1 doesn’t have 0 attribution.
Validity of the findings: no comment
Additional comments: Minor notes:
Is it possible to make references to Figure 2 clearer? Section 2.2.1 doesn’t reference it at all, and 2.2.2 only a little. It would be good to see a comprehensive discussion of the figure, maybe in a new subsection 2.2.3? This doesn’t need to have much more material; it would just clarify things if 2.2.1 and 2.2.2 were exclusively for defining metrics, then 2.2.3 discussed (a) how Shapley values were normalized, (b) how HSIC/DC/AIDC results differ from R2 results, (c) any discussion you have on the differences among HSIC/DC/AIDC results. The only absolutely necessary new thing to include would be the discussion of how normalization occurred.

In Section 3 a little more analysis of the SHAP and Sunnies results could be useful. They both seem to work well, but as you mentioned earlier in the paper, it’s interesting that Sunnies gets results exactly as good as SHAP without ever training a model in the first place! There are many potential benefits, but one that I think should be discussed here is that many model classes have a lot of trouble learning XORs. Thus, not depending on the model learning this kind of interaction is an advantage. (Optionally, you could potentially even show that SHAP with a scikit-learn decision tree model doesn’t correctly learn importance for X1 and X2 because it can't even learn XOR - thus, SHAP's success here depends on learning a good model whereas Sunnies don't).

In section 5, another potential benefit of Sunnies is that ADR type metrics could be very hard to calculate for Cox loss; right now I believe TreeExplainer supports explaining MSE and binary log-losses, but doesn’t support Cox. This could be very hard to implement because unlike other losses, Cox loss for a single sample can depend on all other labels in the dataset! Having a method for residual explanations that doesn’t depend on the loss having a specific convenient form could be very useful. (It is of course possible to calculate Cox loss explanations using model-agnostic SHAP methods or possibly SAGE; whether these are preferable to Sunnies could come down to runtime, fidelity to the model, or other considerations).

Maybe also worth mentioning that the barbell plot method for Sunnies supports a valuable modeling loop that addresses issues in both supervised learning and model-free HSIC type methods. Generally, supervised models run the risk of being misspecified, which is a drawback. Methods like HSIC, DC, AIDC claim to always be able to detect dependence, but they can’t tell you the form of that dependence, which is a drawback. Sunnies help fill both gaps by iteratively training and tweaking your model until ADL and ADP match well. (Main point here that’s not in the paper is that the lack of an ability to make predictions is a drawback for the otherwise nice property of methods like HSIC).

Lines 371-72, should mention SHAP is also an alternative -- Sunnies get reasonable results compared to SHAP as well (i.e., the same results).

Line 381 mentions ADL but doesn’t define it like ADR and ADP.


Instruction: You are one of the reviewers, your task is to write a review for the article. You will be given the title of the article, the number of the round in which the article is located, and your order among the reviewers.

Input:
Title: CONNECTING VIRTUAL REALITY AND ECOLOGY: A NEW TOOL TO RUN SEAMLESS IMMERSIVE EXPERIMENTS IN R
Review round: 1
Reviewer: 1

Output:
Basic reporting: This article is very clear thanks to its good structure and the right expressions used. Literature references used have a quantity adapted to this paper. The majority of the articles cited are very recent and from reputable publishers.
PeerJ recommends a title between 20 and 60 words to be more concise. In your case, it might be interesting to add some details. While reading the current title, the reader might think of a solution to see the results of R in a virtual reality headset.
Experimental design: The research question is well defined in the introduction section: Make the design of virtual reality environments more accessible to R users without virtual reality skills. In addition, the choices of each r2vr module have been justified to ensure that this solution does not add any extra budget to the targeted users. The major advantage of this approach is the availability of the source code and the various resources that allow this package to be reused in another context.
A-Frame has many differences with other solutions such as Unity3d and Unreal. Both development environments also try to be accessible to as many designers as possible. Unity is the leader in virtual reality applications and Unreal Engine offers a Blueprint interface to be more accessible to novices (Dickson, 2017). It would therefore be preferable to add some justification for your choice of A-Frame for ecologists.
Validity of the findings: The section dedicated to the results gives us a better understanding of the case studies mentioned. This essentially describes the final product and its context.
Ideally, we would have liked to know the significant differences for designers or end-users with another solution. If no measurement was possible, then it would be preferable to detail your observations on r2vr's designers in these 3 cases.
At the end of the discussion, advice on how to use it is given. Improvements and problems concerning the security and confidentiality of data storage could be discussed.
Additional comments: This well-written article proposes an innovative and accessible solution for a whole community. Three applications described have been created using the r2vr package. However, it would be interesting to define the criteria on which the situation has improved in order to facilitate experimentation by ecologists.


Instruction: You are one of the reviewers, your task is to write a review for the article. You will be given the title of the article, the number of the round in which the article is located, and your order among the reviewers.

Input:
Title: CONNECTING VIRTUAL REALITY AND ECOLOGY: A NEW TOOL TO RUN SEAMLESS IMMERSIVE EXPERIMENTS IN R
Review round: 1
Reviewer: 2

Output:
Basic reporting: The manuscript is well written and professional, and I enjoyed reading it.
However, there are some key issues with it in it's current form.
For example, the background section begins with a very narrow interpretaion of virtual reality, namely that of an audiovisual system substituting for real world sensory input.
This description is not adhered to throughout the manuscript: audio is never mentioned again.
I checked the supplementary material and the videos were silent there too.
A further problem with this interpretation is it fails to account for dynamic interaction in VR, in particular where users may move and interact with the environment, with such actions resulting in an enhanced sense of presence [1] and a more immersive, 'vivid' experience.
The authors should be more critical of their interpretation and aware of the limitations this imposes on how others may find their system useful.

The second issue I have is the lack of a clear contribution behind this work.
Many prior work has tackled the issue of simplifying work flows for non programmers/experts so they may engage with VR technology (for example, see [2], [3], and [4]).
Its unclear how r2vr differs from these other attempts both in its implementation and its goal: there is overlap at a high level between psychologists, social sciences, and applied sciences (as in this work) to elicit responses from participants while immersed in a virtual environment.
Therefore I question the utility of r2vr over past attempts/existing solutions.
Furthermore, no comparison is made between r2vr and other systems: I recommend the authors perform a feature comparison to help convey *why* their system is both needed and *how* it compares to, or better yet improves upon, prior art/existing solutions.
This being said I very much welcome the open source nature of r2vr!
It is also worth considering the premise behind elicitation within versus outside of virtual environments.
This is not a trivial matter: please see [5] for discussion on administering questionnaires inside virtual environments.

My final issue revolves around the utility of the case studies.
Insufficient detail is provided, making it difficult to understand the point of their inclusion.
There is no discussion regarding how r2vr was perceived as a benefit (again, no comparison to other workflows/tools are provided), and also all 3 case studies were performed by the same research group involved in developing r2vr.
It is best to trial the system with other researchers not involved in the development/production of r2vr, and perform some qualitative analysis, perhaps involving lengthy debrief interviews with other researchers to get their perspsective and opinion on r2vr as a useful tool to add to their repertoire.
I would like further discussion in the paper around the problems of accessibility and non-power user (programmer) use, including how the authors intend to tackle this problem.

[1]\: Slater, M. (2009). Place illusion and plausibility can lead to realistic behaviour in immersive virtual environments. Philosophical Transactions of the Royal Society B: Biological Sciences, 364(1535), 3549–3557. https://doi.org/10.1098/rstb.2009.0138

[2]\: Vasser, M., Kängsepp, M., Magomedkerimov, M., Kilvits, K., Stafinjak, V., Kivisik, T., Vicente, R., & Aru, J. (2017). VREX: an open-source toolbox for creating 3D virtual reality experiments. BMC Psychology, 5(1), 4. https://doi.org/10.1186/s40359-017-0173-4

[3]\: Jangraw, D. C., Johri, A., Gribetz, M., & Sajda, P. (2014). NEDE: An open-source scripting suite for developing experiments in 3D virtual environments. Journal of Neuroscience Methods, 235, 245–251. https://doi.org/10.1016/j.jneumeth.2014.06.033

[4]\: Brookes, J., Warburton, M., Alghadier, M., Mon-Williams, M., & Mushtaq, F. (2020). Studying human behavior with virtual reality: The Unity Experiment Framework. Behavior Research Methods, 52(2), 455–463. https://doi.org/10.3758/s13428-019-01242-0

[5]\: Regal, G., Voigt-Antons, J.-N., Schmidt, S., Schrammel, J., Kojić, T., Tscheligi, M., & Möller, S. (2019). Questionnaires embedded in virtual environments: Reliability and positioning of rating scales in virtual environments. Quality and User Experience, 4(1), 5. https://doi.org/10.1007/s41233-019-0029-1
Experimental design: No experiment is provided: instead, the authors have provided 3 case studies all of which involved the use of r2vr.
Please see the previous section 'Basic Reporting'.
Validity of the findings: It is unclear what the 'findings' from this work are.
The material is quite light, giving a brief business case for r2vr, offering a high level overview of the system design, and then finally 3 case studies involving its use.
The authors conclude that they 'have demonstrated that it is now possible to create VR experiments via a unique software', but as I have pointed out in the 'Basic Reporting' section, the scientific community already knew this.
Additional comments: There is a typo on line 45, and line 198.
Please consider using footnotes for URLs.
The link provided in line 197 returns a 404.


Instruction: You are one of the reviewers, your task is to write a review for the article. You will be given the title of the article, the number of the round in which the article is located, and your order among the reviewers.

Input:
Title: CONNECTING VIRTUAL REALITY AND ECOLOGY: A NEW TOOL TO RUN SEAMLESS IMMERSIVE EXPERIMENTS IN R
Review round: 2
Reviewer: 1

Output:
Basic reporting: The authors have taken into account my suggestions.
Experimental design: The authors have taken into account my suggestions.
Validity of the findings: The authors have taken into account my suggestions.
Additional comments: This well-written article proposes an innovative solution that is accessible to a whole community. The three applications described were created using the r2vr solution. These case studies, together with the source code made available, demonstrate the interest that should be shown in this new method.


Instruction: You are one of the reviewers, your task is to write a review for the article. You will be given the title of the article, the number of the round in which the article is located, and your order among the reviewers.

Input:
Title: CONNECTING VIRTUAL REALITY AND ECOLOGY: A NEW TOOL TO RUN SEAMLESS IMMERSIVE EXPERIMENTS IN R
Review round: 2
Reviewer: 2

Output:
Basic reporting: L45 to 52: In my review I criticized the narrow interpretation of VR. To say many current VR research applications use only a subset of features, meaning multisensory stimuli, is an unfounded statement. For example, a recent survey paper concluded 84% of application use at least one additional sensory modality (haptics) to vision [1]. Please remove this statement or establish a basis for it.

L144 to 171, 225 to 240, and Table 2: These additions are very welcome. However, they require further refinement. For example, the authors make the case of the existing tools requiring knowledge of C# and Unity. This would be a valid stance to take if it weren't for the fact that the authors then discuss the use of technologies and languages beyond base R. Based on my understanding of the r2vr framework--so please correct me if I am wrong--for end users to do anything more than pop-up existing assets in a VR environment, they would need to familiarize themselves with other technologies and languages e.g., python, HTML. In my original review, my point was for the authors to state clearly "convey *why* their system is both needed and *how* it compares to, or better yet improves upon, prior art/existing solutions." Based on my reading of this rebuttal, the business case behind r2vr is to encourage uptake of VR technologies by ecologists through the stratgey of easing the learning curve e.g., non-power users I referred to in my original review. If this is the case, then the authors should state this clearly in the manuscript. With respect to Table 2, there is a lot of information justifying its inclusion in the rebuttal that is missing from the proposed changes to the manuscript e.g., the first paragraph in gray font on page 6 of the rebuttal. Can the authors please include this detail in the manuscript along with detail regarding how pros and cons were elicited from the literature?

[1]: Melo, M., Gonçalves, G., Monteiro, P., Coelho, H., Vasconcelos-Raposo, J., & Bessa, M. (2020). Do Multisensory stimuli benefit the virtual reality experience? A systematic review. IEEE Transactions on Visualization and Computer Graphics, 1–1. https://doi.org/10.1109/TVCG.2020.3010088
Experimental design: No comments other than what is above
Validity of the findings: No comments other than what is above
Additional comments: No additional comments


Instruction: You are one of the reviewers, your task is to write a review for the article. You will be given the title of the article, the number of the round in which the article is located, and your order among the reviewers.

Input:
Title: CONNECTING VIRTUAL REALITY AND ECOLOGY: A NEW TOOL TO RUN SEAMLESS IMMERSIVE EXPERIMENTS IN R
Review round: 3
Reviewer: 1

Output:
Basic reporting: The authors have addressed my previous comments. I have no more.
Experimental design: The authors have addressed my previous comments. I have no more.
Validity of the findings: The authors have addressed my previous comments. I have no more.
Additional comments: No additional comments


Instruction: You are one of the reviewers, your task is to write a review for the article. You will be given the title of the article, the number of the round in which the article is located, and your order among the reviewers.

Input:
Title: ROBUST PROPORTIONAL OVERLAPPING ANALYSIS FOR FEATURE SELECTION IN BINARY CLASSIFICATION WITHIN FUNCTIONAL GENOMIC EXPERIMENTS
Review round: 1
Reviewer: 1

Output:
Basic reporting: (1) The manuscript needs English proofreading.
Experimental design: (1) several data are unbalanced data. So, it is better to use g-mean instead of classification accuracy.
(2) The authors need to specify how they tune the SVM parameters.
Validity of the findings: Non.
Additional comments: Need to add several recent papers, such as
https://doi.org/10.1007/s11634-018-0334-1


Instruction: You are one of the reviewers, your task is to write a review for the article. You will be given the title of the article, the number of the round in which the article is located, and your order among the reviewers.

Input:
Title: ROBUST PROPORTIONAL OVERLAPPING ANALYSIS FOR FEATURE SELECTION IN BINARY CLASSIFICATION WITHIN FUNCTIONAL GENOMIC EXPERIMENTS
Review round: 1
Reviewer: 2

Output:
Basic reporting: The whole manuscript needs English expert for editing.
Experimental design: For the SVM hyperparameters, the authors need to specify how they choose them.
Validity of the findings: Ok.
Additional comments: Non.


Instruction: You are one of the reviewers, your task is to write a review for the article. You will be given the title of the article, the number of the round in which the article is located, and your order among the reviewers.

Input:
Title: ROBUST PROPORTIONAL OVERLAPPING ANALYSIS FOR FEATURE SELECTION IN BINARY CLASSIFICATION WITHIN FUNCTIONAL GENOMIC EXPERIMENTS
Review round: 1
Reviewer: 3

Output:
Basic reporting: The authors of the paper titled " Robust Proportional Overlapping Analysis for Feature Selection in Binary Classification within Functional Genomic Experiments" have proposed a new feature selection method. The authors have done various analysis to establish their points. I think the authors should address my queries to make this paper accepted.
Experimental design: line 194: Why cross-validation is not considered here?

line 196: Why 500 runs are considered, any specific reason for this? What is the reason behind choosing Random forest (RF), support vector machine (SVM), and k-Nearest neighbours?

Table 1: the gene count for nki dataset is very low, not acceptable for classification. The class sizes (96/48) is not clear to me, can you please explain them.

line 303: If there is a possibility of improving the method as you have mentioned in the last para before the Reference section. Then why have not you used that in this paper?
Validity of the findings: line 222: Do provide the selected gene names and give their biological significance? Why they provide better classification results please justify that?

Table 4: Why maximum genes count are 30?

line 294: ''median absolute deviation (MAD) than the measure of the interquartile range (IQR) used in ?''. Reason for ?
Additional comments: The references are very old please update them with the latest references.

The authors should read the entire manuscript carefully and improve the quality of the language. Lots of typo errors are exist.

According to me, the paper has lacked novelty. The latest related research discussion is required to judge the performance of the proposed feature selection method.


Instruction: You are one of the reviewers, your task is to write a review for the article. You will be given the title of the article, the number of the round in which the article is located, and your order among the reviewers.

Input:
Title: ROBUST PROPORTIONAL OVERLAPPING ANALYSIS FOR FEATURE SELECTION IN BINARY CLASSIFICATION WITHIN FUNCTIONAL GENOMIC EXPERIMENTS
Review round: 2
Reviewer: 1

Output:
Basic reporting: The authors have adequately addressed all the comments that raised in a previous round of review.
Experimental design: non.
Validity of the findings: non.
Additional comments: The authors have adequately addressed all the comments that raised in a previous round of review.


Instruction: You are one of the reviewers, your task is to write a review for the article. You will be given the title of the article, the number of the round in which the article is located, and your order among the reviewers.

Input:
Title: ROBUST PROPORTIONAL OVERLAPPING ANALYSIS FOR FEATURE SELECTION IN BINARY CLASSIFICATION WITHIN FUNCTIONAL GENOMIC EXPERIMENTS
Review round: 2
Reviewer: 2

Output:
Basic reporting: The manuscript needs English proofreading.
Experimental design: non.
Validity of the findings: non.
Additional comments: The manuscript needs English proofreading.


Instruction: You are one of the reviewers, your task is to write a review for the article. You will be given the title of the article, the number of the round in which the article is located, and your order among the reviewers.

Input:
Title: USING APPLICATION BENCHMARK CALL GRAPHS TO QUANTIFY AND IMPROVE THE PRACTICAL RELEVANCE OF MICROBENCHMARK SUITES
Review round: 1
Reviewer: 1

Output:
Basic reporting: The submission describes a method by which to minimise a suite of microbenchmarks for use within a CI pipeline, where quick feedback is desirable. Given a reference "application benchmark" and a suite of microbenchmarks, the method uses coverage metrics to boil down the microbenchmark suite into a (hopefully) representative set of bare-minimum microbenchmarks.

In addition, a method is also proposed for bulking up a too-small microbenchmark suite, by pointing out areas of code which are not yet covered.

The submission is well-written and the spelling and grammar is excellent. Referencing seems sufficient and I have no issues with the structure of the article.

The approach is well-explained and I feel that I have a decent grasp of what the authors are proposing.
Experimental design: Unfortunately, I don't think the design of the experiment is quite right.

It appears that the approach is underpinned by the assumption that microbenchmarks which cover the same functions are equal. From experience, it is not only code coverage that characterises a benchmark, but also how frequently certain parts of code are executed.

To frame this in the context of the paper's approach, suppose that a reference application benchmark AB spends 90% of its time in a tight loop calling a function F1. That means that benchmarking measurements of AB are very much coupled with the performance of F1. Inefficiencies in this function will be amplified because the function is called so frequently. If AB is truly a proxy for the production environment, then F1 had better be really well optimised!

Now suppose that there's a microbenchmark MB1 that executes (covers) exactly the same set of functions as AB, but which only calls F1 very infrequently. Suppose that in MB1 only 1% of runtime is spent in F1. The proposed method would select MB1 because, according to the paper's definition of "practical relevance", this is the ideal microbenchmark: the coverage of both the AB and MB1 are identical. Yet performance measurements of MB1 are highly unlikely to discover inefficiencies in F1, since 99% of runtime is spent elsewhere and any performance irregularities caused by F1 are therefore lost in the noise. Thus MB1 is a poor proxy for AB and if F1 did have performance issues, MB1 would fail to find them, whereas running AB would find them immediately.

The experiments conducted then measure the ability for the proposed approach to minmimise the microbenchmark suite into one with similar code coverage as the application benchmark. I'd argue that instead the evaluation should check whether the reduced benchmark suite has similar ability to detect the same performance problems as the application benchmark. At the very least, I think the evaluation should test whether code coverage is a good proxy for this (I doubt it is, but I'm willing to be proven wrong).

To fix this approach properly, the definition of practical relevance would need to be revised to take "hotness" (frequency of execution) of functions into account. Perhaps function nodes in the call graph need some kind of weighting applied.
Validity of the findings: I think the validity of the findings is limited due to the problems outlined in the previous section.
Additional comments: How do you deal with non-determinism in benchmarks, where different parts of the CFG are executed in subsequent executions? Presumably you'd union the CFGs of some 30 runs?

In Section 3.1 I wasn't sure where 6/16 comes from. Should it be 6/19?

"our proof of concept simply selects a random coverage set in case of a tie" -- This introduces non-determinism into your algorithm. Should you be repeating your experiments (say) 30 times and reporting some measure of variation then? You also mention sampling to get the CFG, which is another source of non-determinism.

"covers the application benchmark to the same extend" -> "... to the same extent"

"coverage of all covered function" -> "coverage of all covered functions" or even better "coverage of all executed functions".

Algorithm 2: "{ A | a \in A \wedge ...}" should that be "{ a | a \in A ...}" (lower case 'a' in the set comprehension)?


Instruction: You are one of the reviewers, your task is to write a review for the article. You will be given the title of the article, the number of the round in which the article is located, and your order among the reviewers.

Input:
Title: USING APPLICATION BENCHMARK CALL GRAPHS TO QUANTIFY AND IMPROVE THE PRACTICAL RELEVANCE OF MICROBENCHMARK SUITES
Review round: 1
Reviewer: 2

Output:
Basic reporting: Following are a few ambiguous statements that should be reviewed.

"Depending on the application scenario, the 288 microbenchmarks from which we extracted call graphs were reduced to either 19, 25, or 27."

The context of private and business customers is not clear.
"e.g., if a function calls different other functions depending on whether or not a value corresponds to a business customer, the respective benchmark workload should represent private and business customers in the same distribution..."

"Thus, the determined coverage corresponds to a maximum value, and the actual value might be smaller. "

"Finally, synthesizing microbenchmarks could be a way to increase coverage of important parts of an application, as, for example, identified by an application benchmark."
Experimental design: One of the objectives of the article is to remove the redundancies in micro-benchmark suites. Authors should address the payload aspect because behavior of the redundant benchmark may vary depending on the inherit nature of the payload.

Call graph related approaches are usually used in software testing domain. How can the definition of SUT be satisfied if one function is called only once in the micro benchmark? Authors have agreed that benchmark cannot be a perfect representative of a real application but in this case, a system cannot be characterized as 'under stress' if a benchmark function is called only once. The authors do claim in the last section that benchmark relevance can be increased up to 100% that seems contradictory to previous statement.

The paper has built the proposed technique using the two algorithms of Chen and Lau (1998) and Rothermel et al. (1999) that overlap with software testing domain. Considerable work has been done in the software testing domain to ensure comprehensive coverage. The authors should provide arguments that these relatively old techniques can achieve desired goals in all kinds of applications.

Why is there a need for calling SortSet(C) in each iteration when C does no change during the later steps in Algo 1?

How n will be determined representing the number of benchmarks to be recommended used in Algo 2?Can it not be determined heuristically?

Figure 2 presents an example with 2 entry points. How efficient will be the proposed technique in case the redundancy is not at application level? Figure 2 shows 19 nodes and its is not clear that how overall coverage comes out to be 6/16?
Validity of the findings: Various results have been presented that are related with the RQ2 however the the crucial aspect of "how to improve execution efficiency" is not obvious from the results and discussion.

The authors have suggested that the proposed approach is valid for application that allow profiling in the context of call graphs. Is there any estimate for the percentage of applications that have this characteristic?

Authors should explore the reasons that all 3 applications  behave very similarly in the context of Figure 7.  It can be argued using Figure 7 that there is no need to include more than 5-6 benchmarks. Is this result dependent on the ordering of the inclusion of the benchmarks?

If 40% coverage is provided by the micro-benchmarks used in this experiment, how satisfied is the community that have been using these micro-benchmarks? Were they getting any contradicting results? If yes, are their any efforts to replace these benchmarks with better micro-benchmarks?
Additional comments: No additional comments


Instruction: You are one of the reviewers, your task is to write a review for the article. You will be given the title of the article, the number of the round in which the article is located, and your order among the reviewers.

Input:
Title: USING APPLICATION BENCHMARK CALL GRAPHS TO QUANTIFY AND IMPROVE THE PRACTICAL RELEVANCE OF MICROBENCHMARK SUITES
Review round: 2
Reviewer: 1

Output:
Basic reporting: OK
Experimental design: OK
Validity of the findings: OK
Additional comments: The authors has fulfilled all previous comments


Instruction: You are one of the reviewers, your task is to write a review for the article. You will be given the title of the article, the number of the round in which the article is located, and your order among the reviewers.

Input:
Title: DISTRIBUTION-PRESERVING DATA AUGMENTATION
Review round: 1
Reviewer: 1

Output:
Basic reporting: The paper presents a color-based data augmentation method based on distribution-preserving color that creates plausible image variation by shifting pixel colors to another point in the image distribution. Although, this research may be interesting, in my opinion the paper has several drawbacks that should be addressed.

The introduction is lengthy and does not focus on the contributions of the paper. Moreover, the paper is sometimes repetitive. For instance, the idea of using the mean-shift process by Fukunaga and Hosteler and Comaniciu and Meer is mentioned in Line 108, in Line 134, in Line 169, and in Line 199. Another example is the "experimental setup" section which repeats ideas about the usefulness of transfer learning instead of directly explaining the “experimental setup”. In addition, the experimental setup is repeated in the Test Settings and Quantitative Results sections.

The sentence “the idea also fits in the feature domain, so it can be applied to the features directly”, in Line 119, should be removed from the introduction and, if considered necessary, explained in more detail in the material and methods sections.
Experimental design: My main concern is that the method is compared with a unique augmentation method (random erasing) on a unique dataset (UC Merced Land-use). But, on the one hand, the authors do not explain why this dataset is chosen and why this dataset can benefit for the proposed method. On the other hand, the authors do not explain why the random erasing augmentation method is used to be compared with the proposed method. Random erasing is a method in which training images are generated with various levels of occlusion, which reduces the risk of over-fitting and makes the model robust to occlusion. The goals of both methods seem to be very different.

In my opinion, the authors should try to strive to find some dataset in which the proposed augmentation method could be essential. Furthermore, the results of the proposed method should be compared with more similar methods based on color transformations.
That is proposed in the abstract (Line 25): “…superior performance of existing color-based data augmentation methods”. Libraries such as albumentations or CLoDSA implements several color based transformations which can be used in such as comparison.

The authors propose an image pyramid to increase the number of extracted features. Then, two figures are included showing the effect of this increase in features. The authors explain (lines 166-167) that “the effect of image quantization errors are reduced, which demonstrates the benefit of the image pyramid approach”. Can the authors explain to what extent these errors are reduced?

Figure 11 shows that the execution time of the method is very large for large images. A comparison of execution times of the proposed method and other color-based data augmentation methods used in the Performance Analysis section should be included.
Validity of the findings: There is not an actual Discussion section in which a comparison of the DPDA method and other similar methods proposed in the literature. The first paragraph of this section only includes a repetition of some characteristics of the DPDA method. Furthermore, it is suggested that “the method obtain superior performance compared to other data augmentation approaches, due to plausible augmentation”. But, as I previously mentioned, the method is only tested on a unique dataset and compared with a unique data augmentation method. Moreover, it is not explained why this dataset can benefit of such as plausible augmentation. Finally, the rest of the Discussion section propose future work. In my opinion, some of this future work could try to be addressed before the method is published in a journal. In particular, methods for speeding up the proposal.
Additional comments: Minor points:
-	Line 171 change pdf by PDF
-	Last sentence in line 221 -> moidified
-	The link to the developed library should be included in the paper.


Instruction: You are one of the reviewers, your task is to write a review for the article. You will be given the title of the article, the number of the round in which the article is located, and your order among the reviewers.

Input:
Title: DISTRIBUTION-PRESERVING DATA AUGMENTATION
Review round: 1
Reviewer: 2

Output:
Basic reporting: The paper is well-written and the explanations are clear. In general, the paper provides enough references. However, I think that in the introduction (paragraph starting in Line 56)   some recent data augmentation methods such as MixUp (Zhang et al, 2017), CutMix (Yun et al. 2019), CutOut (De Vries and Taylor. 2017) or AugMix (Hendrycks et al. 2019) should also be included.

References.
Zhang et al. 2017. mixup: Beyond empirical risk minimization. https://arxiv.org/pdf/1710.09412.pdf.
De Vries and Taylor. 2017. Improved Regularization of Convolutional Neural Networks with Cutout. https://arxiv.org/pdf/1708.04552v2.pdf
Yun et al. 2019. CutMix: Regularization Strategy to Train Strong Classifiers with Localizable Features. https://arxiv.org/abs/1905.04899
Hendrycks et al. 2019. AugMix: A Simple Data Processing Method to Improve Robustness and Uncertainty. https://arxiv.org/abs/1912.02781
Experimental design: My main issue with this paper is that the experiments are quite narrow. The authors have focused on just one dataset, so it is not possible to know if this data augmentation regime serves to several contexts. It would be interesting to know if the data-augmentation procedure presented in the paper works not only for image classification but also for other computer vision problems such as object detection and instance segmentation.

Related to the previous point, the authors only compare their approach with the random erase augmentation method since they claimed that this is the state-of-the-art. But, this augmentation method is not the state-of-the-art for all datasets. So, the authors should clarify this. Moreover, it would be necessary to compare the results with other augmentation techniques, specially with color transformations since the method presented in the paper could be framed in this kind of augmentation. Finally, the authors claim in Line 388 that their method can be easily combined with other color or geometric augmentations to increase the performance of models, but this is not proved at all.
Validity of the findings: The results achieved by the authors are not conclusive since they have only focused on a particular dataset and task. Moreover, a thorough comparison with other methods is necessary.
Additional comments: A plus of this work is that they provide the code for their work. However, there is not any instruction about how to install their program, or to reproduce the experiments of the authors.
An additional issue is that the code is implemented in C++ and it seems to require Visual Studio. This has two problems. First, most people working nowadays in deep learning use Python as programming language, so it is not possible to directly use your code (this is admitted by the authors, but it is important to work in this direction). Similarly, I think that it is not possible to install Visual Studio in Linux so many users will not be able to use it. At least the authors should indicate the requirements to run your code. Moreover, it will be helpful if the authors provide the link to the code in the manuscript. Finally, it would be interesting to integrate your methods in libraries such as Albumentations (https://albumentations.readthedocs.io/) or Clodsa (https://github.com/joheras/clodsa).


Instruction: You are one of the reviewers, your task is to write a review for the article. You will be given the title of the article, the number of the round in which the article is located, and your order among the reviewers.

Input:
Title: DISTRIBUTION-PRESERVING DATA AUGMENTATION
Review round: 2
Reviewer: 1

Output:
Basic reporting: The issues reported in the previous version have been addressed.
Experimental design: A minor issue with the Intel Image Classification dataset is that the authors did not include the results obtained using DPDA+FI. The same happens in Table 7. Please be consistent when presenting the results from Tables 1 to Table 7.
Validity of the findings: The approach presented by the authors seems to consistently improve the resulsts in several datasets.
Additional comments: The authors have addressed all my concerns, and it is great that they provide the code for both Windows and Linux. As the authors suggest in the response porting their code to Python will be quite time consuming and is independent from the presented method.


Instruction: You are one of the reviewers, your task is to write a review for the article. You will be given the title of the article, the number of the round in which the article is located, and your order among the reviewers.

Input:
Title: DISTRIBUTION-PRESERVING DATA AUGMENTATION
Review round: 2
Reviewer: 2

Output:
Basic reporting: - The authors have improved the manuscript in overall following the suggestions made by the reviewers.

- Although source code and data source links were provided, authors should consider Python to reach a bigger audience.
Experimental design: - The experimental study was extended by including 2 datasets and a segmentation analysis, which was also suggested in the previous revision.

- There is a concern in Figure 13. The authors should have plotted the execution time regarding the other methods. According authors, there is a great gap between compared methods and the proposal. A section where a balance is made between performance and runtime metrics is needed, e.g. authors could elaborate a new metric where both mentioned metrics are merged.
Validity of the findings: - The authors provided an extended discussion about the proposal and the possibilities to combined it with other data augmentation methods.

- Finally, the authors should consider the execution time as an important issue. So far, the time reported in section “Execution Time Analysis” was for 1 image. An analysis about how long a complete execution takes in every dataset should be included.
Additional comments: No additional comments


Instruction: You are one of the reviewers, your task is to write a review for the article. You will be given the title of the article, the number of the round in which the article is located, and your order among the reviewers.

Input:
Title: A GLOBAL-LOCAL NEIGHBORHOOD SEARCH ALGORITHM AND TABU SEARCH FOR FLEXIBLE JOB SHOP SCHEDULING PROBLEM
Review round: 1
Reviewer: 1

Output:
Basic reporting: More recent literature should be included.

The authors should clearly present the novelty and contribution of their algorithm. In my opinion, it seems to be a conventional hybridization of population based and local search methods, which have been addressed adequately in the past.
Experimental design: Benchmark sets are not sufficient. Other existing problem instances should be considered.

Further reference algorithms should be considered.
Validity of the findings: The algorithm proposed actually failed to match existing ones for some benchmark problem instances, given the very limited sets. Moreover, CPU time is not comparable with papers published years ago. These issues all make its performance questionable.
Additional comments: No additional comments


Instruction: You are one of the reviewers, your task is to write a review for the article. You will be given the title of the article, the number of the round in which the article is located, and your order among the reviewers.

Input:
Title: A GLOBAL-LOCAL NEIGHBORHOOD SEARCH ALGORITHM AND TABU SEARCH FOR FLEXIBLE JOB SHOP SCHEDULING PROBLEM
Review round: 1
Reviewer: 2

Output:
Basic reporting: See below
Experimental design: See below
Validity of the findings: See below
Additional comments: -Compare computational results with none-parametric statistical test like Friedman test, Wilcoxon signed-rank test etc. as advised in the literature [J. Derrac, S. Garcia, D. Molina, F. Herrera, A practical tutorial on the use of nonparametric statistical tests as a methodology for comparing evolutionary and swarm intelligence algorithms, Swarm Evol. Comput. 1(1) (2011) 3-18].
-Why tabu search is selected as the main search procedure? More discussions may be useful.
-For mathematical formulation of the problem see [Ozguven et al., Mixed integer goal programming models for the flexible job-shop scheduling problems with separable and non-separable sequence dependent setup times, Applied Mathematical Modelling, Volume 36, Issue 2, February 2012, Pages 846-858]
-There is more recent literature on the problem that need to be considered by replacing older references. See for example
[Huang, X. and Yang, L. (2019), "A hybrid genetic algorithm for multi-objective flexible job shop scheduling problem considering transportation time", International Journal of Intelligent Computing and Cybernetics, Vol. 12 No. 2, pp. 154-174. https://doi.org/10.1108/IJICC-10-2018-0136]
[Baykasoglu, A., Madenoglu, F.S., Hamzadayi, A., Greedy randomized adaptive search for dynamic flexible job-shop scheduling, Journal of Manufacturing Systems, 56, 425-451, 2020].
-Gantt charts are not necessary.