Instruction: You are one of the reviewers, your task is to write a review for the article. You will be given the title of the article, the number of the round in which the article is located, and your order among the reviewers.

Input:
Title: MULTI-OBJECTIVE SIMULATED ANNEALING FOR HYPER-PARAMETER OPTIMIZATION IN CONVOLUTIONAL NEURAL NETWORKS
Review round: 1
Reviewer: 1

Output:
Basic reporting: The language is mostly clear and easy to understand. Throughout the paper, there are some minor mistakes regarding grammar, phrasing or word choice (e.g. a missing article in line 2), which could benefit from having a native speaker proof-read the text.

Relevant literature is cited and sufficient context is provided. Some more references in lines 28-35 would be helpful (or, alternatively, relating matching references in the following sentences to the examples given in these lines).

Figure 3 (SA-pseudocode):
    Since the acceptance criterion is an important and mostly unique feature of SA, I would recommend including it in the pseudocode directly.

Figure 4:
    The three subfigures use different axis scales, which makes comparison difficult.
    It is also not helpful that the symbols for the SA final front are different between subfigures 3b and 3c.
    I would prefer more detailed figure captions that already describe the difference between initial and final fronts without having to refer to the text.
Experimental design: The topic of the article undoubtedly fits within the scope of the journal and the research question is clearly stated and relevant.

Section 3.1:
    The allocation between inner and outer loop repetitions is usually irrelevant for SA, as long as the cooling factor is adjusted accordingly - however, if this is intended to be shown experimentally, two arbitrarily chosen configurations seem insufficient to me.
    More important is the choice of cooling factor or minimum temperature (choosing one determines the other given a fixed budget of moves), which is not mentioned at all here.

An important part that is missing is a description of or reference to the single-objective SA algorithm used (some parts can be inferred from Section 4, but a definition should certainly be given alongside the algorithm description of MOSA).
Validity of the findings: The authors use publically available benchmark datasets for their evaluation, which are also referenced in the article.
The source code is available online, which is very positive!
It would be great if the raw evaluation results could also be posted in that repository.

Section 3.3:
    I don't think that reporting average objective values is useful here - the whole point of Pareto-fronts is that the quality of a set of solutions cannot be easily expressed as a single value.
Section 3.3.4:
    It seems like the state-of-the-art models used have the same structure across all three benchmark datasets, while the model produced by MOSA was different for each dataset - if this is the case, it should be mentioned, as this would give an advantage to the MOSA-model.

Given that the main limitation for more definitive conclusions is the low number of solutions on the final fronts, evaluations with more iterations would be quite interesting.

I also think that the article would benefit from a comparison with other state-of-the-art algorithms for automated CNN architecture generation (as referenced in Section 1) in order to be able to judge the performance of MOSA and SA compared to population-based approaches.
Additional comments: A short description of the terms Pareto-optimal and Pareto-front would be beneficial (in particular since multi-objective optimization is described - anyone unfamiliar with MOO is unlikely to know these terms).

Section 2:
    In the paragraph starting at line 120, several formulas or variables seem to be missing in the text.

    The role and functionality of the different layers is not easily understandable (maybe also due to missing formulas); section 2.1 could be expanded a bit. In particular, the types of layers (convolution, pooling, strive, fully connected,...) mentioned differ between the text, the attached figure and the table listing the parameters (Table 2). The same happens in Section 2.3, where yet another set of different layers or components is used.


Instruction: You are one of the reviewers, your task is to write a review for the article. You will be given the title of the article, the number of the round in which the article is located, and your order among the reviewers.

Input:
Title: MULTI-OBJECTIVE SIMULATED ANNEALING FOR HYPER-PARAMETER OPTIMIZATION IN CONVOLUTIONAL NEURAL NETWORKS
Review round: 1
Reviewer: 2

Output:
Basic reporting: The work is well structured, and mostly written in a fluent and correct English (bar few typos). The code is provided as GitHub repo, though it is not mentioned in the paper.

Some typos:
- line 39: NAS, which -> remove which
- lines 81-82, does it refer to Lu et al. or Gulcu and Kus?
- line 114: There might be multi fully connected... -> multiple
- line 208: neighbouhood -> neighbourhood
Experimental design: Motivation for the choice of the algorithm aside, the description of the MOSA algorithm is very generic, and impossible to replicate. It is not explained how a new solution is generated, nor any parameter is given (initial and final temperature value, cooling rate, temperature length, etc.). The same applies for the single-objective SA (whose objective function is stated only in Section 4).

The experimental section lacks details and clarity too. How many tuning experiments have been performed? There is no statistical analysis of the results. Which is the computational environment, and which libraries and software packages have been used? How long does it take to perform a full run of MOSA? NAS is a very trendy topic at the moment (see the list at [1]), and the rush to publish papers on this topic may lead to inconsistent experimental practices. Please try to follow the checklist described in [2], which is mostly valid also when there is no comparison against other NAS methods.

Some descriptive parts are a bit unclear, such as the introduction to multi-objective optimization (Section 1, lines 54-63), the description of CNNs (Section 2.1, where the inline math notation is also missing), or the solution representation (Section 2.3). A bit more explanation would be helpful in particular for readers not fully acquainted with CNNs.

There is no explanation about the choice of hyperparameters or their ranges. The authors restrict the number of hyperparameters and their value by "taking into account two criteria, namely computational complexity and classification performance" (Section 2.2, lines 132-134). How is this restriction done? Are all the parameters categorical? For example, in Table 2 there is no mention of the number of layers, how many are used? The authors refer to their previous work (Gulcu and Kus 2020) but a short explanation should also be included in the present paper.

References

[1] https://www.automl.org/automl/literature-on-neural-architecture-search/

[2] Best Practices for Scientific Research on Neural Architecture Search, Marius Lindauer and Frank Hutter, https://arxiv.org/abs/1909.02453
Validity of the findings: This work is very similar to a recent paper by the same authors (cited as Gulcu and Kus 2020), where a variant of SA was used to automatically obtain a CNN. The main difference is that the present work models the NAS task as a bi-objective optimization problem, where the two objectives are classification accuracy and complexity of the network, in terms of the number of parameters obtained. It is however not very clear to me what is the precise goal of this paper, whether to demonstrate the feasibility of the task (some previous works already model NAS as bi-objective optimization problem), or to propose a new algorithm for NAS (there are no sufficient details and experiments neither to claim a new state of the art, nor to properly assess the method). Two additional questions: (i) it seems to me that the MOSA algorithm is generic enough to tackle any NN task, with only the set of N hyperparameters and the data to discriminate, why focusing on CNNs? and (ii) why a multi-objective variant of Simulated Annealing, and not a different algorithm?

To assess the results of the NAS method, it would be interesting to also observe how it performs against LeNet, which is used by the authors as starting solution of their MOSA. The discussion is also short and not very deep. In particular, there is no assessment of the impact of MOSA on the final networks obtained, nor any description of the final networks. On the contrary, the authors state (Section 4, lines 373-374) that the short trainings employed by MOSA can be misleading. In other words, it is not clear whether, or up to which extent, the results reported are due to MOSA or to the longer training of LeNet.

There is no comparison against other NAS methods.
Additional comments: In this work, the authors propose a multi-objective Simulated Annealing (MOSA) algorithm to generate Convolutional Neural Networks (CNNs) taking into account two objectives, accuracy and the number of parameters of the network.

This work belongs to a thriving field of research, namely Neural Architecture Search (NAS, the AutoML task aimed at automatically designing and configuring neural networks), and is perfectly within the journal scope.

I have, however, some concerns about this work, about some unclear descriptions (including the objective of this work), a lack of details about implementation, and a weak experimental section and analysis. The list of concerns is reported in the sections above.

Overall, I do not feel this work can be published in the current state. It is however difficult to assess the performance of the proposed method, as the experiments reported do neither support nor reject its validity.

Other minor comments:

A short description of the three benchmark should also be included.

There is at time a possibly confusion when using the term "parameters", which refers to different things, depending on the level we are operating (e.g. in line 26). Since in ML the parameters are usually the weights, I suggest to maintain a consistent division between "parameters" (the weights of the NN), "hyperparameters" (those optimized by the MOSA), and "SA parameters" (temperature, etc).

Some citations are missing for the works mentioned in the literature review (RS, BO, etc, lines 28-36).


Instruction: You are one of the reviewers, your task is to write a review for the article. You will be given the title of the article, the number of the round in which the article is located, and your order among the reviewers.

Input:
Title: MULTI-OBJECTIVE SIMULATED ANNEALING FOR HYPER-PARAMETER OPTIMIZATION IN CONVOLUTIONAL NEURAL NETWORKS
Review round: 2
Reviewer: 1

Output:
Basic reporting: The language is improved in the revision. Some mistakes remain, mostly missing articles and prepositions (e.g. in line 13: "we pose 'a' CNN hyper-parameter [...] as 'a' bi-criteria [...]"

Missing literature references have been provided. Since this work seems to build at least partially on prior work by the same authors, the previous paper should already be mentioned in the introduction.

The structure of the article could use some further revision to ensure that each section contains all relevant information. In particular, the descriptions of the algorithms used are distributed across multiple sections (Section 3.2.3 should be part of the algorithm description, the description of SA used in this paper appears piecewise in 3.3.1 (552ff) and 4 (624f)). Section 3.2.4 should already state which solutions are selected for longer training (given later in 3.3.2).

Figure 6: The difference between SA and MOSA is barely recognizable - I recommend that you use a log scale for the x-axis instead.
Experimental design: The description of the methods and configuration has been much improved in the revision.
What's still missing is a short description of the RS algorithm (line 540, presumably similar to SA, except that all solutions are accepted?).

The description of the neighborhood and moves could be improved. It is currently not clear, which moves exist (e.g. is "modifiying each convolution block" a single move that affects every conv. block at the same time? Or is one block chosen at random? Are the three subitems all performed sequentially or is one of them chosen? What are the weights for this choice?)
It also seems like the neighborhood parameters (the composition and weights of the different moves) have been chosen arbitrarily. Some motivation should be given for these choices (why increase the probability of adding blocks over time? Why at this rate? Why is there no move that removes blocks?)
Alternatively, including these parameters in any kind of tuning could provide better performance and robustness.

Regarding tuning:
- The initial temperature selection strategy is intended to be a reasonable guess when suitable values are not available prior to the SA run (online tuning). In this case, where offline tuning is performed, potentially much better values could be achieved, either by trying out different values manually or using an automated parameter tuner. The automated initial temperature selection strategy could be used to provide an initial starting value for this tuning.
- The ratio between inner and out loops (e.g. the cooling rate) is mostly irrelevant in this case (geometric cooling, fixed initial and end temperature, fixed iteration budget) for SA in general.
Validity of the findings: The term "competitive results" is misleading - there is a clear tradeoff between the complexity of the model (FLOPs) and the accuracy. To show actual competitiveness with the state-of-the-art, MOSA would have to generate at least one solution along the pareto front that achieves similar accuracy (which is the objective optimized in those other papers).
This is not to say that MOSA has no merit - it simply focuses on another objective criterion; it performs well for a different problem.

At the same time, there is lost potential of MOSA due to missing tuning of neighborhood parameters, and the reduction of the training dataset.
If the full CIFAR-10 dataset is used at least for the longer training runs in the final evaluations (Section 3.3.2), the impact of the latter is only minor, but the model might still give better results under the full dataset.

Another point is that you should not use the best results for MOSA and compare to median results for SA and RS. Either use median or best results for both (median is already skewed anyway, since you pick only good solutions for longer training).
Additional comments: Some minor comments regarding specific lines:

134: FLOPs is Floating Point Operations (not Floating Point Operations per Second (FLOPS)). FLOPS is also mistakenly used in line 544.

280: k depends on the number of objectives in general (not just 2)

333ff: It is not clear whether the return-to-base mechanism is applied to a random solution in the archive or only a solution that dominates X'

555: Why is SA longer than MOSA? The overall algorithm structure is the same and for comparability, training and evaluation of solutions should be performed the same way for both.

562ff: Generational Distance is already described in Section 2.4, repeating the description is not necessary here


Instruction: You are one of the reviewers, your task is to write a review for the article. You will be given the title of the article, the number of the round in which the article is located, and your order among the reviewers.

Input:
Title: MULTI-OBJECTIVE SIMULATED ANNEALING FOR HYPER-PARAMETER OPTIMIZATION IN CONVOLUTIONAL NEURAL NETWORKS
Review round: 2
Reviewer: 2

Output:
Basic reporting: The authors rewrote a large part of the paper, including several additional sections. Overall the paper reads well, but I feel one more pass to polish the text would make it more readable ( e.g. by breaking down the longer periods and adding some punctuation).

At times there is a wrong spacing between in-line formulas and the following text (with space missing) or punctuation (with space, to be removed), e.g. lines 103-104.
Experimental design: The authors now include more details about the algorithmic choices and the experimental setup. While some of these choices are motivated (such as the initial and final temperature values, analyzed thoroughly), other ones seem arbitrary, such as the probabilities for selecting an operator in the local moves. The same applies for the analysis of the cooling rate: only three values are evaluated with three experiments each. The authors report that a statistical test showed no differences, but with only 3 experiments for each value it's hard to expect anything different. More experiments (e.g. 15 for each value, or another value determined from the beginning) might show a different outcome. Another potential issue with the experiment reported is that, for a short MOSA run, a cooling scheme of 0.85 might be still too high and allow too much diversification in the beginning, therefore spending too much time on poor quality solutions, while a lower cooling rate (e.g. 0.4, or 0.6) could immediately focus towards better quality solutions.

The hyperparameters of SGD are often included in this kind of studies, why are they left out here?

The underlying issue here is the extremely expensive function evaluation of each MOSA iteration, for which there is no real shortcut. The authors report 50 hours for one MOSA run on their machines, so it is understandable that they try to minimize the overall time. Unfortunately, too few experiments will be uninformative.
Validity of the findings: The same applies for the comparison with SA and RS reported in Figures 4 and 5: the frontiers are very close, there is probably no statistical difference. But again, this may be a limitation due to the few experiments considered.

The authors now report a comparison with other methods (we can cavil about implementation details, but that's essentially the goal of NAS: automatically configure a neural network), which are reported to have much lower error rates, at the cost of a much higher complexity of the resulting NNs. It would be interesting to see a discussion about the resulting networks from the various methods: the network number 1 obtained with MOSA has a comparable number of parameters than other NAS methods who have higher accuracy, but obtained at a much higher computational cost. Does this mean the networks found with MOSA are sparser?

Overall, it appears that MOSA improves over its starting solution, and obtains results that can be compared in some terms with other methods; it is less clear whether it really improves over the SA previously proposed by the same authors.
Additional comments: I thank the authors for addressing my comments on the first version of this paper. I think the comparisons now reported give an indication about the potential of the method, but more experiments are needed to fully appreciate the extent of the improvement over existing methods (including the SA of the same authors).

Minor:
- line 46: it is shown in many studies... : there should be some citations here
- line 172: an input is not necessarily an image. Aside form other matrix/tensor inputs, 1-D CNNs can be used for example for time series.


Instruction: You are one of the reviewers, your task is to write a review for the article. You will be given the title of the article, the number of the round in which the article is located, and your order among the reviewers.

Input:
Title: MULTI-OBJECTIVE SIMULATED ANNEALING FOR HYPER-PARAMETER OPTIMIZATION IN CONVOLUTIONAL NEURAL NETWORKS
Review round: 3
Reviewer: 1

Output:
Basic reporting: The authors have adequately addressed my concerns regarding the structure.
The english has been improved in several places. In my personal opinion, the writing could still benefit from being read by a native speaker, but it is certainly clear enough to follow.

The figures are much clearer now, log-scale was certainly the right choice for all three of them.

As a minor point on formatting, I would recommend to add some spacing and a uniform style for the move descriptions in 3.1.2
Experimental design: The description of the moves is clear after the revision.

Many of the values and the chosen operations themselves still seem arbitrarily selected, but I recognize that in-depth tuning is practically infeasible.
Unfortunately, the low number of evaluations and the low number of iterations makes it hard to determine how and by how much these parameters influence the quality of results.
I suspect that SA in general may not be the best approach for this kind of problem, as it typically derives its strength from fast applications of large numbers of moves (millions, if not billions). The same algorithm using the Metropolis criterion at a fixed (low) temperature or even random walk accepting only improving solutions might result in the same or even better solutions, as reaching a local optimum in the given number of iterations is unlikely anyway and these simpler alternatives would be easier to tune and waste less time on worsening moves.
The same holds for the structure of the neighborhood, which is quite complex and large for SA. Breaking this up into multiple, simpler types of moves would allow a more in-depth analysis of the impact of each type of change and more focussed tuning - however, it would require proportionally more iterations to reach the same solutions, which of course is undesirable here due to the large evaluation cost.
Of course this does not invalidate the experimental setup of the paper, but it should be kept in mind when attributing the success of the algorithm to any specific component (except the multi-objective setting, which is independent of the solution method itself).
Validity of the findings: As mentioned before, the statistical significance of the results is hard to determine with the given number of evaluations, in particular when comparing between MOSA, RS, and SA.
Certainly more experiments in the future would help in determining the best approach for this problem (see also my comments above).
Additional comments: No further concerns.


Instruction: You are one of the reviewers, your task is to write a review for the article. You will be given the title of the article, the number of the round in which the article is located, and your order among the reviewers.

Input:
Title: MULTI-OBJECTIVE SIMULATED ANNEALING FOR HYPER-PARAMETER OPTIMIZATION IN CONVOLUTIONAL NEURAL NETWORKS
Review round: 3
Reviewer: 2

Output:
Basic reporting: There are still some imprecision in the grammar, such as "We pose A CNN hyper-parameter optimization problem..." in the conclusion, and some missing articles in front of the method names (MOSA and SA) throughout the whole paper.
Experimental design: No comment
Validity of the findings: To overcome the main limitation of NAS studies, namely the computational complexity of the task, I was recently made aware of this work: https://arxiv.org/pdf/2008.09777.pdf

The authors provide a surrogate benchmark of neural architectures on CIFAR-10, so that the evaluation of an architecture becomes way cheaper.

This is a preprint that was released few months ago, after the first submission of this paper, so of course it is not to be taken into consideration for this paper. However, should the authors pursue this line of research, I recommend they look into this benchmark, to be able to perform statistical evaluations of their (and other authors') methods.
Additional comments: Aside from some minor correction in the writing, I have no further comments on this paper.


Instruction: You are one of the reviewers, your task is to write a review for the article. You will be given the title of the article, the number of the round in which the article is located, and your order among the reviewers.

Input:
Title: LATENT BASED TEMPORAL OPTIMIZATION APPROACH FOR IMPROVING THE PERFORMANCE OF COLLABORATIVE FILTERING
Review round: 1
Reviewer: 1

Output:
Basic reporting: Paper is well introduced and written in clear English.
Experimental design: Experiments cover theoretical hypothesis.
Validity of the findings: Findings are valid as authors claims with given significance evidence.
Additional comments: No additional comments


Instruction: You are one of the reviewers, your task is to write a review for the article. You will be given the title of the article, the number of the round in which the article is located, and your order among the reviewers.

Input:
Title: LATENT BASED TEMPORAL OPTIMIZATION APPROACH FOR IMPROVING THE PERFORMANCE OF COLLABORATIVE FILTERING
Review round: 1
Reviewer: 2

Output:
Basic reporting: - The English language that used in this article is clear and unambiguous.
- The literature that are provided in the article are sufficient and up to date.
- clear methodology for tackling and solving the problem with good structure.
- The datasets and raw data that were used are real-world datasets, which are clear defined and cited.
- The presented result are met the hypotheses of the paper
Experimental design: no comment
Validity of the findings: no comment
Additional comments: - The term collaborative that is mentioned in the title  is not being defined and used in the article. in deed is not illustrated to improve the performance of filtering
- The cluster formation is not explain well, how many clusters? based on what? etc.


Instruction: You are one of the reviewers, your task is to write a review for the article. You will be given the title of the article, the number of the round in which the article is located, and your order among the reviewers.

Input:
Title: LATENT BASED TEMPORAL OPTIMIZATION APPROACH FOR IMPROVING THE PERFORMANCE OF COLLABORATIVE FILTERING
Review round: 1
Reviewer: 3

Output:
Basic reporting: no comment
Experimental design: no comment
Validity of the findings: no comment
Additional comments: The paper is well written, and it has the potential to change the way temporal drifting and sparsed datasets are treated.  The authors have developed a method to apply a Latent-based Temporal Optimization (LTO) approach to improve the prediction accuracy of CF by learning the past attitudes of users and their interests over time. The methodology part is well explained. However, few issues are raised.
-	the results need to be improved. The reasons behind each finding should be explained by the support of previous related studies or authors' justifications.
-	there exist some typo errors in figures 9 and 10. Specifically, in fig 9, the label font type is not consistent with the ones used in other figures. Also, no scaling is provided at the horizontal axis. The authors may need to scale the axes appropriately. While in fig 10, the horizontal axis title is not very clear.


Instruction: You are one of the reviewers, your task is to write a review for the article. You will be given the title of the article, the number of the round in which the article is located, and your order among the reviewers.

Input:
Title: LATENT BASED TEMPORAL OPTIMIZATION APPROACH FOR IMPROVING THE PERFORMANCE OF COLLABORATIVE FILTERING
Review round: 2
Reviewer: 1

Output:
Basic reporting: The work is clear, author contributions to stat-of-art is reasonable.
Experimental design: Experiments cover theory.
Validity of the findings: results are reasonable
Additional comments: No additional comments


Instruction: You are one of the reviewers, your task is to write a review for the article. You will be given the title of the article, the number of the round in which the article is located, and your order among the reviewers.

Input:
Title: LATENT BASED TEMPORAL OPTIMIZATION APPROACH FOR IMPROVING THE PERFORMANCE OF COLLABORATIVE FILTERING
Review round: 2
Reviewer: 2

Output:
Basic reporting: - The level of English language of the paper  is good enough
-  The paper is rich with the contents and has up to date literature references
- Paper is well structured.
- The significant result have met the objective
Experimental design: The experimental and methodology that presented in the paper are well defined
Validity of the findings: The finding of the paper and conclusions are well stated, which linked to objectives of research paper and  supported the arguments
Additional comments: The contribution of the paper is a publishable values


Instruction: You are one of the reviewers, your task is to write a review for the article. You will be given the title of the article, the number of the round in which the article is located, and your order among the reviewers.

Input:
Title: LATENT BASED TEMPORAL OPTIMIZATION APPROACH FOR IMPROVING THE PERFORMANCE OF COLLABORATIVE FILTERING
Review round: 2
Reviewer: 3

Output:
Basic reporting: No comment
Experimental design: No comment
Validity of the findings: No comment
Additional comments: The authors have done well to follow all comments and made all the necessary corrections.


Instruction: You are one of the reviewers, your task is to write a review for the article. You will be given the title of the article, the number of the round in which the article is located, and your order among the reviewers.

Input:
Title: STATE-OF-THE-ART IOV TRUST MANAGEMENT A META-SYNTHESIS SYSTEMATIC LITERATURE REVIEW (SLR)
Review round: 1
Reviewer: 1

Output:
Basic reporting: Revision:
-The section Open research issues should be enhanced(long term vision on future research challenges. This is very important for a good survey).
-Important references are missing and should be added:
-Mohammad Wazid,et al:LAM-CIoT: Lightweight authentication mechanism in cloud-based IoT environment. J. Netw. Comput. Appl. 150 (2020)
-Jun Zhou,et al:Secure and privacy preserving protocol for cloud-based vehicular DTNs.IEEE Transactions on Information Forensics and Security 10 (6), 1299-1314,2015.
-Mohammad Wazid, et al:Design and Analysis of Secure Lightweight Remote User Authentication and Key Agreement Scheme in Internet of Drones Deployment. IEEE Internet of Things Journal 6(2): 3572-3584 (2019).
-Harsha Vasudev,et al:Secure message propagation protocols for IoVs communication components. Comput. Electr. Eng. 82: 106555 (2020).
-Sravani Challa, et al: Design and analysis of authenticated key agreement scheme in cloud-assisted cyber-physical systems. Future Gener. Comput. Syst. 108: 1267-1286 (2020)
Experimental design: no comment
Validity of the findings: no comment
Additional comments: Revision:
-The section Open research issues should be enhanced(long term vision on future research challenges. This is very important for a good survey).
-Important references are missing and should be added:
-Mohammad Wazid,et al:LAM-CIoT: Lightweight authentication mechanism in cloud-based IoT environment. J. Netw. Comput. Appl. 150 (2020)
-Jun Zhou,et al:Secure and privacy preserving protocol for cloud-based vehicular DTNs.IEEE Transactions on Information Forensics and Security 10 (6), 1299-1314,2015.
-Mohammad Wazid, et al:Design and Analysis of Secure Lightweight Remote User Authentication and Key Agreement Scheme in Internet of Drones Deployment. IEEE Internet of Things Journal 6(2): 3572-3584 (2019).
-Harsha Vasudev,et al:Secure message propagation protocols for IoVs communication components. Comput. Electr. Eng. 82: 106555 (2020).
-Sravani Challa, et al: Design and analysis of authenticated key agreement scheme in cloud-assisted cyber-physical systems. Future Gener. Comput. Syst. 108: 1267-1286 (2020)


Instruction: You are one of the reviewers, your task is to write a review for the article. You will be given the title of the article, the number of the round in which the article is located, and your order among the reviewers.

Input:
Title: STATE-OF-THE-ART IOV TRUST MANAGEMENT A META-SYNTHESIS SYSTEMATIC LITERATURE REVIEW (SLR)
Review round: 1
Reviewer: 2

Output:
Basic reporting: no comment
Experimental design: no comment
Validity of the findings: no comment
Additional comments: The authors conducted a systematic literature review (SLR) using PRISMA guidelines to find historical and current trends of trust management methodologies in IoV/VANETs. They have screened several conference and journal papers and selected 31 papers finally to provide an in-depth analysis of them. Authors lay out some open challenges and future research directions with various kinds of important summary tables. The following are some of my observations about the article.

1. Why context-awareness should be part of the review was not clear to me. It should be elaborated for the reader in the introduction part.

2. What exactly is a context in terms of TM should be defined?

3. Texts in 120-123 appear to be misplaced.

4. A table highlighting the gray areas that relate to the open research challenges will be very useful.

5. Keywords that are used in lines 220-225 show formatting error.

6. What exactly was the reason conference papers excluded in some places of the summary should be detailed?

7. How many journals and conference papers were excluded based on their filtering, might be important for the readers especially in the 3rd phase.

8. Please check the sentence of 280-282. Not clear.

9. Figure 4 can represent at least the fourth phase.

10. Assumptions section from line 429 should be elaborated more to relate to the table.

11. "In a real situation, this may result in highly adverse security consequences" such as?

12. Give references for 488-489.

13. Can table 3 have the information from fig 6 to show the technology overlap?


Instruction: You are one of the reviewers, your task is to write a review for the article. You will be given the title of the article, the number of the round in which the article is located, and your order among the reviewers.

Input:
Title: STATE-OF-THE-ART IOV TRUST MANAGEMENT A META-SYNTHESIS SYSTEMATIC LITERATURE REVIEW (SLR)
Review round: 2
Reviewer: 1

Output:
Basic reporting: This study focuses on trust management (TM) in the IoV/VANETs/ITS
(intelligent transport system). Trust has always been important in vehicular networks to ensure safety. A variety of techniques for TM and evaluation have been proposed over the years, yet few comprehensive studies that lay the foundation for the development of a “standard” for TM in IoV have been reported.

Important references are missing and should be added:
Sravani Challa, et al: Design and analysis of authenticated key agreement scheme in cloud-assisted cyber-physical systems. Future Gener. Comput. Syst. 108: 1267-1286 (2020)
Mohammad Wazid, et al:Design and Analysis of Secure Lightweight Remote User Authentication and Key Agreement Scheme in Internet of Drones Deployment. IEEE Internet of Things Journal 6(2): 3572-3584 (2019).
S Jangirala, et al:Designing secure lightweight blockchain-enabled RFID-based authentication protocol for supply chains in 5G mobile edge computing environment.IEEE Transactions on Industrial Informatics, DOI: 10.1109/TII.2019.2942389,2020
MB Mollah, et al:Secure data sharing and searching at the edge of cloud-assisted internet of things.IEEE Cloud Computing 4 (1), 34-42,2017.
Harsha Vasudev, et al:Secure message propagation protocols for IoVs communication components. Comput. Electr. Eng. 82: 106555 (2020)
Z.Yan,et al:A security and trust framework for virtualized networks and software‐defined networking.Security and communication networks 9 (16), 3059-3069,2016
Q.Jing,et al:Security of the Internet of Things: perspectives and challenges.
Wireless Networks 20 (8), 2481-2501, 2014
Z.Yan,et al: A survey on trust management for Internet of Things.
Journal of network and computer applications 42, 120-134,2014
Experimental design: no comment
Validity of the findings: no comment
Additional comments: Important references are missing and should be added:
Sravani Challa, et al: Design and analysis of authenticated key agreement scheme in cloud-assisted cyber-physical systems. Future Gener. Comput. Syst. 108: 1267-1286 (2020)
Mohammad Wazid, et al:Design and Analysis of Secure Lightweight Remote User Authentication and Key Agreement Scheme in Internet of Drones Deployment. IEEE Internet of Things Journal 6(2): 3572-3584 (2019).
S Jangirala, et al:Designing secure lightweight blockchain-enabled RFID-based authentication protocol for supply chains in 5G mobile edge computing environment.IEEE Transactions on Industrial Informatics, DOI: 10.1109/TII.2019.2942389,2020
MB Mollah, et al:Secure data sharing and searching at the edge of cloud-assisted internet of things.IEEE Cloud Computing 4 (1), 34-42,2017.
Harsha Vasudev, et al:Secure message propagation protocols for IoVs communication components. Comput. Electr. Eng. 82: 106555 (2020)
Z.Yan,et al:A security and trust framework for virtualized networks and software‐defined networking.Security and communication networks 9 (16), 3059-3069,2016
Q.Jing,et al:Security of the Internet of Things: perspectives and challenges.
Wireless Networks 20 (8), 2481-2501, 2014
Z.Yan,et al: A survey on trust management for Internet of Things.
Journal of network and computer applications 42, 120-134,2014


Instruction: You are one of the reviewers, your task is to write a review for the article. You will be given the title of the article, the number of the round in which the article is located, and your order among the reviewers.

Input:
Title: STATE-OF-THE-ART IOV TRUST MANAGEMENT A META-SYNTHESIS SYSTEMATIC LITERATURE REVIEW (SLR)
Review round: 2
Reviewer: 2

Output:
Basic reporting: No comments
Experimental design: No comments
Validity of the findings: No comments
Additional comments: The authors have addressed most of my concerns and it improved the quality of the paper. Hence, I recommend accepting the paper for publishing in this journal.


Instruction: You are one of the reviewers, your task is to write a review for the article. You will be given the title of the article, the number of the round in which the article is located, and your order among the reviewers.

Input:
Title: ERROR CURVES FOR EVALUATING THE QUALITY OF FEATURE RANKINGS
Review round: 1
Reviewer: 1

Output:
Basic reporting: No comment
Experimental design: No comment
Validity of the findings: No comment
Additional comments: The paper is scientifically sound and well written, with an exhaustive set of experiments on a suite of different synthetic and real datasets. I have only two points I would like to see better clarified
- there is a non-negligible overlap with Slavkov, 2018; although this is acknowledged by the authors,  I think they should stress more the novelty of the present contribution w.r.t. their preliminary work;
- the number of alternatives shown (e.g. the different predictive models for building the FFA/RFA curves) provides a broad and solid landscape, but they are not helping the researcher wanting to apply the proposed method (e.g. what should I do when different models provide very different rankings? Which model should I trust). I think that the paper would greatly benefit by adding a working recipe schema for ta practitioner wanting to practically use the method in an effective way.

Finally, the main text is quite dense and rich in figures - I would rather move more material in the Appendix/SuppMat to better highlight the key messages in the main text avoiding them to sink in a sea of results which may prevent a non-specialistic reader to fully grasp the overall meaning.


Instruction: You are one of the reviewers, your task is to write a review for the article. You will be given the title of the article, the number of the round in which the article is located, and your order among the reviewers.

Input:
Title: ERROR CURVES FOR EVALUATING THE QUALITY OF FEATURE RANKINGS
Review round: 1
Reviewer: 2

Output:
Basic reporting: The authors address an interesting problem within the context of feature ranking. They present a method for evaluating feature ranking algorithms.  The method is based on computing the correctness of the feature ranking based on the accuracy achieved with it using different learning algorithms. Based in two different chains and a simple idea it seems to provide meaningful information about the rankings.

The state-of-the-art work is very complete, although I would suggest to include more recent references about approaches to measure feature stability.

The contribution is stated clearly.
Experimental design: Extensive experimental work is carried out incluging synthetic  and real world dataset.

However, the choice of the best learning methods for building the curves ( SVMs and K-NN) is not well justified. I encourage the authors to clarify this point.

Could this depend on the type of dataset: number of instantes, dimensionality, noise ?
Validity of the findings: Specify the limitations and drawbacks of the proposed method.

Specify if this methods could apply to regression problems.
Additional comments: No additional comments


Instruction: You are one of the reviewers, your task is to write a review for the article. You will be given the title of the article, the number of the round in which the article is located, and your order among the reviewers.

Input:
Title: ERROR CURVES FOR EVALUATING THE QUALITY OF FEATURE RANKINGS
Review round: 2
Reviewer: 1

Output:
Basic reporting: no comment
Experimental design: no comment
Validity of the findings: no comment
Additional comments: All the raised issues have been reasonably solved.


Instruction: You are one of the reviewers, your task is to write a review for the article. You will be given the title of the article, the number of the round in which the article is located, and your order among the reviewers.

Input:
Title: ERROR CURVES FOR EVALUATING THE QUALITY OF FEATURE RANKINGS
Review round: 2
Reviewer: 2

Output:
Basic reporting: The authors have addressed all the changes that were requested. I recommend the acceptation of the paper.
Experimental design: ok
Validity of the findings: ok
Additional comments: No additional comments


Instruction: You are one of the reviewers, your task is to write a review for the article. You will be given the title of the article, the number of the round in which the article is located, and your order among the reviewers.

Input:
Title: ITERATIVE IMPROVEMENT IN THE AUTOMATIC MODULAR DESIGN OF ROBOT SWARMS
Review round: 1
Reviewer: 1

Output:
Basic reporting: The introduction provides useful background on the automatic design of collective behaviours for robot swarms, and sets the research in the context of previous work on AutoMoDe, clearly stating the aims of the article.

However, very few papers written by people outside of this research group are surveyed, which seems a little biased. Admittedly there is not much literature that is directly relevant to this work, but Jones et al. (2016) should at least be discussed in detail.

In some places there are long lists of citations such as:

"A general methodology for the manual design of robot swarms is still missing and existing approaches either operate under restrictive assumptions (Hamann and Worn, 2008; Kazadi, 2009; Berman et al., 2011; Brambilla et al., 2014; Reina et al., 2015; Lopes et al., 2016) or are labor intensive, time consuming, error prone, and nonreproducible (Brambilla et al., 2013; Francesca and Birattari, 2016; Bozhinoski and Birattari, 2018)"

"Neuro-evolutionary swarm robotics has been successfully applied to several missions (Quinn et al., 2003; Christensen and Dorigo, 2006; Hauert et al., 2009; Trianni and Nolfi, 2009), yet a number of limitations remain"

Please briefly explain the specific relevance of each of these citations, rather than listing them without further comment.

The rest of the article is well-written, and presents the research in a logical progression. The description of iterative improvement and the specific variant of stochastic hill-climbing used is particularly good. I'm not convinced that the proofs of completeness for the perturbation operators add much to the paper though - perhaps illustrating them with worked examples would improve their contribution.

The supplementary material is a useful addition, but videos of the generated behaviours would help significantly to appreciate the qualitative differences in the solutions produced by each algorithm.

The caption for Figure 1 should be more descriptive - please state here what each symbol represents (behaviours, conditions, selector, and sequence nodes).

The caption for Figure 2 should also describe the images in more detail - e.g. that these are ARGoS simulations with 20 e-puck robots, with light sources and coloured floor patches.

Figure 3 seems to have a formatting issue with the chart labels, which makes it look like BTFSM is one label, rather than BT FSM next to each other for Minimal/Random/Hybrid. This is fixed in Figure 4 and 5, so please update Figure 3 to match these.

Figure 3/4/5 - specifying "120sec" in each panel seems redundant, as this is already explained on line 477. Either remove this or factor it out to the figure captions.

The definition of the perturbation operators is a little inconsistent - some refer to a state 's' or a transition 't', while others are less formalised. Also, it is not clear how certain transitions/states are selected by the search algorithm. For example "Remove a transition from the finite-state machine: Remove a transition t. Do not remove the only outgoing or incoming transition of a state." - how is the transition 't' chosen? Sometimes the word "arbitrary" is used when referring to them - does this mean they are chosen at random (uniform distribution)?

The article suffers from not being sufficiently self-contained, and would therefore be difficult to understand for someone not familiar with previous AutoMoDe publications. The set of behaviours and conditions used to construct the controllers is not defined (behaviours like anti-phototaxis are only mentioned in the Discussion section), the details of the pseudo-reality simulation are vague, and information required for reproducibility such as the dimensions of the arena and floor patches is missing. While I appreciate that these details have been published in previous papers, please reiterate the key details here.

There are a few grammatical errors and typos throughout - please proofread carefully before final submission.
Experimental design: The experimental design is quite strong - the swarm is sufficiently large, standard tools (e-puck robots and ARGoS3 simulator) are used, appropriate case study swarm tasks were chosen, and the performance of iterative improvement is compared against that of AutoMoDe-Chocolate/Maple, GP, and EvoStick.

The data gathering is rigorous - experiments are carried out in both simulation and "pseudo-reality" in lieu of real hardware. However, there is no statistical analysis of the data obtained. The methods are not quite described in sufficient detail to be replicated, but the source code is available online in a GitHub repository.

Although the installation instructions were comprehensive, there was no clear documentation on how to reproduce the experiments presented in the paper. There is some documentation in each separate code repository, but instructions/scripts for reproducing the data presented in this article would be appreciated.
Validity of the findings: The discussion of the results tends to focus on the difference in performance between the FSM and BT variants of each algorithm, and their ability to cross the pseudo-reality gap, rather than the differences in performance between each algorithm.

Although the BT variants seem to produce lower-scoring behaviours, the spread of the data is much smaller and the results are therefore more consistent. Conversely, the FSM variants tend to produce a wide spread of data, suggesting that they do indeed get stuck in local optima. FSMs also seem to suffer worse than BTs when attempting to cross the pseudo-reality gap - why is this?

The article would benefit from deeper analysis of the experimental results, that provides valuable insight into the data. There is some relatively superficial analysis and conjecture, but not much in the way of concrete detail that explains the observed results.

It would be helpful to see examples of the generated FSMs and BTs, with an explanation of how these relate to the scenarios, and why they achieve good/poor performance (particularly in comparison to the other methods such as AutoMoDe-Chocolate/Maple).

There are a few conclusions that are not convincingly supported by empirical evidence:

"We found no indication that the search space is formed in such a way that it would hinder a local search algorithm to effectively find an optimum"

"We did not conduct a landscape analysis but the results of local search are close to methods known to avoid local optima. This corroborates our hypothesis, that the search space is regular and well-behaved, as we could not find any evidence that iterative improvement remains trapped in local optima."

"The design methods that operate on behavior trees, do not improve their solution quality significantly, when the budget is increased. This indicates that all considered design methods for the restricted behavior tree architecture find solutions close to the actual optimum for that restricted architecture."

Without knowing the global optima for each scenario, not being able to improve upon the results with larger simulation budgets is not sufficient evidence to suggest that the algorithms are not getting stuck in local optima. Particularly as there is no comparison to a global search algorithm, or something like a genetic algorithm that includes elements of global search (crossover) as well as local search (mutation), it seems difficult to argue that your local search algorithms are not all getting stuck in local optima.

Can you quantify the size of the search space, and comment on how thoroughly this is searched by the algorithms analysed?
Additional comments: This research is an interesting incremental extension to the existing body of AutoMoDe research, and deeper analysis of the experimental results could provide some valuable insights that would inform future work.


Instruction: You are one of the reviewers, your task is to write a review for the article. You will be given the title of the article, the number of the round in which the article is located, and your order among the reviewers.

Input:
Title: ITERATIVE IMPROVEMENT IN THE AUTOMATIC MODULAR DESIGN OF ROBOT SWARMS
Review round: 1
Reviewer: 2

Output:
Basic reporting: none
Experimental design: none
Validity of the findings: none
Additional comments: This paper describes a comparative study for the iterative improvement optimisation technique for automatic design method of two control architectures namely Finite-State Machines FSM and Behavior Trees BT for swarm robotics systems. The study evaluates the performance of a set of optimisation algorithms (i.e., Chocolate, Maple, Minimal-FSM, Minimal-BT,
 Random-FSM, Random-BT, Hybrid-FSM, Hybrid-BT, GP and EvoStick) for the two control architectures (i.e., FSM, BT).  The study describes the properties of FSM and BT which must have in order to considered a valid instance of control software. Additionally, the study explains a set of possible perturbation operators performed on FSM and BT in order to transform them to another valid finite-state machine and all the optimisation algorithms are introduced briefly.

Generally, this paper is properly organised and well written. A set of extensive experiments is described to provide sufficient conclusive evidence on the outcomes of this research. Supplementary material provides the necessary resources which are sufficient to replicates the work described in the paper. In my opinion, the results comprise useful message to the research community in the field of automatic design methods to generate collective behaviour of robot swarm.

However, I have few comments to be addressed in the followings:

1- It has been mentioned in table1 (p.8), the period of control cycle is 100 ms. However, in mission section (p. 13- L. 477)  you referenced table1 and you mention the experiment runtime is 120 s. What do you mean by the former number? And why the two numbers are different?

2-  I do not understand the use of 2 black spots in the SCA task. Despite the explanation in the discussion section which indicates the spots are not a very useful cue to find the shelter. It is also not clear how the robot develops a strategy to search for the shelter and hide inside it. Please, elaborate on this.

3- It is not described in the foraging task what is the mechanism used to evaluate the performance of the foraging mission. In particular, how to measure that the robot visit (moved over) foraging site (one of the black spots) then visit the nest site (moved white area) subsequently. Please, explain this.

4- It would be good if you include video of successful trials in the supplementary material. This could help for visual inspection of the most effective developed strategies.

5. In fig 3, 4, and 5. It would be good if the y-axis (score) is normalised between [0,1]. I understand the difference in the score range of values is due to the difference in the objective function of each mission. However, the current y-axis is confusing readers when comparing the graphs in each figure. The overall comparison of the missions in one figure indicates that one mission outperformed others when a unified score metic is assumed between the graphs.

6. I believe the inclusion of Evostick in the study is irreverent in the context of the paper since it uses ANN. The main objective of the study is to compare the iterative improvement optimisation for FSM and BT therefore the presence of ANN in one instance (in Evostick) is consistent with the theme of the paper.


Instruction: You are one of the reviewers, your task is to write a review for the article. You will be given the title of the article, the number of the round in which the article is located, and your order among the reviewers.

Input:
Title: ITERATIVE IMPROVEMENT IN THE AUTOMATIC MODULAR DESIGN OF ROBOT SWARMS
Review round: 2
Reviewer: 1

Output:
Basic reporting: Previous concerns have been addressed.
Experimental design: Previous concerns have been addressed.
Validity of the findings: Previous concerns have been addressed.
Additional comments: Thank you for taking the time to address my concerns - the revisions made to the paper and the supplementary material are a significant improvement.

I still think it would be helpful to see examples of the generated FSMs and BTs, even if only in the supplementary material. In your rebuttal later you said that "we felt it inappropriate to directly include figures of all discussed instances of control software. Instead we made them available on the supplementary materials page", however the closest I can find at the linked URL is text-based encodings of the controllers (within data.zip). Could a visualisation of the controller be included with each controller.txt, or could you provide a script that allows the reader to generate a visualisation from these text encodings?

My only other concern is that the newly-added Table 2 that lists "noise models" doesn't provide context for the values. Presumably these are the noise parameters used for the ARGoS sensors/actuators, but I don't think these currently have consistent semantics (see this open issue raised on the ARGoS GitHub repository: https://github.com/ilpincy/argos3/issues/120). Perhaps the DEMIURGE argos3-epuck plug-in resolves this issue for the e-puck-specific sensors/actuators, but it would help to state whether uniform or Gaussian noise distributions are used, and what the values in the table represent (e.g. standard deviations).


Instruction: You are one of the reviewers, your task is to write a review for the article. You will be given the title of the article, the number of the round in which the article is located, and your order among the reviewers.

Input:
Title: APPLICATION OF DEEP AUTOENCODER AS AN ONE-CLASS CLASSIFIER FOR UNSUPERVISED NETWORK INTRUSION DETECTION: A COMPARATIVE EVALUATION
Review round: 1
Reviewer: 1

Output:
Basic reporting: Interesting survey and comparison of autoencoder architectures used for one-class classification and their application to two well known IDS datasets. Some comments:
- The graphics are very clear and well done
- I understand that you opted for the deterministic version of autoencoders since variational autoencoders are not included, that is ok, but at least should be mentioned and included in the reference works. For example works such as: Variational data generative model for intrusion detection; Conditional variational autoencoder for prediction and feature recovery applied to intrusion detection in iot
- The quantitative analysis is well done and complete
Experimental design: The experimental design is relevant and complete providing convincing results
Validity of the findings: The study presents interesting results which are not themselves novel but the rigorous presentation and interesting survey can be useful to other researchers interested in the field of application of machine learning to IDS.
Additional comments: As mentioned earlier, it is an interesting work useful to the IDS research community. The only missing point is the inclusion of some additional reference to works in the field as pointed above


Instruction: You are one of the reviewers, your task is to write a review for the article. You will be given the title of the article, the number of the round in which the article is located, and your order among the reviewers.

Input:
Title: APPLICATION OF DEEP AUTOENCODER AS AN ONE-CLASS CLASSIFIER FOR UNSUPERVISED NETWORK INTRUSION DETECTION: A COMPARATIVE EVALUATION
Review round: 1
Reviewer: 2

Output:
Basic reporting: The article use clear, unambiguous, technically correct text. I don't feel qualified to judge about the English language and style. The list of references is appropriate.
Citations are not in acceptable format of ‘standard sections’. PeerJ uses the 'Name. Year' style with an alphabetized reference list, example (Smith et al., 2005).
Figures 10, 11 quality should be improved.
The paper ‘self-contained,’ represent an appropriate ‘unit of publication’, and include all results relevant to the hypothesis. The paper has no formal results.
Experimental design: The topic of the paper is appropriate for the journal PeerJ Computer Science.
Research question well defined, relevant & meaningful.
Methods described with sufficient information to be reproducible by another investigator.
Validity of the findings: It is clear from the paper what has been done and why. The scientific quality of the paper is good. The conclusions connected to the original question investigated, and are limited to those supported by the results.
Additional comments: Figure 2 what is “Avg”? If it means Average, explain why.
In equation 2 “N” is not defined.
In algorithm 1 Phase II: Testing, step 2 – “ifError” insert space after and clear Italic font – should be “if Error”, “else” must be bold.
In algorithm 1 Adam stochastic gradient optimization method is used. Authors should motivate why they use Adam method.
In case of Sparse autoencoder (SSAE) the sparsity penalty term is optimized minimizing the Kullback–Leibler (KL) divergence, but in algorithm 1 only Adam method mentioned.
In line 111 remove point, text in line 111“.Rifai et al. (2011)” should be “Rifai et al. (2011)”
For denoising autoencoder (DAE), Contractive Autoencoder (ContAE) and Convolutional Autoencoder optimization method used should be mentioned.
Tables 1 and 2 in columns “Value”, “Input size” and “Output size” data units should be shown, what is (41,1) percent’s, bytes or others?
 Tables 1 and 2 described in section 3.5 but mentioned in section 4.1. The same is with tables 3 and 4, described in section 4.1 but mentioned in section 4.2.
The authors presents a comparative evaluation as mentioned in abstract “This study ﬁlls this gap of knowledge presenting a comparative evaluation of diﬀerent AE variants for one-class unsupervised intrusion detection.” In addition, Experimental Framework proposed (see fig. 8). In such a case, it did not clear explained while and in which context sections “4.3.2 Model Training”, “4.3.3 Model Evaluation” of the paper became models? In the paper model did not proposed.


Instruction: You are one of the reviewers, your task is to write a review for the article. You will be given the title of the article, the number of the round in which the article is located, and your order among the reviewers.

Input:
Title: A MACHINE LEARNING FRAMEWORK FOR THE PREDICTION OF CHROMATIN FOLDING IN DROSOPHILA USING EPIGENETIC FEATURES
Review round: 1
Reviewer: 1

Output:
Basic reporting: This is an interesting paper that applies different flavours of machine learning to predict TADs from ChIP-seq data. The Introduction and literature review is well-written. The discussion is well written as well. The Methods section and some figures can be a bit improved. I believe the manuscript needs just a minor revision to address the points below.

Major points:
The methods section needs to be clarified in terms of the experimental data processing. In particular,
1) “To normalize the input data, we mean-centered and scaled each feature to the unit variance” – not clear what exactly was done, please expand.
2) Not clear what is shown on Supplemental Figure 2, even the X and Y axes are not labelled.
3) Please describe in detail the procedure of data normalisation (with equations and/or software tools references when needed).

Minor points:
4) Abstract: “Chriz” -> “Distribution of protein Chriz (Chromator)”.
5) Page 3, line 12: separate “software Armatus” and “Filippova et al.” by some punctuation.

6) Figures 1D, 3 -> increase font sizes so that the smallest font size would be the same as the font size in the main manuscript.
7) Supplementary Figure 2 -> add captions to the X and Y axes and expand figure legend.
8) The first reference in the reference list lacks author names.
9) What is meant by the “sequence size” in Figure 3A? By the way, it would be good to introduce letters “A” and “B” for the two panels in Figure 3.
10) Page 4, line 155: “introduce” -> “Introduced”.
11) Is it possible to say how good the prediction is in terms of standard quantities such as specificity or recall, or anything like this that will tell me whether the wMSE value achieved with the model is really good in comparison with typical performance of other models in the field?
Experimental design: See details in section 1. Basic reporting
Validity of the findings: See details in section 1. Basic reporting
Additional comments: This is an interesting paper that applies different flavours of machine learning to predict TADs from ChIP-seq data. The Introduction and literature review is well-written. The discussion is well written as well. The Methods section and some figures can be a bit improved. I believe the manuscript needs just a minor revision to address the points below.

Major points:
The methods section needs to be clarified in terms of the experimental data processing. In particular,
1) “To normalize the input data, we mean-centered and scaled each feature to the unit variance” – not clear what exactly was done, please expand.
2) Not clear what is shown on Supplemental Figure 2, even the X and Y axes are not labelled.
3) Please describe in detail the procedure of data normalisation (with equations and/or software tools references when needed).

Minor points:
4) Abstract: “Chriz” -> “Distribution of protein Chriz (Chromator)”.
5) Page 3, line 12: separate “software Armatus” and “Filippova et al.” by some punctuation.
6) Figures 1D, 3 -> increase font sizes so that the smallest font size would be the same as the font size in the main manuscript.
7) Supplementary Figure 2 -> add captions to the X and Y axes and expand figure legend.
8) The first reference in the reference list lacks author names.
9) What is meant by the “sequence size” in Figure 3A? By the way, it would be good to introduce letters “A” and “B” for the two panels in Figure 3.
10) Page 4, line 155: “introduce” -> “Introduced”.
11) Is it possible to say how good the prediction is in terms of standard quantities such as specificity or recall, or anything like this that will tell me whether the wMSE value achieved with the model is really good in comparison with typical performance of other models in the field?


Instruction: You are one of the reviewers, your task is to write a review for the article. You will be given the title of the article, the number of the round in which the article is located, and your order among the reviewers.

Input:
Title: A MACHINE LEARNING FRAMEWORK FOR THE PREDICTION OF CHROMATIN FOLDING IN DROSOPHILA USING EPIGENETIC FEATURES
Review round: 1
Reviewer: 2

Output:
Basic reporting: No comment.
Experimental design: No comment.
Validity of the findings: No comment.
Additional comments: In the manuscript by Rozenwald et al., a machine learning approach for the prediction of chromatin folding in Drosophila is described. The study is performed by recurrent neural networks.

The manuscript reads well and is very clear. Overall it is a suitable contribution to the journal.

Comments:

1) The Adam optimizer is mentioned on p.5. It will be nice to elaborate a bit and provide a citation.

2) Future work should be added.


Instruction: You are one of the reviewers, your task is to write a review for the article. You will be given the title of the article, the number of the round in which the article is located, and your order among the reviewers.

Input:
Title: ENHANCEMENT OF PROPERTIES IN MIZAR
Review round: 1
Reviewer: 1

Output:
Basic reporting: The language and article format of the manuscript
should be checked to communicate meaning clearly and
use professional English as much as possible.

For example:

  Line 54 - rather than "a bunch of computer programs",
    perhaps "a collection of computer programs"

  Line 94 - "...permissive assumptions are missed" means
    "...permissive assumptions are omitted"?

  Line 119 - in place of "...variables are permitted what
    enables to introduce schemes", "...variables are permitted
    which enables the introduction of schemes" or "...variables
    are permitted enabling the introduction of schemes" might
    be better

  Table 1 - table title should be placed above the table
Experimental design: The research objective (or main questions that
need to be answered) should be clarified in the
abstract and paper, especially from a computer science
standpoint.  The journal does not accept case studies
or case reports.
Validity of the findings: The author makes a good effort to make the
contents of the paper self-contained, but it
is difficult to link the conclusion to the
research question (which should be revised
and clarified in the experimental design) of
the work.
Additional comments: No additional comments


Instruction: You are one of the reviewers, your task is to write a review for the article. You will be given the title of the article, the number of the round in which the article is located, and your order among the reviewers.

Input:
Title: ENHANCEMENT OF PROPERTIES IN MIZAR
Review round: 1
Reviewer: 2

Output:
Basic reporting: The paper is written in a clear and understandable form.
Sections 1,2 and 3 give a rich and complete snapshot of the current state of the verification research revolving around the Mizar proof checker.

I only have some minor remarks about style and form, reported below.

In the abstract, the apposition "proof-assistant" explaining what Mizar is only appears at its second occurrence.
It would be better to qualify Mizar at its first occurrence.

'"Property" in Mizar is a construction that can be used to...': reads strange.
What about 'Properties in Mizar are keywords that can be used to...' or
'A property in Mizar is a construction that can be used to...'?

l67: "ways how" -> "how"
l184: The "Next submodule"
l190: "much reacher"
l424: "properties of used in the theorems functors."
Experimental design: Section 3 provides a detailed background of the role of properties in the Mizar language and of their current limitations.
This allows the author to clearly point out what the paper contributes.
Again, I only have minor remarks about some points where there possibly is some technicalities excess.

The word "functor" has usually a different meaning in mathematics and theoretical computer science, coming from category theory. Perhaps the reader should be warned about this.
There are other instances where specific Mizar jargon is used and where some reference or explanation would benefit the reader. For example, l71: permissive assumption.

line 30: to satisfy the interested reader, it would be better to have a reference which uses identifications or explain what they are.
Validity of the findings: The contributions of the paper are:
1) a generalisation of the existing properties feature of Mizar allowing to apply them to functions/operations of arbitrary arities;
2) correspondingly, an optional, additional "wrt" keyword allowing the user to precisely specify which arguments of the function/operation are affected by the property;
3) a new property (fixpoint-free)

1 and 2 are formally described in section 3, and 3) in section 4.
Wrt 3), some small investigation about its impact on MML is proposed.
I am surprised that only 3 occurrences have been found; I don't think this means the new property is useless.
On the contrary, I suspect many more occurrences can be found, making it useful.
There are many numerical functions satisfying this property. Even not taking into account functions whose output type differs from their arguments' types, which should automatically satisfy that property.

Regarding 1) and 2), section 3.2 is important for researchers working within (as opposed to merely using) theorem proving.

If I may suggest a possible line of further investigation: 1) and 2) seem intimately related to the notion of currying, which I understand is defined in MML. Would that be possible to link the two things together? If yes (and I am not sure, given the first-order nature of Mizar!), how?

The small example at the end of section 3.1 is intriguing: I understand that some properties are not enforceable at definition time because they only apply to a subclass of the defined items. However, is there a workaround in such cases, maybe based on cluster registrations?
Additional comments: No additional comments


Instruction: You are one of the reviewers, your task is to write a review for the article. You will be given the title of the article, the number of the round in which the article is located, and your order among the reviewers.

Input:
Title: ENHANCEMENT OF PROPERTIES IN MIZAR
Review round: 2
Reviewer: 1

Output:
Basic reporting: The author addressed my points in a satisfactory manner.
Therefore, I am happy to confirm the positive overall assessment of the paper expressed in my previous review.
Experimental design: The author addressed my points in a satisfactory manner.
Therefore, I am happy to confirm the positive overall assessment of the paper expressed in my previous review.
Validity of the findings: The author addressed my points in a satisfactory manner. In particular, I am happy with the technical explanation addressing the question about currying.
Therefore, I am happy to confirm the positive overall assessment of the paper expressed in my previous review.
Additional comments: The diff paper version and the colour-coded rebuttal eased my job: I would like to thank the author for the appreciated effort.


Instruction: You are one of the reviewers, your task is to write a review for the article. You will be given the title of the article, the number of the round in which the article is located, and your order among the reviewers.

Input:
Title: BIG DATA CLUSTERING TECHNIQUES BASED ON SPARK: A LITERATURE REVIEW
Review round: 1
Reviewer: 1

Output:
Basic reporting: No comment.
Experimental design: No comment.
Validity of the findings: No comment.
Additional comments: This survey studies the previous literature and the existing applications on big data clustering techniques based on Spark, and the authors outline a commonly used pipeline in clustering methods based on Spark in a systematic way and focus on discussing by giving a comprehensive review and 93 direct researchers to the strong and weak points of the current Spark-based clustering methods. After that, they discuss the different Spark clustering algorithms. Finally, they present the discussion on the clustering big data using Spark and future direction.
 However, the topic is quite old. This paper fails to represent a high-quality survey with in-depth analysis, comparison, and discussion. The specific comments are shown as follows.
1. As a survey paper, I don’t see many tables to make comparisons and no timeline to represent the evolution of big data clustering methods, which makes this paper quiet weak.
2. To review the data clustering based on Spark is a large topic, I recommend the authors to pick a distinguished perspective to study this topic, instead of attempting to review multiple aspects of it.
3.  As we step into the AI era, I also recommend the authors could add some machine learning-based works to make this survey more consistent with the technique changes.
4. It would be better to give new insight into the challenges and future research direction.
5. The figures in this paper are not clear enough. The text in the figures are blurred.


Instruction: You are one of the reviewers, your task is to write a review for the article. You will be given the title of the article, the number of the round in which the article is located, and your order among the reviewers.

Input:
Title: BIG DATA CLUSTERING TECHNIQUES BASED ON SPARK: A LITERATURE REVIEW
Review round: 1
Reviewer: 2

Output:
Basic reporting: Designing and developing distributed and scalable machine learning algorithms, including clustering algorithms, is an active research topic. Apache Spark is one of the distributed general-purpose cluster-computing frameworks popular among both industry and academia. Evaluation of the developments of clustering algorithms based on Spark technology is timely and useful. A few good reviews around clustering algorithms in the big data paradigm is already available. However, a proper evaluation of distributed clustering algorithms primarily focusing on Spark and its characteristics beneficial.
Experimental design: The authors have followed a scientific approach to identify relevant literature. The review has been broadly organized into three main categories based on the proposed taxonomy. The authors have identified a fair number of recent publications related to the scope. However, it would have been easy to understand if different algorithms were discussed under a suitable sub-heading under each branch.
Validity of the findings: The authors have briefly discussed the characteristics of each algorithm they have reviewed. However, it is recommended to include a systematic approach to evaluate each algorithm to show the reviewed algorithms' strengths and weaknesses. Further, you may include each algorithm's ability to tackle key big data challenges - volume, variety and velocity. Additionally, specific criteria can be proposed to validate the algorithms' relative strengths and weaknesses through experiments.

The proposed taxonomy is well known and can be applied for this scope as well. Fig 3 can be improved by mentioning each algorithm's name under each branch, so that the readers can easily capture the overview of the reviewed algorithms immediately.
Additional comments: In general, the flow of the paper is good. However, the readability of the section V may be increased by introducing appropriate sub-headings.


Instruction: You are one of the reviewers, your task is to write a review for the article. You will be given the title of the article, the number of the round in which the article is located, and your order among the reviewers.

Input:
Title: BIG DATA CLUSTERING TECHNIQUES BASED ON SPARK: A LITERATURE REVIEW
Review round: 1
Reviewer: 3

Output:
Basic reporting: 1.	The English Language should be improved to avoid typographical errors. Some examples of typographical errors include lines 69, 143, 181, 232, 257, 325, 340, 447, 470 and 539.

2.	The introduction is detailed and well referenced.

3.	The structure of the paper needs improvement. For instance, the abstract did not provide a summary of what the systematic review is all about. There was too many background information in the abstract. The abstract is expected to give a little background briefly, then the problem, methods, results, and significance.  Besides, I think the background section should come before the literature review section. Moreover, the discussion section should deliberate on the findings/results, but the authors used this section for another literature review or related work.

4.	Although there are existing literature or systematic reviews on big data analytics, the authors dwelt specifically on clustering analysis of big data with Apache Spark, which is commendable.

5.	The subject matter was adequately introduced.
Experimental design: 1.	The paper is a systematic literature review and has applicability in other fields, most especially where they deal with a large scale of data. It is within the scope of the journal.

2.	The background section needs improvement, especially under the challenges of clustering big data. The authors looked at one direction, the V’s. It will be more comprehensive if authors could also look at other challenges like concept drift, scalability, integration, fault-tolerance, consistency, timeliness, load balancing, privacy, and incompleteness, etc.
Validity of the findings: 1.	While the paper has tried to address big data clustering with Apache spark, some things are still missing. The authors claimed to address five questions, but only the first two were answered. One would have expected the authors to explicitly pick the questions one after the other and provided answers to them from the selected 91 papers. Even the first two that were addressed were implied.

2.	The conclusion still needs improvement. The conclusion is not linked with how the five questions raised in the paper were answered. The conclusion slightly mentioned future direction but still needs more elaboration.
Additional comments: 1.	A summary of the final 91 selected papers should be presented in a table. This should reflect the number of papers finally selected from each of the sources (database) used.

2.	Line 393-396, the definitions of divisive and agglomerative hierarchical clustering were interchanged.

3.	Referencing style is not consistent. Some started with Authors’ surname while others with initials.

4.	Line 156-161. A better spark architecture should replace figure 1. The components mentioned, driver, executors, and RDD are not reflected in the figure.

5.	The research question in line 232 is not a complete statement. I think the authors missed “are the”.

6.	In line 257, the correct the spelling, “Elsevier”

7.	All the three figures in the paper were not cited.


Instruction: You are one of the reviewers, your task is to write a review for the article. You will be given the title of the article, the number of the round in which the article is located, and your order among the reviewers.

Input:
Title: BIG DATA CLUSTERING TECHNIQUES BASED ON SPARK: A LITERATURE REVIEW
Review round: 2
Reviewer: 1

Output:
Basic reporting: The paper has been revised according to the comments. However, it still needs to be further improved.
Experimental design: No comment.
Validity of the findings: No comment.
Additional comments: The paper has been revised after a first review process. It has been improved in most parts but some shortcomings are still present.
1. The identifier of chapters is not sequential.
2. The writing style of the manuscript is inconsistent. For example, some subtitles are capitalized, others are not.
3. The terms used in this manuscript is also inconsistent, for example, “k means”, “k-means” , “k-mean”.


Instruction: You are one of the reviewers, your task is to write a review for the article. You will be given the title of the article, the number of the round in which the article is located, and your order among the reviewers.

Input:
Title: BIG DATA CLUSTERING TECHNIQUES BASED ON SPARK: A LITERATURE REVIEW
Review round: 2
Reviewer: 2

Output:
Basic reporting: 1. There is an improvement in the use of language as compared to the previous version of the paper. However, a Langauge editor can still work on the paper for further improvement. For instance, in line 302, there should be space between 'methods' and 'The'. Line 525, correct the spelling of 'volumn'

2. Authors have improved on the structure but there are still some things to look into. Authors should look at my comments under the validity of findings
Experimental design: Survey Findings/Results should come immediately after the Survey Methodology. Titling section V with "Spark Clustering Algorithm" (the section where the authors claim that it provides most of the answers to the survey questions) does not connote to the readers as the answers to the survey questions.
Validity of the findings: All the research findings, that is, answers to the five survey questions should be under a section titled 'Survey Findings' or simply put, 'Results'. Answers to the questions should not be implied as it is presently. Authors should pick them one after the other and address them.

For example, authors may decide to use the following format:

V. Survey Findings
The findings of the survey in line with the survey questions are presented in this section.

Research Question 1: What are the types of Spark-based clustering methods?
Authors will now provide answers from the selected 91 papers and answer the question.

Research Question 2:
.
.
.
Research Question 5

Following this format will make it clear to the audience.
Additional comments: 1. Line 238 - Authors may remove the statement, "These questions ... paper". As  I have suggested earlier, all the answers to the survey questions should appear under one section.

2. Section V is another literature review. Research findings should be relevant answers to the survey questions from the selected 91 papers and not another literature review.

3. The same observation is noted in the discussion section. Authors should focus on the implications of the survey findings/results and not another literature review.


Instruction: You are one of the reviewers, your task is to write a review for the article. You will be given the title of the article, the number of the round in which the article is located, and your order among the reviewers.

Input:
Title: A HYBRID METHOD FOR HEARTBEAT CLASSIFICATION VIA CONVOLUTIONAL NEURAL NETWORKS, MULTILAYER PERCEPTRONS AND FOCAL LOSS
Review round: 1
Reviewer: 1

Output:
Basic reporting: Clear English is used throughout. Literature references are sufficient. Professional article structure, figures and tables are provided.
Experimental design: 1.	Since from fig. 1 the first part of your model is cnn, therefore the question about heart beat signal is important. What is the input to your cnn model?
2.	The training of the system was not defined. Discuss how the model was trained and show advances of your approach.
Validity of the findings: 1.	Please provide samples of morphological features on cnn output, and also show how the signal looks during cnn processing.
Additional comments: 1.	Why the model is called deep neural network? Actually there is not much relation to models sourced in deep learning, and giving such name just because of use of cnn+nn is not good. I suggest to revise the title to sth what shows a hybrid approach of
2.	Please provide samples of morphological features on cnn output, and also show how the signal looks during cnn processing. these two structures.


Instruction: You are one of the reviewers, your task is to write a review for the article. You will be given the title of the article, the number of the round in which the article is located, and your order among the reviewers.

Input:
Title: A HYBRID METHOD FOR HEARTBEAT CLASSIFICATION VIA CONVOLUTIONAL NEURAL NETWORKS, MULTILAYER PERCEPTRONS AND FOCAL LOSS
Review round: 1
Reviewer: 2

Output:
Basic reporting: The document is well structured, the raw data is provided and the results are clear. However, there are several issues that need to be addressed before publication.

One of the most important issue is that authors need to provide greater justification for their study and better background research. In particular, the related work section should include more similar research. There are several previous research studies that use CNN to classify arrhythmia and others that use the focal loss in heartbeat classification to  address the problem of the imbalance classes. Authors should cite these papers and highlight the advantages/differences of their own work with respect to previous work. For example, the paper "Detection and classification of arrhythmias at the cardiologist level in ambulatory electrocardiograms using a deep neural network" which is the state of the art in the field, or other papers such as "Detection of arrhythmias using a deep convolutional neural network with long-lasting ECG signals",  "The classification of the electrocardiogram beats based on a deep convolutional neural network and the focal loss” (https://doi.org/10.1016/j.compbiomed.2020.103866).  In addition, the comparison of the results should be done with more recent work. For instance, the reference "Mondéjar-Guerra V, Novo J, Rouco J, Penedo MG, Ortega MJBSP, and Control. 2019" uses the same dataset and train/test procedure than the present paper.

The statement "Since CNN has better noise immunity" should come back with a citation.

The next important issue is the paper formulas. They must have a consistent notation to improve readers' understanding. The loss of cross entropy has a minus sign before the sum. The same is true for the formulas for FL and FL'. Also the pt and pi in the formulas for FL and FL' are the same thing. When the authors explain the convolution, the letters for the size of the filter, the output of the convolutional layer, etc., must be the same in all formulas. Please rewrite all formulas correctly.

In the sentence "Due to variations between patients in the ECG signal, the RR interval of different patients cannot be directly compared, in this work we subtract the average from the patient's RR interval to eliminate the effect". It is not clear what you mean by the average of the patient's RR interval. Do you use the entire patient's time series to calculate the average? Do you subtract the average of all the RR characteristics?  Please be more specific.

Finally,  the authors state "We adopt a multilayer perceptron (MLP) classifier because it can form an end-to-end classification with CNN, that is, feature extraction and classification can be performed simultaneously."  However, RR features are preextracted, so not all feature extraction can be performed simultaneously with their method. Please clarify this point in the text.

Some minor comments:
1.	The English language should be improved. Mainly, there are some grammar mistakes , for instance:
a.	The output of the convolution layer IS calculateD as follows:
b.	is one of the MOST widely used activation functions
c.	BE a mini-bacth of the entire training set, the output of the batch normalization IS as follows
d.	The output layer neuron is equal to the predicted classes, and in the paper the neuron is 4. Please rephrase the part” and in the paper the neuron is 4”
e.	The cross-entropy loss is THE most widely used in deep neural network classification
f.	BE the ground truth vector

2.	Please also consider:
a.	Rephrase this sentence with a full point “ The proposed method adopts both the morphological characteristics and dynamic information of the heartbeat at the same time. THE morphological characteristics are extracted through a convolutional neural network.”
b.	Althoug it is mentioned later in the text, I recommend to clarify in the introduction that the authors use a 1 dimensional CNN.
c.The time interval between two consecutive R-peaks is normally called the RR interval (Ruangsuwana et al. 2010), which represents the dynamic information of the heartbeat. consider replacing "represents " with a word that fits better
Experimental design: Due to the lack of a good related work, it is not clear how the present research fills an identified knoledge gap.

The methods are described with sufficient detail and information can be replicate.
Validity of the findings: Conclusions should highligth the advantages/disadvantages of the present method respect to similar works.
Additional comments: No additional comments


Instruction: You are one of the reviewers, your task is to write a review for the article. You will be given the title of the article, the number of the round in which the article is located, and your order among the reviewers.

Input:
Title: A HYBRID METHOD FOR HEARTBEAT CLASSIFICATION VIA CONVOLUTIONAL NEURAL NETWORKS, MULTILAYER PERCEPTRONS AND FOCAL LOSS
Review round: 2
Reviewer: 1

Output:
Basic reporting: the paper is revised
Experimental design: the paper is revised
Validity of the findings: the paper is revised
Additional comments: the paper is revised


Instruction: You are one of the reviewers, your task is to write a review for the article. You will be given the title of the article, the number of the round in which the article is located, and your order among the reviewers.

Input:
Title: A HYBRID METHOD FOR HEARTBEAT CLASSIFICATION VIA CONVOLUTIONAL NEURAL NETWORKS, MULTILAYER PERCEPTRONS AND FOCAL LOSS
Review round: 2
Reviewer: 2

Output:
Basic reporting: Minor comments:

1) "Although the performance of heartbeat classification is improved, these works mainly focus on using CNN to extract the heartbeat morphological features. While arrhythmia not only changes the morphology of the heartbeat but also causes abnormal RR intervals". Please re-elaborate. The main focus of the classification is to improve the classification no matter how. You can argue that CNN would be improved/simplified by integrating the RR

2)"Two median filters are combined to REMOVE THE BASELINE WANDERING of the ECG signal in the paper. First, the QRS complexes and P-waves are removed using a 200-ms width median filter, and then a 600-ms width".

3) "the hearbeat CLASSES ARE imbalanced" not the hearbeats.

4) Although the authors have improved the formulas, the CNN part is still a bit confusing. The convolution formula (line 187) should say that m is the size of the filter. This is mentioned but later in the text.

5) "Let t_i and p_i BE the ground truth and the estimated probability for each category i in C, the cross-entropy loss is computed by". What is C?

6) "In particular, the PPV of SVEB is 68.34%, indicating that the proposed method has a strong capability to recognize SVEB" No. It is better than the other methods listed in the table, but a PPV of about 70% cannot be qualified as a strong capability.

7) Grammar is better, but there is still room for improvement. The authors should carefully review the manuscript. For example:

- "Let t_i and p_i BE the ground truth and the estimated probability for each category i in C, the cross-entropy loss is computed by".
- Then the morphological features are combined with the RR intervals features AND input into the MLP to perform heartbeat classification.
- The kernel of the convolution is reduced as the network structure becomes deeper.
- I do not understand this sentence "The results related to PPV and SE are close to or surpass those obtained with existing works expect for F. Accuracy and Marco-F1 reached 92.53% and 66.09%, respectively. "
- For clarity consider including a full point in this sentence.  "Fig. 1 shows the overall structure of the proposed method. The proposed method includes three steps: ECG denoising, feature extraction, and classification. THE the feature extraction step contains RR intervals features extraction and morphological features extraction via CNN architecture".
Experimental design: The methods are described with sufficient detail and information to replicated.
Validity of the findings: Authors should provide some background in the text on the results obtained with their method for class F, which are much lower than those obtained with the other methods. Any idea why it works worse? The authors claim that they used focal loss rather than loss of cross entropy to better address unbalanced class distribution.
Additional comments: No additional comments


Instruction: You are one of the reviewers, your task is to write a review for the article. You will be given the title of the article, the number of the round in which the article is located, and your order among the reviewers.

Input:
Title: DEVELOPMENT OF A CODED SUITE OF MODELS TO EXPLORE RELEVANT PROBLEMS IN LOGISTICS
Review round: 1
Reviewer: 1

Output:
Basic reporting: no comment
Experimental design: no comment
Validity of the findings: no comment
Additional comments: The author was coding the key sub-routines for solving Logistic Problems. That is nice. But for a research paper that wants to be published, exploiting existing, unimproved algorithms would be judged as less innovation. Further, the analysis of different strategies is also weak. For example, the author depicted both the Continuous Review Model and the Periodic Review Model for inventory management. Still, one cannot determine which model is more suitable for a specific instance in terms of the author’s description.

A large practical logistics management system involves the integration of many software components. That is a very complex problem belonging to the research category of software engineering. Still, I did not see the discussion to handle such issues.

In short, I do not think the article is appropriate for publication as a research paper unless the author can show adequate innovation. But the codes in the paper are good material for learning. If the author can share these codes (such as creating a project in GitHub), that would be nice not only for learners but also for broadening the exposure of the author’s work.


Instruction: You are one of the reviewers, your task is to write a review for the article. You will be given the title of the article, the number of the round in which the article is located, and your order among the reviewers.

Input:
Title: DEVELOPMENT OF A CODED SUITE OF MODELS TO EXPLORE RELEVANT PROBLEMS IN LOGISTICS
Review round: 1
Reviewer: 2

Output:
Basic reporting: The author is commended on assembling a series of problems and their solutions into a single framework. The technical aspects of each part are succinctly explained, however, it is necessary to give a paragraph at the end of each section to explain where the reader can find the most up-to-date related material.

Tables, figures showing code, and graphs must be improved for readability, as they are hard to read in small font-size.

In this reviewer´s opinion, the title should be: A coded suite of models to explore relevant problems in logistics. This would reflect the current orientation of the article and be transparent about its objective.
Experimental design: It is understood that bringing together several problems/solutions into a single framework is an important effort. The illustrative cases are important, however, it would help the users to have a single case involving all parts of the framework. This will help to emphasize what can and cannot be done with it.
Validity of the findings: The article is geared towards people who are very familiar with logistics, mathematical programming, and computer coding. To this public, the proposed framework has the potential to be very useful. I believe there is no methodological novelty to each problem, although the framework is a useful contribution.
Additional comments: This work seems to be potentially very useful for the practitioners in logistics. The author is commended on undertaking this unification effort.


Instruction: You are one of the reviewers, your task is to write a review for the article. You will be given the title of the article, the number of the round in which the article is located, and your order among the reviewers.

Input:
Title: DEVELOPMENT OF A CODED SUITE OF MODELS TO EXPLORE RELEVANT PROBLEMS IN LOGISTICS
Review round: 2
Reviewer: 1

Output:
Basic reporting: no comment
Experimental design: no comment
Validity of the findings: no comment
Additional comments: I accept the explanation for this work’s importance. Moreover, additional analyses and the detail of a single illustrative case brighten up this article. I recommend it for publication on Peer J.


Instruction: You are one of the reviewers, your task is to write a review for the article. You will be given the title of the article, the number of the round in which the article is located, and your order among the reviewers.

Input:
Title: DEVELOPMENT OF A CODED SUITE OF MODELS TO EXPLORE RELEVANT PROBLEMS IN LOGISTICS
Review round: 2
Reviewer: 2

Output:
Basic reporting: The author has refocused the paper as instructed. The paper is now better represented by its title, has scaled down its previous scope, and presents a very useful assembly of code and a blue print of problems the combination of which constitutes a novelty in logistics.
Experimental design: The author made an effort to marshal an experiment that could be analyzed sequentially across the different tools in the suite. This is much better for readability and, certainly for future comparisons as a system (not so much for the particular methods on each phase).
Validity of the findings: The combination of code to attack a wide variety of problems in logistics makes of this work a novel contribution in its whole.
Additional comments: No additional comments