Instruction: You are one of the reviewers, your task is to write a review for the article. You will be given the title of the article, the number of the round in which the article is located, and your order among the reviewers.

Input:
Title: 3D TEXTURE-BASED FACE RECOGNITION SYSTEM USING FINE-TUNED DEEP RESIDUAL NETWORKS
Review round: 1
Reviewer: 2

Output:
Basic reporting: This paper is clearly not ready for publication.

First, there is the writing. The logical flow of the paper is completely flawed and the paper is not coherent and confusing. I am not even able to understand why this is 3D face recognition if all images and the descriptors are 2D. Some examples: the word "Thus" in the abstract does not follow from the previous sentence, The first paragraph (and the others) of the introduction jumps from one topic to another. The related work section jumps from ideas to almost irrelevant minor details and it also reports recognition results without pointing to the exact benchmark. H is not defined in Eq. 2.

Second, it is not clear what is the claimed novelty of the paper since it mostly relies on existing methods.
Experimental design: The paper needs much work on all parts leading to the experiments that I did not read this section.
Validity of the findings: The paper needs much work on all parts leading to the experiments that I did not read the results section.
Additional comments: Without additional work, the manuscript cannot be published. Focus on making clear and coherent arguments. Clearly state what you do and why it is novel.


Instruction: You are one of the reviewers, your task is to write a review for the article. You will be given the title of the article, the number of the round in which the article is located, and your order among the reviewers.

Input:
Title: A SYSTEMATIC REVIEW ON LITERATURE-BASED DISCOVERY WORKFLOW
Review round: 1
Reviewer: 1

Output:
Basic reporting: This review is clearly written, and considers many technical aspects of literature based discovery in a very systematic and easy-to-read manner.
Experimental design: The study design is a major problem. The intent of the authors is to make a systematic review, but the queries that they entered are too specific and woefully inadequate to find all relevant articles. They missed some articles by Swanson himself (e.g., Swanson, 2011) and missed five articles by me that were written after 2007. I did not attempt to find all relevant articles that they missed, but I believe they also missed many articles including by Kostoff, Trevor Cohen, Rindflesch, Spangler, and so on.
Validity of the findings: Descriptively, the text appears to be valid as a snapshot of a certain type of literature based discovery. However, the text does not discuss much about emerging models not based on ABC (Smalheiser, 2012) or related issues such as identifying and resurrecting neglected lines of research. So, although it is nice in organizing a description of many ABC studies, it has limited value in pointing the way towards the future.
Additional comments: No additional comments


Instruction: You are one of the reviewers, your task is to write a review for the article. You will be given the title of the article, the number of the round in which the article is located, and your order among the reviewers.

Input:
Title: A SYSTEMATIC REVIEW ON LITERATURE-BASED DISCOVERY WORKFLOW
Review round: 1
Reviewer: 2

Output:
Basic reporting: The manuscript is well-written and highly readable. Literature-based discovery (LBD) inherently addresses multidiciplinary topics, hence the current review is clearly within the scope of this journal.

The manuscript provides a valuable perspective toward LBD and can certainly benefit researchers in the field. However, there are several aspects of the manuscript that the authors should consider revising before being recommended for a full acceptance. To aid with editorial decision, I have categorised these aspects into "Highly Important" and "Important" categories:

Highly Important

1.1 Notwithstanding the comprehensiveness of this review, there have at least been 3 significant LBD reviews published since 2017: Sebastian et al. and Henry and McInnes in 2017, and Gopalakrishnan et al. in 2019. In view of these competing reviews, the authors is advised to put forward a stronger case to justify the contribution of the current review to the domain. Firstly, the authors should consider providing an additional comparison against the most recent review by Gopalakhrisnan et al. (2019) (https://www.sciencedirect.com/science/article/pii/S1532046419300590). Secondly, more robust discussions on the differentiating feature of the current review against the 3 preceding review papers should be included in order to strengthen the manuscript's contribution. Currently, such discussion is only limited to one paragraph (line 81 - 91) which is not sufficient.

1.2 Readers should expect a review paper to offer valuable expert insights into the LBD field. As such, it is not enough to merely present highly organised facts about LBD methods and systems (which the current manuscript has done very well). The authors are encouraged to provide more insightful commentaries of pertinent research and development gaps of LBD. There is an attempt to address these in the "Conclusion and Future Work" section, but similar commentaries have been found in other recent review papers mentioned above. For instance, the authors may consider dwell a little more of the issues of usability of LBD systems (i.e. HCI) which is not much talked about elsewhere.

1.3 The authors should also emphasise on reviewing the issues surrounding LBD's discovery models. The ABC model should not be taken for granted and ought to be scrutinised. Considering drawing conceptual links with three important LBD commentaries: Davies (1989) https://www.emerald.com/insight/content/doi/10.1108/eb026846/full/html, Smalheiser (2012) https://onlinelibrary.wiley.com/doi/full/10.1002/asi.21599 , and Smalheiser (2017) https://www.ncbi.nlm.nih.gov/pmc/articles/PMC5771422/ (note that all 3 papers are not present in the current review).

Important

1.4 Content in line 717 - 753 is a duplication of the content in line 682 - 716.

1.5 Consider adding graphics (as appropriate) to improve readibility of the manuscript.

1.6 Paraphrase the quote in line 46 - 50.

1.7 Kindly check for and fix grammatical/punctuation errors, such as in line 78 - 80, 144, and 176.

1.8 "heterogeneous bibliographic information" should be correctly stated as "heterogeneous bibliographic information network" (line 74).
Experimental design: The overall study design is sound. The review is organized logically into coherent paragraphs/subsections.

2.1 One of the most obvious strength of the current review is its systematic review approach. To my knowledge, no other LBD review has attempted to adopt such rigorous study design. Having said that, the authors should further elaborate on these procedure to strengthen their contribution.

2.2 Another strength of this manuscript is its additional emphasis on reviewing LBD workflow. Including more reviews on how LBD systems has been used in various real world scientific investigation scenarios is essential.

2.3 Sources are adequately cited, although several recent LBD methods/systems might have been overlooked by the authors, e.g:
https://journals.plos.org/plosone/article?id=10.1371/journal.pone.0215313
https://academic.oup.com/bioinformatics/article/35/9/1553/5124276

Consider including additional articles above in the review.
Validity of the findings: 3.1 On page 3, a little more discussion is needed to justify the selection of the 8 research questions. This is important as the rest of the review hinges on these questions. Also, consider removing "RQ" from the rest of the manuscript to avoid the impression that the manuscript is taken straight from a postgraduate thesis.

3.2 Consider moving the discussions currently located in the "Conclusion and Future Work" section to a dedicated section called "Discussions and Future Work". Keep the conclusion brief.
Additional comments: Providing detailed lists of various aspects in LBD workflow is certainly a strength of the authors' manuscript. They may serve as a useful, easy-to-refer guide for future LBD researchers/developers.

On the other hand, to publish a new LBD review in the span of 3 years during which 3 reviews have been previously published certainly demand justifications to minimise redundant contributions.

Assuming that the authors address the various feedback above, I believe that the current manuscript will have two important contributions to make to LBD: 1) providing the first systematic review of the field, and 2) providing a fresh look at LBD from the workflow and usability perspectives.


Instruction: You are one of the reviewers, your task is to write a review for the article. You will be given the title of the article, the number of the round in which the article is located, and your order among the reviewers.

Input:
Title: A SYSTEMATIC REVIEW ON LITERATURE-BASED DISCOVERY WORKFLOW
Review round: 2
Reviewer: 1

Output:
Basic reporting: No comment
Experimental design: No comment
Validity of the findings: No comment
Additional comments: The coverage of the review is much improved.


Instruction: You are one of the reviewers, your task is to write a review for the article. You will be given the title of the article, the number of the round in which the article is located, and your order among the reviewers.

Input:
Title: A SYSTEMATIC REVIEW ON LITERATURE-BASED DISCOVERY WORKFLOW
Review round: 2
Reviewer: 2

Output:
Basic reporting: No comment. The authors have comprehensively addressed the previously suggested improvements.
Experimental design: No comment. The authors have comprehensively addressed the previously suggested improvements.
Validity of the findings: No comment. The authors have comprehensively addressed the previously suggested improvements.
Additional comments: You have rigorously addressed my previous suggestions for improvement. I believe the comprehensiveness and the fresh perspectives offered by this paper represent a significant contribution to the expanding field of literature-based discovery.


Instruction: You are one of the reviewers, your task is to write a review for the article. You will be given the title of the article, the number of the round in which the article is located, and your order among the reviewers.

Input:
Title: A SPARSE-MODELING BASED APPROACH FOR CLASS SPECIFIC FEATURE SELECTION
Review round: 1
Reviewer: 1

Output:
Basic reporting: The subsection: "A Sparse-Modeling Based Approach for Class-Specific Feature Selection" is actually too descriptive, while instead, a more algorithmic presentation would better suit to understand the proposed approach. I would suggest that the Authors still keep the current explanations, but that they also add the actual algorithm for SMBA-CSFS in pseudocode, naturally, with explanation of the symbols used in the algorithm. This should certainly be corrected.

Supplementary Table 2 caption: "The different colours represent the rank positions obtained: red first position, blue second position and green third position" should be removed from the caption.

Language issues and typos are present occasionally in the paper. It should be proof-read by the Authors.
For example:
45: "For these reasons, it is unsuitable using microarray data as they are" -> "For these reasons, it is unsuitable to use microarray data as they are"
82: "An active developing field of statistical learning is around the notion" -> "An active developing field of statistical learning is focused around the notion"
155: "4,026genes." -> "4,026 genes."
204: "Sparse-Modeling Based Approach is nothing else that our SMBA-CSFS model but that only take into" -> "Sparse-Modeling Based Approach is nothing else than our SMBA-CSFS model, but that only takes into"
Experimental design: 99: "Once the class sample set of the training set has been split apart, it may be possible that each class-subset results unbalanced. Therefore, the SMOTE (Chawla et al., 2002) re-sampling method is applied to balance each class-subset."

The Authors do not describe how the splitting is performed. Since, if you split the dataset into subsets that belong to each class, then each class-subset contains only the samples of that class. There are no samples of the other classes. Why is then SMOTE needed, this makes no sense, or am I missing something?

118: "O will belong to the class that received more votes among the tied classes."

This is unclear. If the classes are tied, then they have the same number of votes. Then how can a class have more votes than others?

137: "The number of genes that are used in the multiclass classification task"

There are two classes, i.e. this is a binary classification problem, not a multiclass one.
Validity of the findings: 227: "SMBA-CSFS is able to better discriminate among the classes of the LUNG C,LUNG D, CAR, DLBLC data sets in both cases, when top 20 and 80 features are considered."

Actually, for DLBCL, the claim can only be made for the first case, because in the second case, at least two other methods appear to be slightly better for 20 features. This is a minor issue.
Additional comments: The paper is mostly well-written, interesting and detailed in the performed analysis. The methods used are sound. I would recommend its acceptance after the corrections are made according to the provided comments.


Instruction: You are one of the reviewers, your task is to write a review for the article. You will be given the title of the article, the number of the round in which the article is located, and your order among the reviewers.

Input:
Title: A SPARSE-MODELING BASED APPROACH FOR CLASS SPECIFIC FEATURE SELECTION
Review round: 1
Reviewer: 2

Output:
Basic reporting: Although the paper is well written but organization is not good. English is OK. Sufficient references are included.
Experimental design: The motivation of the work is not mentioned clearly. The author should give a brief problem definition.

Although you have compared the method with existing methods such as mRMR, relief-F you can compare it with PSO-based, ACO-based feature selection methods also.
Validity of the findings: The experimental results look satisfactory but more experiments for statistical validation is required.

T-test is needed to compare the proposed method with other competing methods.
Test the method with some other high dimensional datasets.
The authors report accuracy only but other validation measures such as precision, recall and ROC curve should be given.
Additional comments: No additional comments


Instruction: You are one of the reviewers, your task is to write a review for the article. You will be given the title of the article, the number of the round in which the article is located, and your order among the reviewers.

Input:
Title: A SPARSE-MODELING BASED APPROACH FOR CLASS SPECIFIC FEATURE SELECTION
Review round: 2
Reviewer: 1

Output:
Basic reporting: Only minor English language errors are still present, see below. Otherwise, the manuscript is well-written and structured.

Lines 14-15: "... and might speed the experimental validation up." -> "... and possible speed-up of the experimental validation"

In SMBA algorithm pseudocode:
- in line "Output", there should be "set of features", not "set of feature"
- there is a closing parenthesis missing in line 5

In "Algorithm 1":
- y, m, C and K should be in italic
- in line 10, there should be "subset of features selected", not "subset of feture selected"

Line 132: "O" should be in italic

Line 139: "the most m representative" -> "the m most representative"

Line 183: "190 samples in 14 classes, breast..." -> "190 samples in 14 classes: breast..."

Line 255: "f-measure" -> "F-measure"
Experimental design: No comment.
Validity of the findings: No comment.
Additional comments: The authors have answered satisfactory to all my queries.


Instruction: You are one of the reviewers, your task is to write a review for the article. You will be given the title of the article, the number of the round in which the article is located, and your order among the reviewers.

Input:
Title: AVOIDING “CONFLICTS OF INTEREST”: A COMPUTATIONAL APPROACH TO SCHEDULING PARALLEL CONFERENCE TRACKS AND ITS HUMAN EVALUATION
Review round: 1
Reviewer: 1

Output:
Basic reporting: See PDF.
Experimental design: See PDF.
Validity of the findings: See PDF.
Additional comments: See PDF.


Instruction: You are one of the reviewers, your task is to write a review for the article. You will be given the title of the article, the number of the round in which the article is located, and your order among the reviewers.

Input:
Title: AVOIDING “CONFLICTS OF INTEREST”: A COMPUTATIONAL APPROACH TO SCHEDULING PARALLEL CONFERENCE TRACKS AND ITS HUMAN EVALUATION
Review round: 1
Reviewer: 2

Output:
Basic reporting: Writing is clear and unambiguous, with professional word choice throughout.
Article structure and general presentation is high-quality. Spacing of equations and tables could be improved (there is too much space), but this seems like a pretty minor quibble.
Could be useful to mention time-frame of talk scheduling; shorter turnaround time helps motivate the need for an automated method.
Citations are improperly embedded in sentences in some places (e.g. line 185, 190, etc.)
Data is made available online.
Related work is a little sparse, but it could be because (as the authors point out) there may not be much related work on conference scheduling.
Topic modelling is typically introduced with the Blei 2003 citation.
Text should stay within boundaries of image in Figure 3, “g” in “algorithm” is cut off in Figure 3.
Experimental design: Contributions are:
Use of topic modeling to measure talk similarity combined with stochastic optimization of a global objective that targets both intra-session similarity and inter-session dissimilarity
Lack of a need for human intervention
The lack of a need for human intervention seems to be contradicted by the fact that one of their test cases required significant modification before being deployed in the real world
On line 300, it’s said that 0.5% of talk pairs that share a session in the automated schedule are retained in the modified schedule for Evolution 2014
None of the experiments indicate that the automated method is either necessary or sufficient for producing reasonable conference schedules, especially in light of the extreme changes required by the conference organizers. 2 potential suggestions to loosely show necessity and/or sufficiency of the automated method are:
An analysis of the differences between the manual and the automatic schedules. The authors could provide an appendix with some sample time slots from the manual and automatic schedules. Note that in the manual schedule (https://www.evolutionmeetings.org/uploads/4/8/8/0/48804503/2014_program.pdf), sessions are labelled with easily understood subject areas. Can the automated sessions be labelled in this fashion? One or two examples would go a long way. It would also be nice to see what a time slot would look like if all sessions were created at random, or using the greedy initialization method.
Adding another automated method to the user survey to serve as a baseline. It is unclear whether the similarities between automated and manual schedules in terms of the 2 metrics derive from success of the automated method or from wording of the survey questions and/or desire of the annotators to “settle” for reasonable schedules when there exist much better schedules. That is, it is not apparent that a very simple baseline (or a random approach) would not satisfy the 2 criteria assessed in the survey.
Many of their methods were straightforward applications of existing methods, and those were described adequately. Some parts may not have been described enough to enable full replication (e.g. line 192, where they describe their use of “a modification of Kruskal’s algorithm”), but overall it is well done.
A survey is certainly the right kind of evaluation for this problem. This kind of thing really needs human evaluation.
Validity of the findings: Findings reported in the abstract are misleading - For example, the abstract claims that “[the authors] find that simulated annealing improves the objective function over an order of magnitude relative to the schedule manually produced by the program committee,” yet it is not clear here that the objective function is a newly defined objective that does not have proven relevance to conference scheduling yet (objective is motivated by the Evolution 2013 blog post, but is not directly shown to be relevant to the assessed quality of resulting schedules).
Conclusions are well-stated, with some (but not a lot) of discussion of further improvements. The suggestion to incorporate co-authorship and co-citation networks seems like a good one.
It is interesting that scheduling constraints did not significantly impact the discrimination ratio, and that Evolution 2014 organizers suggested focusing on coherence within sessions rather than disparity between sessions.
No discussion of why the HC-A and SA-A strongly outperform the HC-B and SA-B methods, which would be interesting, though not necessary.
Additional comments: I find the scientific results weak: both a straightforward application of previous methodology, and somewhat misleading statements of conclusions from results.

The paper is, however, well-written, and it’s nice that the method was tested on real conferences. The authors describe their approach well enough. One particularly nice touch was the inclusion of images of manual conference organization in Figure 1, which gives the reader a strong understanding of the difficulty of manually scheduling conferences.
There are some minor formatting issues that could be improved, particularly with regard to the amount of whitespace and the formatting of equations (for example, equations should be centered).
I appreciate that the data is made available online.


Instruction: You are one of the reviewers, your task is to write a review for the article. You will be given the title of the article, the number of the round in which the article is located, and your order among the reviewers.

Input:
Title: AVOIDING “CONFLICTS OF INTEREST”: A COMPUTATIONAL APPROACH TO SCHEDULING PARALLEL CONFERENCE TRACKS AND ITS HUMAN EVALUATION
Review round: 1
Reviewer: 3

Output:
Basic reporting: Clear and unambiguous professional English is used throughout.

Although unusual to include photos like those in Figure 1 in technical papers, they perfectly illustrate the arcane current manual practices that are still so widespread.

123. An introductory sentence to set out the structure of the METHODS section would let the reader know what is coming in sections 1.1 to 1.6.

155. Cosine similarity will be between 0 and 1 where the vector elements contain only positive values. Is this is the case here, rather than -1..1?

157. "We introduce an objective function called the Discrimination Ratio...". Is this ratio new and something introduced here? It reminds me of cluster analysis measures and the case for novelty would benefit from some justification as to why the Discrimination Ratio is different to some of the internal measures often used to evaluate cluster quality. Highlighting the difference would substantiate the claim. The novelty is most likely to be in the use case, in which case I'd suggest including a reference to a standard cluster analysis function.

164. A sentence spelling out when the ratio will be high, middle and low will help the reader to remember the intuition.

203. "T_{m}ax" needs explaining or should this be T_{max}?

217. "a Greedy or [an] ILP schedule"

240. "typically have [constraints?] that restrict"

241. "talks competing [in?] awards"

264. "deemed" has negative connotations and "judged" would be a better word to use here

265. You should briefly mention how you get from topics to pairwise similarity (weight vectors?)

296. "following manual procedures that elude easy description", made me laugh out loud, but it is true and underscores how arbitrary assignment is.

300. Given the substantial changes made by the program committee (affecting 99.50% of pairs), some further details about the changes would help the reader to understand the result. For example, it might be interesting to compare the before/after Discrimination Ratios of the two assignments (and the two components of the ratio) and/or topic extraction (or top tfidf terms) of the resultant sessions to see whether there is a pattern to the adjustments. This may go some way to investigating whether the later feedback from the committee about "coherence first and disparity second" is supported/evident here.

303. As it is, this major disagreement between the assignments is left hanging in this very short paragraph and the paper launches straight into the much longer 2.2.2 without really finishing off 2.2.1. This is a shame because 2.2.1 should not be seen as a negative result and that point should be made before going on to almost defend the result in 2.2.2. There is no objectively correct schedule, a point made later in the discussion, but perhaps trail that idea here and be less defensive of the differences; instead focus on what can be asked and learnt from the difference.

Table 2 could be made more readable by reducing the width of the left column and widening the middle.

Table 3 could be made more readable by widening to leftmost column to avoid word wrap and, perhaps, right aligning the value columns.

Tables 1 & 4 are overly wide for their content, making them harder to scan from variable to variable description. Suggest reducing overall width or reducing left column width.

Section 3, DISCUSSION, would benefit from splitting in two so that that concluding comments and future work is presented under its own heading. The discussion could then focus on discussion of the results presented here.
Experimental design: There are a lot of uncontrollable externalities when experimenting on real-world conferences, but the authors have done a good job in designing experiments within these constraints. The experimental design for automated scheduling using 2013 conference data to inform their experiments on the core problem of the 2014 conference is principled and well thought through.

While the expert analysis (2.2.2) to compare automated and manual schedules is well designed, small number statistics dictate that no strong conclusions can be drawn from 29 data points. The work is still interesting and thought provoking, but it should be presented as a preliminary study to suggest further work rather than anything to draw firm conclusions from. It is still important to publish and share such work, but its presentation needs to clearly state that this is only an exploration rather than a full rigorous experiment.

I commend the authors for observing best practice open science and publishing their software and data on Zenodo. A few comments below:

- README.md does not state python version and other dependencies, doesn't include instructions for running the programs or how to reproduce the results in the paper. An orientation of the file tree in the readme would be useful given there are so many folders. Minor point: commentary refers to 2014 in future tense.

- Readability of the python code would benefit from pylint/pep8 formatting (which can be done automatically in many editors, so not a good excuse for not following best practice).

- A comment at the head of each file would make the code easier to understand.

- The inclusion of import statements (rather than just the call to main()) is a little confusing and can upset code assist/linting in some editors; even if the code is invoked via inclusion rather than as main, the imports will only happen once no matter how many times they are invoked.

- Minor tidying up suggestion: remove the Mac ".DS_Store" files scattered throughout the tree.
Validity of the findings: By using post-hoc analysis of 2013 conference data to inform the live scheduling for a much larger 2014 conference timetable. The smaller size of the problem addressed in the 2013 data enabled them to test multiple starting conditions that, by intuition alone, one might have expected to produce a better final allocation, as measured by the Discrimination Ratio. The result that the starting condition was largely unimportant was surprising and is potentially a valuable insight into the dominant constraints in the scheduling problem - irrespective of the subsequent major adjustments made manually by the program committee.

One criticism of this and other work in this area is the lack of longitudinal, historical accumulation of data from past conferences to feed into the current scheduling. It is common to use authors' past publications to profile their suitability for reviewing papers submitted for peer review but I have not yet seen the same historical perspective applied to session scheduling. True, the paper makes a (probably correct based on personal experience) claim that schedule conflicts are a common feedback topic in conference feedback. Future work might perhaps make such feedback one of the inputs to scheduling but before that can happen, the community needs better tools.
Additional comments: Not scientifically important, but readers may be curious as to why results from 2013-14 conferences are being reported now. This is not a criticism but perhaps some narrative could be given to say why now. My own thoughts are that mainstream practice and tools for conference organisation have hardly changed since 2013-14 and there is little re-use of prior work where organisers decide to build conference support tools, so publishing this work and supporting codes now is a valuable and welcome contribution to the field.


Instruction: You are one of the reviewers, your task is to write a review for the article. You will be given the title of the article, the number of the round in which the article is located, and your order among the reviewers.

Input:
Title: AVOIDING “CONFLICTS OF INTEREST”: A COMPUTATIONAL APPROACH TO SCHEDULING PARALLEL CONFERENCE TRACKS AND ITS HUMAN EVALUATION
Review round: 2
Reviewer: 1

Output:
Basic reporting: - Although I still doubt the added value of Figure 1, I respect the authors' decision to keep it in the manuscript.
- Compared to the previous version, the literature section is now more comprehensive.
- Line 122: the IP abbreviation has not been used/explained before.
- Line 130: briefly explain how attendance is measured.
- Line 147: I would replace "the solution aims to" with "the goal is to".
- Line 158: the word 'profiles', i.e. binary participant preference vectors, is not explained and can be confusing to the reader.
- Line 305: I would rephrase “the ability to escape” to e.g. “potential/possibility to escape”.
- I appreciate the inclusion of a "Preliminaries" section. However, the caption of Table 1, as indicated in my original comment, is still false, as the table does not contain a single variable. Rather, the table exclusively contains parameters. This has an impact on line 208 as well. Regarding the SA entries in Table 1: the temperature Z is included in Table 1, yet the constant alpha is not. The Y entry is coupled with the description "number of seed talks for the greedy algorithm". The word algorithm should be changed to heuristic.  Same for the R-entry.
- I would suggest a more precise title for Section 1.7: Discrimination ratio metaheuristics
- The legend for Figure 3 is awkwardly placed. Why not, for sake of consistency, do the same as in Figure 2?
- Line 487: typo 'althogh'.
Experimental design: - I am still of the opinion that the scientific results are too weak/insufficient to warrant publication. A number of my previous suggestions for improving the paper were relegated to 'future research'. In what follows, I provide 3 concrete suggestions which, when properly implemented and included in the paper, can make the paper salvageable. I am sympathetic to the authors and know these changes will take significant time/effort, but, when made, the quality and contribution of the paper would be greatly improved.
Suggestion 1: provide an exact approach (a MIP-formulation), along with bounds (can just use the current reported best bound is the computation times are extensive, e.g. > 2 hours) and computational results. The heuristics would be better motivated, if it turns out they are (i) much faster than the exact IP approach and/or (ii) give relatively good solutions. The current motivation in lines 273-274 is simply insufficient.
Suggestion 2: in Section 1.6, provide ‘sufficiently precise’ pseudo-code for the generation of initial schedules (i.e. separate pseudo-code for 1.6.1, 1.6.2 and 1.6.3), so that they can be independently implemented and lead to similar results. In that pseudo-code, also include how you deal with the constraints in L, as well as the how many schedules are generated. In addition, the authors should mention whether it is checked that all generated initial schedules are distinct. Are duplicates possible?
Pseudo-code is also required in Section 1.7.
Suggestion 3: also add computation times for the metaheuristics. Compare these with the exact approach. Also compare the solution values with either exact solutions or some bound.
With these suggestions, the paper can be a proper benchmark for future studies. Additionally, the contribution w.r.t. computational performance would clearly make the article in-scope for PeerJ CS.

- Lines 262-265: are the random schedules all unique?
- Line 268: how is Y changed for the R different schedules? And are those initial schedules unique?
- Line 307-308: is it checked that all initial schedules are unique?
- Line 316: how were Z and alpha set?
- Line 344: the metaheuristics are not guaranteed to find even a feasible solution. A few sentences explaining what can be done in such a case would be in place. Some options I can think of: separate constraints in hard and soft constraints, or minimize the number of violated constraints. Taking this into account in the current metaheuristics is not required.
- Line 362: So R = 50?
- As the authors indicate in their reply, the bars representing two standard errors in Figures 2 and 3 are barely visible. I completely missed them in my original review. My suggestion is then to make those bars more visible (e.g. by enlarging them).
Validity of the findings: - Lines 273-274: this statement is not backed up by findings at all. Some IP models with millions of variables can be solved in mere seconds. The authors can, as mentioned before, implement an exact approach and provide computational results to back up this claim.
- Lines 394-397: without an exact, i.e. optimal, solution, the authors can't really make interesting statement regarding the presence of constraints.
- Lines 461-462: this could still be the case after the metaheuristics in this paper are applied.
- Lines 519-520: the authors claim to present fast heuristics. However, this claim is unsubstantiated, as not a single computation time is given in the paper (see suggestion 3 as well).
Additional comments: - The authors have clarified the concepts 'session' and 'concurrent'. However, in line 182, I think it would be better to state the starting and ending time of concurrent sessions coincide/are the same, rather than just overlap. Overlap might mean partial overlap, which is not allowed. This would be more in line with lines 540-542.
- My original comment regarding algorithms and heuristics is still not fully addressed. In Sections 1.6 and 1.7, the (false) usage of the word algorithm is still rampant. It also occurs in Section 2.


Instruction: You are one of the reviewers, your task is to write a review for the article. You will be given the title of the article, the number of the round in which the article is located, and your order among the reviewers.

Input:
Title: DESIGNING SPACES TO SUPPORT COLLABORATIVE CREATIVITY IN SHARED VIRTUAL ENVIRONMENTS
Review round: 1
Reviewer: 1

Output:
Basic reporting: Although lengthy, the paper is well written with clear language and structure. It provides a thorough overview of the literature and introduces the reader into the subject matter of the experimental work. The paper introduces several hypotheses that are tested and the results are discussed and summarized at the end of the manuscript. The results are presented in a clear way for the most part. Some of the figure captions could be improved with additional context and explanations.
Experimental design: The work addresses important area of shared virtual experience in the context of sound which has been by and large neglected by the VR community.

The experimental design section needs to provide more information on the subjects. It is only mentioned in the abstract that 52 users were tested, while no specific information is given on the subjects' background, age, recruitment process.

Information on the experimental hardware setup should be provided: what VR headset was used, how was the tracking of the user position done, how were the hands tracked (input device).

The experiment includes four different conditions that the subjects experienced. Multitude of conditions poses threat to bias the subjects, however the authors address the learning effect by randomization and repeated use. The final analysis is performed on the last two sessions which seems adequate to address the issue of learning.
Validity of the findings: The results are discussed in details while the authors also provide summary of the results that and several key findings that could be used as guidance in designing collaborative spaces.
Additional comments: Captions for Figures should provide additional information on what results are presented.

Figure 4, it's not fully clear how this was presented tothe users. Could there be examples added for each what the users saw?

Figures 5 and 6 should include value of N (number of measurements). The chart labels for questions could perhaps also include the theme of the questions Q1-11, as in Table 2 (in parenthesis next to the question). Why are questions there labeled as CQ1-11 while labels in Figures are Q1-11?

Figure 7, caption should explain what the labels mean; location (shown as dots), direction (shown as arrows), etc. Why was this particular group selected?

Figure 8 needs more explanation. Is this top view? How was it measured?

Page 13, last paragraph.. "The redder/bluer..." should be rephrased to refer to the color intensity.


Instruction: You are one of the reviewers, your task is to write a review for the article. You will be given the title of the article, the number of the round in which the article is located, and your order among the reviewers.

Input:
Title: DESIGNING SPACES TO SUPPORT COLLABORATIVE CREATIVITY IN SHARED VIRTUAL ENVIRONMENTS
Review round: 1
Reviewer: 2

Output:
Basic reporting: The authors present a shared virtual environment system called LeMo, which was developed to support collaborative music making between pairs of people. This study tested four boundary configurations of shared and personal space within the environment, with the goal of determining which configuration is preferred for creative collaboration by users. The authors build on their previous work in understanding personal space, or territory, within the virtual environment. In this paper, the territory is manipulated through an experimental design and tested with 26 pairs of participants.

Basic reporting was overall okay, but needs minor revisions. I would like the authors to address the following items:

Lines 102-104 need more clarification as to how the authors addressed the concern in their project, and how it differs (or not) from this study. Additional context would help here.

Line 106, define 'tabletop'.

Lines 171-172, address how the new system differs from the old one, and briefly why the changes were made. The authors also note that Figure 2 is from the other 2019 study, which is confusing to readers since it was stated that the new system is very different. Did the interface not change?

The results section could use clarifications. Please review and clarify the language in lines 350-414.

Figures 6 and 7 have corresponding colors to indicate conditions, but the distinction is not strong on black/white printouts. Please make it more clear without using color which condition is which.

Line 464: What is Cpi? I think this is a typo.
Experimental design: There were multiple areas where what appeared to be research questions were stated  (lines 46-47 and 76-78). Please clarify which set of questions you sought to answer and ensure that they are the same question if repeated elsewhere in the paper.

The experimental design itself was fine with four independent variables. The authors accounted for ordering effects through counterbalancing. The questionnaire and interviews after seem standard.

Hypotheses were clear.

I do have a concern about the Post Session Questionnaire in Table 1. The questions were either phrased positively or neutrally, with no negative statements (e.g. The spatial configuration of this virtual world was difficult for me to understand). Please provide rationale for using positive or neutral phrases in the questionnaire.

The description of how the study was conducted was thorough. The statistics provided were sufficient to understand how each condition performed.

Interview transcription and analysis was clear. The theme on learning effects was especially useful seeing as this is a limitation of such work, so it was important to address this in the results.
Validity of the findings: Aside from the language clarifications requested above, this study was controlled and methods/procedures were implemented in a standard way.

The participant quotations brought the statistical findings to life and provided a deeper level of analysis to the themes that arose out of the questionnaire. Removal of the LeMo feedback (lines 618-621) kept the paper in focus.

There are minor grammatical errors in the Discussion section. I recommend a grammar check to rectify these errors.

The discussion related back to the results of this study and tied in with the results of the author's previous study of the same system. It would be helpful if the authors restated their research questions at the beginning of the discussion, and used them as references for the structure of the discussion.

Key findings are clearly stated and are based on the results of the study. Any speculation was presented using terms such as 'suggest' or 'seems'. Design implications are also based on the results and the discussion.

Overall the findings and design implications are interesting and of value for researchers working in the creativity field and/or the virtual environment space.
Additional comments: No additional comments.


Instruction: You are one of the reviewers, your task is to write a review for the article. You will be given the title of the article, the number of the round in which the article is located, and your order among the reviewers.

Input:
Title: MOTIVATIONAL STRATEGIES AND APPROACHES FOR SINGLE AND MULTI-PLAYER EXERGAMES: A SOCIAL PERSPECTIVE
Review round: 1
Reviewer: 1

Output:
Basic reporting: This paper is a generally well-written literature review of motivational strategies in exergames. The language used throughout is of a high standard with few grammatical errors or typos. Claims made by the paper are supported by references to the literature.
The paper is logically structured, though I question the placement of tables and figures after the references as opposed to at the relevant locations in the body of the paper. Perhaps this is to do with the review manuscript format? The tables and figures are clear and well laid out, though figure 2 does not appear to provide any information that is not already present in the text. Given that figure 2 is taken from cited work, I question whether the paper is improved by the presence of this figure. Relevant raw data is shared.

Table 1 has entries for what appears to be three sets of table notes, but only one note is provided, that a duplicate of text provided above the table.

This review is relevant to the scope of the journal, and of interest both within the disciplines of computer science and health. Although there have been other reviews related to exergames in the last few years, the particular focus of this paper on motivational strategies makes this paper sufficiently distinct from these existing reviews.

The paper is well motivated, with the retention problem of exergames clearly identified, justifying the review topic. However, I note that the background section of the abstract makes an unclear claim related to this introduction. Line 18 states “[exergames] have been shown to address a variety of health-related issues.” As it stands, this reads like exergames offer health benefits beyond those given by the increase in exercise levels. If this is what the authors are claiming, this statement needs to be supported by literature. If the authors are just claiming that the benefit lies in increasing exercise levels, this sentence should be reworded to make that clear.
Experimental design: This paper falls within the aims and scope of the journal.

This paper follows a standard approach for conducting a literature review, with a database search and snowballing from an initial set of papers. However, reporting of the methodology would benefit from some clarification.

The exact terms the authors use in their database search are unclear. It reads as if the search terms listed on lines 111 and 112 were searched independently, but given the numbers reported, this seems unlikely. My own search for the term ‘active video games’ gives many more results than the authors report, or many fewer if searched in quotation marks.

The databases selected for searching should provide an appropriate coverage of the relevant literature. Was article full text searched for the terms, or just titles and abstract?
The authors also mention that the search included reference sections of previously gathered articles, but do not mention how those articles were chosen.

Line 133 states that the majority of journal articles have a high citation count. Is this relevant, and if so, is a mean with such a high SD the best way to report this fact?

Sources are thoroughly cited with relevant quotations. However, there are cases where papers are discussed but the relevance of the points identified is unclear. For example in lines 592 to 595, the authors identify specific traits associated with two different levels of adherence to playing a specific game, but the authors do not effectively connect this to the rest of the discussion in the paragraph.

The review is sensibly organised into appropriate sections and subsections.
Validity of the findings: The authors’ conclusions are clearly stated and supported by their findings in their survey of the literature. However, the conclusion section could better summarize the items discussed through the paper: the author’s discussion is broad but the summarization is brief.

Gaps in the literature and research questions for future work in the field are clearly identified. Section 3.3.1 groups social experiences and personality-based solutions as a single area, but although some personality-based solutions in the literature happen to target social experiences, these topics are not inherently related. As such, this section might be improved by separating these concepts.

It is strange to identify gamification as a motivational element in exergames. Rather, gamification is an inherent property of exergames: an exergame without gamification is no longer an exergame, merely exercise.
Additional comments: Overall this is a good paper, though a few things need to be reworded or clarified before publication.

Below is a list of formatting minor errors and odd phrasings that I identified:

There appears to be some inconsistencies in citation style within the text. In some cases, multi-author papers are listed as First Author et al., while in other cases the full set of authors are listed. An example is the contrast between the citations on lines 472 and 473, but this happens throughout the text. Additionally, on line 572, a paper is cited by the first author’s full name, rather than their surname.

There is an unnecessary comma on lines 403, 509, 685.

Line 242 “the grow strength of the avatar” this phrase doesn’t make sense.

Line 282 “abundant” should be “abundance”

Line 471 “In the context exergames” -> “In the context of exergames”

Line 555 “tailor to individual” doesn’t make sense. Maybe “tailoring to the individual”?

On line 632, competitive and cooperative social elements are listed as items that “might not be present in other games such as sedentary video games or online games”. There is no reason identified to believe that exergames are any more likely to contain these elements than sedentary video games, and I would expect that online games are in fact more likely to contain these elements than any other sort of game.

Lines 601 - 603 make a claim that certain topics are “on the research agenda for many researchers”. Although this is certainly likely to be true, it’s difficult to prove without a survey of the researchers involved and the section would be stronger without this claim.


Instruction: You are one of the reviewers, your task is to write a review for the article. You will be given the title of the article, the number of the round in which the article is located, and your order among the reviewers.

Input:
Title: MOTIVATIONAL STRATEGIES AND APPROACHES FOR SINGLE AND MULTI-PLAYER EXERGAMES: A SOCIAL PERSPECTIVE
Review round: 1
Reviewer: 2

Output:
Basic reporting: The general reporting of the literature review is of satisfactory quality. The authors demonstrated a good command of language. The paper is well-written and easily understood. The topic of sustainability of exergame is of interest to the PeerJ CS community.
Experimental design: The study design is unsatisfactory. Plenty of papers have been published in this area and the authors only identified 30 papers based on their criteria which is shocking. The paper should rather be more focused on a subset of the area such as sociality that was later brought up in the discussions. The paper reads like a literature review done for a MS thesis and is entirely inadequate for publication.
Validity of the findings: One of the biggest drawback is that the paper failed to define what the authors mean by "sustainability" as the paper primarily focuses on the "retention problem". Does retention mean sticking to the same exergame for a prolonged period of time? What if the players moved onto another game? How long is considered to be "sustainable"? The authors kept on referring to the intended outcomes of the surveyed research papers using the words "physical activities" and "exergames" interchangeably, but it isn't clear why those two are even at all correlated with one another. Does playing more exergames mean increased physical activities in the general sense? It seems that the authors fundamentally confounded these basic concepts from the onset, and the research is build on flawed premises. Also, the sections also seem rather arbitrary. Are competitive and cooperative elements of an exergame features of "gamification"? How would the authors differentiate situational interests from other contextual factors that affect a player's motivation? The literature is scattered and fragmented and no new insights is generated from the way the authors synthesized the 30 papers. The gaps section also doesn't build on the literature review at all. Personality traits is something that has been investigated in previous gaming literature which were curiously left out of the 30 papers reviewed by the authors.
Additional comments: It'd be wise if the authors would focus their review on a smaller, more identifiable, and more focused research problem of exergame, and perhaps conduct a small empirical study to demonstrate the intended claims. As of now, the scope seems to be beyond the authors' abilities to handle and this work is not of publishable quality.


Instruction: You are one of the reviewers, your task is to write a review for the article. You will be given the title of the article, the number of the round in which the article is located, and your order among the reviewers.

Input:
Title: MOTIVATIONAL STRATEGIES AND APPROACHES FOR SINGLE AND MULTI-PLAYER EXERGAMES: A SOCIAL PERSPECTIVE
Review round: 2
Reviewer: 1

Output:
Basic reporting: See general comments.
Experimental design: See general comments.
Validity of the findings: See general comments.
Additional comments: I am largely satisfied with the changes the authors have made to the paper since my original review. Below are some further grammar errors and issues with choice of language that I suggest are corrected before publication, without need for re-review.

Line 57:

"maintaining players motivation and keeping actively engaged" change to "maintaining players' motivation and keeping them actively engaged"

Lines 60-62:
Use of terms retention and sustainability interchangeably is questionable, unless there is a good reason for it such as frequent use of both terms in the literature, I would prefer the authors just use retention as sustainability holds some different implications.

Line 67: "the participation" change to "the word participation"

Line 77 "exist" -> "exists"

Line 126: If the listed databases are all of the databases used, the word "including" is unnecessary. If they are not, then all databases used in the search should be used.

Line 180 "particularly focused" -> "we focused" or "we particularly focused"

Line 692: Differentiation of sedentary videogames and online games is an odd choice for contrasting with an exergame. Both sedentary and non-sedentary games can be online.

Line 733: "We learned" is not an appropriate choice of words.

Line 744 "decrease" -> "decreased"


Instruction: You are one of the reviewers, your task is to write a review for the article. You will be given the title of the article, the number of the round in which the article is located, and your order among the reviewers.

Input:
Title: COMPONENT-ORIENTED ACAUSAL MODELING OF THE DYNAMICAL SYSTEMS IN PYTHON LANGUAGE ON THE EXAMPLE OF THE MODEL OF THE SUCKER ROD STRING
Review round: 1
Reviewer: 1

Output:
Basic reporting: The article presents a simple component-oriented acausal modelling system for application to a particular type of problems (modelling of a sucker rod string). The system is implemented in Python and utilises the available Python numerical libraries such s SciPy and SymPy. Models are specified using Python classes describing reusable components. Model variables are represented by SymPy symbols and equations are specified in a symbolic form using SymPy package. Differential equations are internally discretised in time using the first order Euler and trapezoidal rule methods yielding a system of non-linear equations (“difference equations”). Model instances are connected using pins producing an additional set of connectivity equations. Equations from the hierarchical model definition are collected and numerically solved using SymPy or SciPy packages. The system is applied to modelling of a sucker rod string and results compared to the results from Modelica.

The article requires significant changes and major revisions. In the next revision the following comments should be addressed by the authors.

The English language must be thoroughly improved throughout the manuscript. In its current state the manuscript is not acceptable for publication.

Introduction and the background are described well and the prior work is well referenced (again, the English language should be improved).

The article is not well structured and I suggest the following changes (in accordance with an acceptable format of ‘standard sections’):
1. Introduction (including the problem statement)
2. Methods/Implementation (the main characteristics and principles, and the implementation details of the pycodyn module, description of the components etc.)
3. Use cases/applications
 3.1 Case 1: Simulation of free vibrations of the sucker rod string (full description of the problem, including equations for every component and references in the literature regarding the theoretical background)
   - Model implementation(s) in Python
   - Model implementation in Modelica
 3.2 Case 2: Simulation of the pumping process by the two-section string (full description of the problem, including equations for every component and references in the literature regarding the theoretical background)
4. Results
5. Discussion
6. Conclusions
This is only a general idea, the authors are free to modify it.

The Results and Discussion sections are missing and the simulation results are only briefly presented but not discussed at all. The focus in the Implementation section should be more on general principles of the modelling system rather than explanation of the Python code.

Source code listings are given in both manuscripts and as as supplemental information. I suggest to keep the supplemental listings and remove from the manuscript those starting at lines 283, 327, 352, 368 and 398. The parts of interest can be referenced in the main text by a class or a function name.
Supplemental listings should be reorganised in the following way:
- Supplemental listing S1 should contain the pycodyn module(s) for two methods used (Euler and trapezoidal rule)
- Supplemental listing S2 should contain the Python model from Case 1
- Supplemental listing S3 should contain the Modelica model from Case 1
- Supplemental listing S4 should contain the Python model from Case 2
Again, this is only a general idea, the authors are free to modify it.

The provided source code for Case 1 and 2 models should be prepared for running in Python with no modifications from readers. In addition, the current models work only in Python 2 and should be modified to work in Python 3, too.
Experimental design: The research falls within the aims and scope of PeerJ Computer Science. However, the main idea and the novelty of the framework are not clearly described. A comparison to the available modelling approaches should be given and discussed in details (not only listed).

I encourage the idea of modelling frameworks based on high level programming languages and reuse of existing libraries for numerical integration. However, in general, performing a simulation requires the following tasks: (1) model specification (simulator-dependent), (2) internal processing of the model specification and generation of a system of equations, and (3) numerical solution of a system of equations (in general, simulator-independent). While the idea of basing a component-oriented system for model specification on SymPy is plausible, the numerical solution procedure in this work is not satisfactory and requires significant improvements. For instance, the numerical performance of the current implementation is extremely low and application to stiff problems is problematic. Furthermore, the numerical stability and accuracy are not taken into consideration (important for the first order Euler method) and the local and the global error control are not performed.

On the one hand, description of models and generation of a system of equations can be performed in many different ways depending on the type of the problem and the method applied by a simulator. On the other hand, the numerical solution procedure always requires the same information and is rather generic. Hence, the existing simulators provide either a modelling language or an API for model specification but perform the numerical solution process using the state-of-the-art ODE/DAE and linear solvers implemented in C/C++ (for performance reasons). Very low numerical performance in the pycodyn system can be resolved by using some of the available DAE solvers. For instance, the SymPy symbols for time derivatives can be added to components in the same way as for the variables. Furthermore, by assigning an index to every variable (and corresponding time derivative) it could be possible to map SymPy symbols to one-dimensional arrays of floating point values used by a DAE solver. This way it would not be difficult to implement residual and jacobian routines required by a DAE solver (i.e. Assimulo). Thus, the  additional code required to use i.e. Assimulo as a DAE solver includes a function to copy the data to and from the solver specified arrays using the mapping between pycodyn symbols and data arrays in the residual and jacobian functions. This way the performance and the quality of the numerical solution will be significantly improved. This part should be straightforward to implement and would provide a valuable addition to the framework.
Validity of the findings: Event generation is listed as a feature in the Abstract and Statement of the problem sections but not implemented.

It is stated that the results correspond to practical dynamometar cards obtained on real wells (line 593). This statement should be supported by providing the data from the real wells and a discussion added to the Discussion section.
Additional comments: No additional comments


Instruction: You are one of the reviewers, your task is to write a review for the article. You will be given the title of the article, the number of the round in which the article is located, and your order among the reviewers.

Input:
Title: COMPONENT-ORIENTED ACAUSAL MODELING OF THE DYNAMICAL SYSTEMS IN PYTHON LANGUAGE ON THE EXAMPLE OF THE MODEL OF THE SUCKER ROD STRING
Review round: 1
Reviewer: 2

Output:
Basic reporting: The article use clear English and is understandable.

However, first sentence in abstract is too strong and disputable. E.g. "poor interoperability" in the case of Modelica is not true - see FMI standard, various free fmi libraries for python (PyFMI), C++, .NET. "High cost of learning" is disputable as for specific domain experts it's more understandable than general programming language "Complexity" of implementation is disputable again. While libraries might seem complex, implementing models in some domain using existing libraries enables user to express much more complex systems very easily and plausibly: https://doi.org/10.1109/EMBC.2015.7319192 or https://doi.org/10.1016/j.bbe.2017.08.001
Experimental design: Listing from from row 131 (in Modelica) could be compared with the similar parts from row 247 (in Python) as it could be visible some bioler-plate code and major differences - e.g. rather as figure with two columns - one in Modelica and second with corresponding part in Python.
Validity of the findings: In conclusion row 614 "block diagram" is used usually in terms of causal modeling while we're talking about acausal which is opposite of "block diagram" thus I suggest either use "acausal diagram" or "component diagram". The same for caption of figure 1 and 3.

While the difference method is used, it should be discussed more about stability and robustness in conclusion section, it is just mentioned in Introduction. And compare with e.g. DASSL available in Modelica tools.
Additional comments: Congratulation for a very interesting and elegant library allowing to do acausal modeling in Python.
The real strength of the existing specialized modeling language are domain specific libraries, it would be interesting whether such libraries e.g. Modelica libraries can be transpiled to such tools like yours.

I found minor issue in supplementary materials.
The Modelica model is not reproducible in OpenModelica Editor tool. However with small edit is working. It should be wrapped in some package definition e.g.:
'''
within ;
package Pycodyn
  extends Modelica.Icons.Package;

[ content of supplementary material.mo]

end Pycodyn;
'''

I suggest to consider to upgrade to Python version 3 (e.g. using 2to3). Python 2 is going to end of life in 2020.


Instruction: You are one of the reviewers, your task is to write a review for the article. You will be given the title of the article, the number of the round in which the article is located, and your order among the reviewers.

Input:
Title: R-DECO: AN OPEN-SOURCE MATLAB BASED GRAPHICAL USER INTERFACE FOR THE DETECTION AND CORRECTION OF R-PEAKS
Review round: 1
Reviewer: 1

Output:
Basic reporting: Overall the manuscript is well written, while I found the description of Decision (Line 97-103) is not clear, which needs to be further improved. Perhaps a figure would help.
Experimental design: As mentioned above, please provide details in the Decision step on Page3.
Validity of the findings: The authors mentioned that the software is freely available, but the provided link  (http://homes.esat.kuleuven.be/jmoeyers/R DECO.zip) seems to be not working.
Additional comments: No additional comments


Instruction: You are one of the reviewers, your task is to write a review for the article. You will be given the title of the article, the number of the round in which the article is located, and your order among the reviewers.

Input:
Title: R-DECO: AN OPEN-SOURCE MATLAB BASED GRAPHICAL USER INTERFACE FOR THE DETECTION AND CORRECTION OF R-PEAKS
Review round: 1
Reviewer: 2

Output:
Basic reporting: The paper is well written and easy to follow.
Experimental design: no comment
Validity of the findings: no comment
Additional comments: The paper presents the description of an useful open-source Matlab based graphical user
interface for the detection and correction of R-peaks,  the QRS complex detection is based on and  envelope-based procedure used in combination with an adapted version of a traditional threshold-based QRS detector. The paper is well written and easy to follow. The proposals are good, and the results are interesting.  However, some minor sections of the paper need to be clarified.

Author stated that decrease in performance is generally due to poor signal quality. Therefore it should be interesting to have a discussion about the Noise Tolerance performance of the proposed QRS Detection
algorithm against noise. What is the effect of the SNR on the proposed tool? Can you tell us what is the threshold limit of SNR? under which SNR the tool can still have good results, and in which conditions the result may be seriously contaminated.  An example illustrating this point should interesting for the reader.

Can you quantify the noise effects in the QRS detector by using the noise stress test recommended by the ANSI/AAMI EC38 standard using the records from the MIT-BIH Noise Stress Test Database?


Instruction: You are one of the reviewers, your task is to write a review for the article. You will be given the title of the article, the number of the round in which the article is located, and your order among the reviewers.

Input:
Title: URBAN PUBLIC BICYCLE DISPATCHING OPTIMIZATION METHOD
Review round: 1
Reviewer: 1

Output:
Basic reporting: 1) I suggest that the title could be more specific and unnecessary words like ‘research on’ could be removed
2) Line 13, NSGA2 should be written in the full name when it first appears in the article.
3) In introduction, references should be cited to support the authors’ arguments. For example, list reference to support the sentence that the mainstream regional division method is based on the urban administrative area.
4) Authors should summarize the development of relevant research and analyze their disadvantages. Although authors briefly state the limits of existing approaches, the research gap and the main contribution of this article are not clear enough.
Experimental design: no comment
Validity of the findings: 1) Experiment results should be analyzed with more details.
2) Could you add a figure to compare routes of rebalancing using different approaches?
Additional comments: 1) The English language should be improved.
2) The figure number is not matched in the context.
3) I recommend that figures could be uploaded with higher resolution.


Instruction: You are one of the reviewers, your task is to write a review for the article. You will be given the title of the article, the number of the round in which the article is located, and your order among the reviewers.

Input:
Title: URBAN PUBLIC BICYCLE DISPATCHING OPTIMIZATION METHOD
Review round: 1
Reviewer: 2

Output:
Basic reporting: I am pleasantly surprised to see the paper full text accessible on PeerJ website.
Not sure whether to continue with this review, however, would like to do the needful as agreed.

This paper attempts to present a methodology on policy of dispatching the shared bicycle system at transit-stations or terminals. A community discovery algorithm is used to address this problem. The authors claim an improvement in comparison to the results obtained by NSGA algorithm to the tune of 20-50% of estimated scheduled distance.

The paper in general is very relevant and contemporary. Some issues are:

1. Many places authors with first name initials appear in the text,
Line 39, T. Tulabandhula [1]
Line 41, G. Q. Pan [2]

2. Incomplete reference (few samples):
[1] Tulabandhula, T., & Bodas, T. P. (2018). Method and system for dispatching of vehicles in a public transportation network
[13] Doreswamy, & Ghoneim, O. A. (2018). Traffic jams detection and congestion avoidance in smart city using parallel k-means clustering algorithm.
[22] GAO, X., Tian, Y., Sun, B., GAO, X., Tian, Y., & Sun, B. (2018). Multi-objective optimization design of bidirectional flow passage components using rsm and nsga ii: a case study of inlet/outlet diffusion segment in pumped storage power station. Renewable Energy, 115.

3. Unrelated/Irrelevant references
[24] Data was supposedly taken from this paper (see lines 260-2610
"New York Public Bicycle [24] is a people-benefit project launched by the New York City Government. Figure 5.3 displays the spatial distribution of rental sites." The reference mentioned is on crash and helmet related aspects.
[25] Shixiong Jiang, Wei Guan, Zhengbing He, and Liu Yang, “Exploring the Intermodal Relationship between Taxi and
447 Subway in Beijing, China,” Journal of Advanced Transportation, vol. 2018, Article ID 3981845, 14 pages, 2018.
448 https://doi.org/10.1155/2018/3981845.
449 26. Chao Guo, Cheng Gong, Haitao Xu, and
Experimental design: The data as explained in the paper from public sources. This is essentially from New York and Chicago cities.

New York city data (Reference 22) data is not available here

Chicago city data (reference 31)

Chicago public bicycle data [31]. paper is not accessible nor it has a doi to search. Thus I'm  unsure of the data availability.

Python Codes are provided with the manuscript, but data is missing.
Validity of the findings: Thus it would be difficult to validate the findings given these limitations. This study amounts to using Technique A is better than rest B, C etc., without gaining any insights into the problem of allocation of bicycles to various points in the city. Also this paper needs to establish why Technique A is better. Sections 4 and 5 needs improvement in terms of this comprehension and being less abstract.
Additional comments: 1) The article can be improved by trying to address the issues brought out here in.
2) The pitch (USP) of the paper needs to be highlighted
3) References need to be improved (complete and relevant ones only needed).


Instruction: You are one of the reviewers, your task is to write a review for the article. You will be given the title of the article, the number of the round in which the article is located, and your order among the reviewers.

Input:
Title: URBAN PUBLIC BICYCLE DISPATCHING OPTIMIZATION METHOD
Review round: 1
Reviewer: 3

Output:
Basic reporting: The paper proposes an improved Community Discovery algorithm based on Multi-objective Optimization (CDoMO) in order to solve the problem of regional division of public bicycles. The algorithm fully considers the special law of public bicycle lease/return, and in order to balance the scheduling workload between areas, the regional scheduling workload index is proposed. In particular, two objective functions are considered, i.e., minimize the variance between the estimated dispatch distances between each area and minimize the variance between the numbers of sites in each area. Moreover, the proposed algorithm can work in real time and dynamically.
Experimental design: no comment
Validity of the findings: no comment
Additional comments: The paper is well-organised and the exposition is very clear.
Moreover, I really appreciate the figures shown by the authors, even if their resolution needs to be improved.

I have just a little concern about the adopted acronyms. Please, use capital letters for words involved in the acronym.


Instruction: You are one of the reviewers, your task is to write a review for the article. You will be given the title of the article, the number of the round in which the article is located, and your order among the reviewers.

Input:
Title: URBAN PUBLIC BICYCLE DISPATCHING OPTIMIZATION METHOD
Review round: 1
Reviewer: 4

Output:
Basic reporting: The article focused on the arimethic of how to get reasnonable dispatching area division, it is a true interesting topic. Unfortunately, the paper has some major weakness like below.  1) The article should include sufficient introduction and background to demonstrate how the work fit into the dispatching research of public bicycle. For example, there are more literature on how to divide dispatching area should be appriciately referenced.  Besides, the prediction of bicycle leaseing demand is also a importand reseach point that occpies the whole Section 3.3, 3.4.1 and 3.4.2 in the paper, but there is no any background introduction about it and we know there are tremendous prior literature about it.    2) It is difficult to estimate the main contribution of this paper due to the lack of professional article structure and sufficient introduction. From my view, there lacks deep understanding on the physical issue of how to divide the dispatching area of pyblic bicycle.    3)   The algorithem is not descriped properly such as formula (3.11) , (3.12).  Why it is the right formua to fit for your research topic?    4) There are many small errors in article writing and orgnanizing,  for example, After Section 3.3, the followed should be Section 3.3.1 and 3.3.2  instead of Section 3.4.1 and 3.4.2.
Experimental design: no comment
Validity of the findings: no comment
Additional comments: The article focused on the arimethic of how to get reasnonable dispatching area division, it is a true interesting research. Unfortunately, the paper has some major weakness like below.  1) The article should include sufficient introduction and background to demonstrate how the work fits into the dispatching research of public bicycle. For example, there are more literature on how to divide dispatching area should be appriciately referenced.  Besides, the prediction of bicycle leaseing demand is also a importand reseach point that occpies Section 3.3, 3.4.1 and 3.4.2 in the paper, but there is no any background introduction about it and we know there are many prior literature about it.    2) It is difficult to estimate the main contribution of this paper due to the lack of professional article structure and sufficient introduction. From my view, there lacks deep understanding on the physical issue of how to divide the dispatching area of pyblic bicycle.    3)   The algorithem is not descriped properly such as formula (3.11) , (3.12).  Why it is the right formua to fit for your research topic?    4) There are many small errors in article writing and orgnanizing,  for example, After Section 3.3, the followed should be Section 3.3.1 and 3.3.2  instead of Section 3.4.1 and 3.4.2.


Instruction: You are one of the reviewers, your task is to write a review for the article. You will be given the title of the article, the number of the round in which the article is located, and your order among the reviewers.

Input:
Title: URBAN PUBLIC BICYCLE DISPATCHING OPTIMIZATION METHOD
Review round: 2
Reviewer: 1

Output:
Basic reporting: 1) The references issues remain largely unaddressed  particularly comments 1, 2 and 3

2) incomplete references 1, 31 etc (in the v1 manuscript)

3) Data availability is not addressed: just the bikesharing programme addresses are given: I do not see link to any data, except the static locations.

4) I would like to know whether the dynamic bike availability at locations are shared by these cities with the researchers.
Experimental design: no comment
Validity of the findings: no comment
Additional comments: No comments


Instruction: You are one of the reviewers, your task is to write a review for the article. You will be given the title of the article, the number of the round in which the article is located, and your order among the reviewers.

Input:
Title: SYNTHETIC DATASET GENERATION FOR OBJECT-TO-MODEL DEEP LEARNING IN INDUSTRIAL APPLICATIONS
Review round: 1
Reviewer: 1

Output:
Basic reporting: The authors propose a method based on 3D modeling and photogrammetry to create synthetic datasets of objects for warehouse management. The results confirm the validity of the approach, in the form of a CNN trained on the synthetic data and achieving high classification results in real world applications.
The paper is interesting, technically sound, and well-written. However, some details are too sparsely described, such as:
•	The ‘3d modeling’ section is rather sparsely described. Maybe consider extending the description to help ensure reproducibility of the procedure.
•	More details on the synthetic datasets and the training data should be provided. How many classes are generated? How many images per class? What is the size of the image?
•	More details on the training procedure should also be provided. E.g., how many epochs are used to train the network? On which computing platform?
Based on these considerations, it is my opinion that the paper requires a major revision to increase the clarity of the description and ensure the reproducibility of the procedure and the results.
Experimental design: Experimental procedure is too sparsely described. In particular:
•	More details on the synthetic datasets and the training data should be provided. How many classes are generated? How many images per class? What is the size of the image?
•	More details on the training procedure should also be provided. E.g., how many epochs are used to train the network? On which computing platform?
Validity of the findings: 'no comment'
Additional comments: No additional comments


Instruction: You are one of the reviewers, your task is to write a review for the article. You will be given the title of the article, the number of the round in which the article is located, and your order among the reviewers.

Input:
Title: SYNTHETIC DATASET GENERATION FOR OBJECT-TO-MODEL DEEP LEARNING IN INDUSTRIAL APPLICATIONS
Review round: 1
Reviewer: 2

Output:
Basic reporting: 1.Some figures are fuzzy. For example, the Figure 1 is a little bit blurry, it could be better if a higher resolution one is used. Besides, in the Figure 7, the label of abscissa is incomplete.
Experimental design: 1. In the subsection of Image Classification, authors just use their synthetic dataset to do the experimentation with different CNN architectures. However, they also need to use real datasets for comparative experiments to make their results convincing.
Validity of the findings: 1. The whole paper lacks enough innovation points, and the methods used are mostly those proposed by others. The author needs to expound his new method and innovation point emphatically
Additional comments: In the section of LIMITATIONS, the paper shows that "the process of acquiring 3D models and preparing them for rendering required roughly 15 minutes of manual work per product." When the object is large, the author should have further thoughts on how to efficiently scan it with less resources and time.


Instruction: You are one of the reviewers, your task is to write a review for the article. You will be given the title of the article, the number of the round in which the article is located, and your order among the reviewers.

Input:
Title: SYNTHETIC DATASET GENERATION FOR OBJECT-TO-MODEL DEEP LEARNING IN INDUSTRIAL APPLICATIONS
Review round: 2
Reviewer: 1

Output:
Basic reporting: It is my opinion that the authors answered the issues raised by the reviewers in a satisfactory way. Therefore, I have no further comments.
Experimental design: no comment
Validity of the findings: no comment
Additional comments: No additional comments


Instruction: You are one of the reviewers, your task is to write a review for the article. You will be given the title of the article, the number of the round in which the article is located, and your order among the reviewers.

Input:
Title: SYNTHETIC DATASET GENERATION FOR OBJECT-TO-MODEL DEEP LEARNING IN INDUSTRIAL APPLICATIONS
Review round: 2
Reviewer: 2

Output:
Basic reporting: no  comment
Experimental design: no comment
Validity of the findings: no comment
Additional comments: Authors have well corrected the deficiencies and weaks I put forward.I'm satisfied with the revised paper.


Instruction: You are one of the reviewers, your task is to write a review for the article. You will be given the title of the article, the number of the round in which the article is located, and your order among the reviewers.

Input:
Title: LEVERAGING ASPECT PHRASE EMBEDDINGS FOR CROSS-DOMAIN REVIEW RATING PREDICTION
Review round: 1
Reviewer: 1

Output:
Basic reporting: The paper is well written, self-contained and easy to read.
Literature review is comprehensive.
Experimental design: The experimental design is appropriate.
However, I feel that a very relevant baseline is missing. Some background is required to make my point clear:

The two models in this paper create vectorial representations of reviews. These representations are built by  concatenating a general word embedding-based representation (aggregating the embeddings of all words within the review) with aspect phrase embeddings (aggregating embeddings of words within phrases formed by opinions words + nouns or verbs). The opinion words are taken from a given opinion lexicon.

Proposed representations outperform the baseline that uses only a "general" word embedding representation. This is because representations coming from opinion words are being amplified in the final vector (they are included in the general embedding and the the aspect phrase embedding).

The paper would benefit a lot from further experiments including representations based purely on the opinion words. Perhaps simple representations that  only use  the embeddings of the opinion words or just the polarities of these words would work well.

It is hard to judge the contribution of this work without seeing these results.
Validity of the findings: Conclusions are well stated, but as discussed above, some additional experiments are required in order to judge the validity of the findings.
Additional comments: No additional comments


Instruction: You are one of the reviewers, your task is to write a review for the article. You will be given the title of the article, the number of the round in which the article is located, and your order among the reviewers.

Input:
Title: LEVERAGING ASPECT PHRASE EMBEDDINGS FOR CROSS-DOMAIN REVIEW RATING PREDICTION
Review round: 1
Reviewer: 2

Output:
Basic reporting: The paper presents an approach for rating prediction based on reviews leveraging on aspect phrase embeddings extracted from the reviews. This, in turn, enables the development of both in-domain and cross-domain review rating prediction systems. Particularly, the paper address an interesting issue within the rating prediction problem, which is the prediction for less popular, usually neglected domains. I think the problem of cross-domain in small domains is interesting and the experimental results suggest that there is an improvement in prediction for such domains.
Experimental design: The experimental results and the dataset used are correctly reported. The selection of the baselines need, however, more justification. The selection of the work of Fan and Khademi (2014) is hard to understand. It is a baseline using only text, which is the relevance of this particular work? And also why not to include other state-of-the-art works as baselines. The explanation of that other works made use of user data is not clear, which kind of data and why not to use it here?.
Validity of the findings: I think the conclusions can be more compelling about the contribution to the cross-domain prediction in small domains and include a discussion analyzing possible implications or even further improvements. For example, the results would be even better considering translating the knowledge from big domains (like hotels or restaurants) to small ones that are somehow related? What if domains can be hierarchically organized?
Additional comments: The paper have, however, some issues that can be improved. I believe the first three sections (Introduction, Background and Related Works) are a little misleading about clarifying the motivation and contribution of the paper. On the one hand, the authors emphasize the problem of the  cross-domain in small domains, this problem is hardly discussed in the related works (which is the novelty of the paper in this regard? why not transfer learning or other approaches?). On the other hand, they also claim as a contribution to the extraction of aspect phrases mentioned in the text, but it is not clear the significance or previous works in adding phrases to aspect opinion mining. The three sections should be more focused toward highlighting the contribution(s) and the difference with other approaches. The second section may be even removed or merged with the other two.


Instruction: You are one of the reviewers, your task is to write a review for the article. You will be given the title of the article, the number of the round in which the article is located, and your order among the reviewers.

Input:
Title: NIPS4BPLUS: A RICHLY ANNOTATED BIRDSONG AUDIO DATASET
Review round: 1
Reviewer: 1

Output:
Basic reporting: Important issues:

- In the abstract, this dataset is qualified as "the first richly annotated birdsong audio data set." This is a very strong affirmation that is not clearly supported in the manuscript. I suggest adding a section or a table, comparing this dataset with datasets used in previous studies in bioacoustics.

- In lines 53-55, it is said that "many ecological datasets lack temporal annotations of bird vocalisations even though they are vital to the training of automated methods that predict the temporal annotations [...]" This is not always true since some recognition techniques are designed to handle data with imprecise annotations (e.g., multi-instance learning, or weakly supervised learning; as an example, see https://arxiv.org/pdf/1606.03664.pdf). I suggest to the authors mentioning this type of algorithms and how the dataset can help them (they might work better having some data precisely annotated, also, it might help for performance evaluation).

Less important points:

- The origin of the name "NIPS4B" could be explained in the introduction.
Experimental design: - Locations, annotation procedure, and settings are well described.
Validity of the findings: No comments
Additional comments: - In this paper, a fully annotated dataset of birdsong recording is introduced.

- Since the dataset was used in 2013, it would be important to enhance the details that were recently added.


Instruction: You are one of the reviewers, your task is to write a review for the article. You will be given the title of the article, the number of the round in which the article is located, and your order among the reviewers.

Input:
Title: NIPS4BPLUS: A RICHLY ANNOTATED BIRDSONG AUDIO DATASET
Review round: 1
Reviewer: 2

Output:
Basic reporting: The paper describes a database for bird vocalizations detection and species recognition in audio recordings.
The suggested annotated dataset is useful because the task of localizing vocalizations in the spectrogram for several species is important but there aren’t many available databases around. The paper is well written though it would benefit from a revision.
It is a small database of few hundred of recordings and it will not suffice to train generic bird-recognizers but it is a good start and we hope that it would be enlarged in the future. This paper stands a merit but I suggest the following changes:
1.	SM2BAT,SMX-US give details of these products i.e. company, country city.
2.	A paper must be self-sufficient. The reader should not have to refer to other papers that have part of the info that this paper needs. Having said that, I suggest you include in the supplement a) the full list of bird species b) the annotation file of the database
3.	I cannot read the x-axis legend of Figure 2. The legend of regions at top right falls on the measurements. One way is to rotate by 90o end enlarge. Another way is to include a better quality figure in the supplementary material
4.	Provide a link of where the database can be found (most important suggestion). This paper is not of any use to the reader if it is not accompanied by a link pointing to the database and annotation files. I made a quick check and I believe it is available but you need to make clear to the readers of this journal where it is located.
Experimental design: The suggested annotated dataset is useful because the task of localizing vocalizations in the spectrogram for several species is important but there aren’t many available databases around.
Validity of the findings: Please, provide a link of where the database can be found (most important suggestion). This paper is not of any use to the reader if it is not accompanied by a link pointing to the database and annotation files. I made a quick check and I believe it is available but you need to make clear to the readers of this journal where it is located.
Additional comments: No additional comments


Instruction: You are one of the reviewers, your task is to write a review for the article. You will be given the title of the article, the number of the round in which the article is located, and your order among the reviewers.

Input:
Title: NIPS4BPLUS: A RICHLY ANNOTATED BIRDSONG AUDIO DATASET
Review round: 1
Reviewer: 3

Output:
Basic reporting: This paper reports on a new richly annotated birdsong audio dataset, called NIPS4BPLUS, which has both label and temporal annotation information for each 5-sec short audio recording. The author showed the procedure of data collection and annotation in detail and showed example use of the data set for machine learning which was reported in another paper.
Experimental design: The procedure of data creation is well described. However, I think that the more information about the temporal annotation is beneficial for readers to understand the characteristics of the data set more clearly.
For example, the total song (or call) duration and song count for each species over all recordings can provide basic information of the temporal annotation.
Validity of the findings: As an example use section shows the NIPS4BPLUS can provide a good dataset for machine learning tasks. I think that it would be better to elaborate more on whether and how the newly added temporal annotation data can facilitate learning performance or learning procedures.

In addition, the authors emphasized that this is the first richly annotated birdsong audio dataset. However, the Bird-DB also appears to be a rich data set in that it provides exact temporal annotation information of various species in recordings.

Arriaga JG, Cody ML, Vallejo EE, Taylor CE. 2015. Bird-DB: a database for annotated bird song sequences. Ecol Inform. 27:21–25.
Additional comments: I believe that the paper and the proposed dataset can significantly contribute to birdsong classification studies if some more information is provided as described above.


Instruction: You are one of the reviewers, your task is to write a review for the article. You will be given the title of the article, the number of the round in which the article is located, and your order among the reviewers.

Input:
Title: NIPS4BPLUS: A RICHLY ANNOTATED BIRDSONG AUDIO DATASET
Review round: 2
Reviewer: 1

Output:
Basic reporting: The authors clearly describe the advantages of their fully annotated dataset of bird vocalizations. They provide references to other bioacoustic datasets and mention the processing and recognition techniques that have been applied to them. I suggest considering, as an additional dataset, this one: https://www.kaggle.com/c/mlsp-2013-birds

It has been used in several publications, like the ones below:

https://www.sciencedirect.com/science/article/pii/S1574954115000102
https://link.springer.com/article/10.1007/s11265-016-1155-0
https://academic.oup.com/biolinnean/article/115/3/731/2440478
Experimental design: In the manuscript, the authors provide some statistics of the dataset. I suggest including also the structure of the folders and the format of the files.
Validity of the findings: My main concern is that the dataset is not available. I suggest to share the dataset and the source code to generate the statistics provided in the manuscript, and some demos to process the recordings.

On line 121, it is said that only species that covered at least  7 recordings were included. I guess that that number wouldn't be enough to train a recognition model.
Additional comments: Since the purpose of this manuscript is to describe a dataset, it should be publically available. I would suggest accepting the manuscript after taking a look at the dataset.

Below are some minor suggestions:

- Line 100: The meaning of the abbreviation should be introduced the first time that is mentioned
- Line 108: Try to avoid the redundancy on the phrase "The recorders used to acquire the recordings"


Instruction: You are one of the reviewers, your task is to write a review for the article. You will be given the title of the article, the number of the round in which the article is located, and your order among the reviewers.

Input:
Title: NIPS4BPLUS: A RICHLY ANNOTATED BIRDSONG AUDIO DATASET
Review round: 2
Reviewer: 2

Output:
Basic reporting: The authors have answered/corrected all points raised by my review.
As far as it concerns me, the manuscript should be accepted for publication
Experimental design: The authors have answered/corrected all points raised by my review.
As far as it concerns me, the manuscript should be accepted for publication
Validity of the findings: The authors have answered/corrected all points raised by my review.
As far as it concerns me, the manuscript should be accepted for publication
Additional comments: The authors have answered/corrected all points raised by my review.
As far as it concerns me, the manuscript should be accepted for publication


Instruction: You are one of the reviewers, your task is to write a review for the article. You will be given the title of the article, the number of the round in which the article is located, and your order among the reviewers.

Input:
Title: NIPS4BPLUS: A RICHLY ANNOTATED BIRDSONG AUDIO DATASET
Review round: 2
Reviewer: 3

Output:
Basic reporting: The manuscript was significantly improved according to the reviewers' comments.
Experimental design: The manuscript provided sufficient information about the proposed dataset.
Validity of the findings: The more detailed description of the benefit of the use of the proposed data set was provided.
Additional comments: The manuscript was significantly improved according to the reviewers' comments.
I really appreciate detailed replies to my comments.


Instruction: You are one of the reviewers, your task is to write a review for the article. You will be given the title of the article, the number of the round in which the article is located, and your order among the reviewers.

Input:
Title: CONCURRENT DESIGN OF CONTROL SOFTWARE AND CONFIGURATION OF HARDWARE FOR ROBOT SWARMS UNDER ECONOMIC CONSTRAINTS
Review round: 1
Reviewer: 1

Output:
Basic reporting: Well written, well structured. The authors are professionals and this shows. Background and context are OK, but I noted one missing reference that evolves sensory layout on a swarm of real robots.

https://www.researchgate.net/publication/290169158_Evolution_Individual_Learning_and_Social_Learning_in_a_Swarm_of_Real_Robots

Figures and Tables are well made.

The main point of criticism is about the contributions stated in the Intro. That offers two contributions, of which the second one “(ii) we generalize the notion of automatic design of robot swarms to include economic constraints.” is heavily overclaimed. What I mean is the ALL design problems are subject to multiple constraints. Furthermore, constraints can be divided into several categories. Including economic constraints is NOT generalizing the notion of (automatic) design. It merely considers constraints (one, to be precise) from the category Economic.
Experimental design: The main RQ is  "Can Waffle select mission-specific hardware together with an appropriate control software?"
and this is a relevant one. Note that the question concerns a particular SYSTEM (Waffle), not a generic algorithm or methodology.

The experiments are well designed and conducted. I have one question, though, Table 4 shows specific values of parameters - how were these values determined?
Validity of the findings: The main RQ was  "Can Waffle select mission-specific hardware together with an appropriate control software?"
and indeed
"In the experiments presented in the paper, Waffle was able to concurrently design the control software and configure the hardware of a robot swarm."

This leads to revisiting the first contribution: "we address the concurrent automatic design of control software and the automatic configuration of the hardware". I would prefer to sharpen this to what the paper really does and demonstrates.
Additional comments: Good work, good paper, but the value is blurred by the somewhat misplaced / misphrased main contributions.