Instruction: You are one of the reviewers, your task is to write a review for the article. You will be given the title of the article, the number of the round in which the article is located, and your order among the reviewers.

Input:
Title: LANDMARK-BASED HOMOLOGOUS MULTI-POINT WARPING APPROACH TO 3D FACIAL RECOGNITION USING MULTIPLE DATASETS
Review round: 3
Reviewer: 1

Output:
Basic reporting: see below
Experimental design: see below
Validity of the findings: see below
Additional comments: I have reviewed the new version of the manuscript “Landmark-based homologous multi-point warping approach to 3D facial investigation of multi-datasets” where the Authors describe a new/improved method of placing (semi)landmarks on 3D surfaces of human faces.

I have to say I have mixed impressions about this new version. On one hand, I have to commend the Authors for the effort they put in and I have to say that the manuscript has certainly been improved. On the other hand, however, it seems this manuscript is not improving at the rate I was hoping for, and it seems still far from being acceptable for publication.

Differently from previous reviews where I have been very detailed, here I will provide just a few pointers, and I invite the Authors to go back to my last review, as various issues are still there and need further clarification/rewriting/improvement.

What is new here compared to what is existing is still not 100% clear to the reader. This should also be incorporated in the Abstract.

A series of statements are still unwarranted and unsupported.
For instance, at lines 51-52, a claim is made that “the complex and time-consuming computational process has made research into 3D facial landmarks generally unreproducible.”. This is a very strong statement, not very well supported by evidence. The Authors, if they want to make such claims, should refer to actual results, where reproducibility (or similar parameter, such as repeatability) is quantified, citing the relevant research.
Similarly, at lines 285-288 a suggestion is made that by using projection and iterative sliding “Using this method, the issues of flawed data are automatically handled” and that “This eliminated the hassle of preliminary imputation of missing landmarks”. This, at least in the way it is written, is not correct. Normally, landmarks are missing because a certain structure is missing from the data, and no sliding is going to take care of that per se.
Again, these are just examples.

Some of the analyses are not adequately/properly described.
For instance, at lines 353-354 a statement is made that “The statistical significance of pairwise differences in mean shapes was assessed in 1,000 permutations using the Procrustes distance by a pooled within-group” and this does not correspond to any analysis I know of. Probably, the Authors are conflating in the same sentence a test of differences based on permutations and Procrustes distances with a CVA, which uses pooled within group covariance and is described in the next sentence. Similar unclear references to “pooled within group” are made elsewhere (L418-419).

As I mentioned before, here what the Procrustes ANOVA/analysis of error is saying is not clear, as the Authors do not describe exactly how the analysis has been performed.
Earlier, I had given a suggestion of one way of carrying such an analysis. Now, from the section at lines 316-319 is not clear whether what I had suggested is exactly what has been done. If it is, the Authors should clarify that the same individuals (apparently 5 per dataset) were acquired nine times each using the three templates.
Importantly, if this is what they have done, the Authors should be clear when discussing what their results mean. That is, this technique simply allows to say that changing the template used, the shape does not change that much (compared to other biological and non-biological sources of variation). This does not mean that the method is generally speaking robust (as implied in the Abstract).
Similarly, the Authors should warn the reader that the differences found between datasets are at least in part (hard to say how much) due to differences in the way these datasets were acquired.

I should notice that for all purposes in biology a significant allometric effect accounting for 13.28% of total variation (lines 435-436) is far from negligible.

Further, I am not clear about whether all material has been updated.
For instance, Figure 1 still reports “average landmark error” and “bias” which have been removed from the text, and Figure 5 still refers to asymmetry.
I also note that the “track changes” version the Authors have provided for review was not showing all the changes from the previous draft (in fact, it was showing very few compared to the ones made).

There are still enormous problems with the English language and, to reiterate, even if the rest was perfect, I would not be able to suggest acceptance with such an English language.
I do understand that not being native speakers creates a challenge. However, too much is lost in translation for a reader to actually understand what the Authors mean.

In their reply to my comments, the Authors wrote “All necessary grammatical blunders have been corrected before we finally handed over the manuscript to professional native speaker editor.”. Well, whoever was this “professional native speaker editor”, he/she did a very poor job, as even typos and misuse of English language are still present. For instance, at line 161 “in three-dimensional” is used in lieu of “in three dimensions”. Or at line 122, where the Authors write “The approach demonstrated suitability cases”, probably meaning something like “The approach proved suitable in cases”.

Most importantly, here the main issue is not typos or grammar, but the fact that what the Authors actually mean is not understandable. This is why I suggested a native English speaking colleague: this must be someone who understands what the Authors have done and mean. A few examples of this:

-	The sentence at lines 25-27, in the Abstract, is not clear as a reference to “datasets” is made; do the Authors mean “each of the target objects in a given dataset” when describing their method? (as this is the part where how the method works is presented); similarly, what does “and the workflow is compared” means in the context at line 29?

-	At line 40, the Authors write “Besides the pose-invariant and illumination challenges”, but in the previous sentence the change of pose is presented as a challenge. Considering that “being pose-invariant” means that “pose does not matter”, it is not clear what the Authors mean. Probably, they meant “Besides variation due to pose and illumination affecting 2D face data” (or something along these lines)

-	The sentence at lines 156-159 is not clear; probably the Authors refer to using a simpler workflow, compared to the one they cite, by performing an existing step in Viewbox; but the way this is written many interpretations are possible

-	The Authors claim that “Our analyses demonstrated significant differences in facial symmetry” (L463) but this is not what – I think – the Authors have done as they have compared the average shape (of the symmetric component) across groups and the symmetric component of shape between individuals; this is quite difference from finding differences in symmetry (which would mean that one group is more symmetric than another, for instance)

In addition to this, there is a broader issue of how the manuscript is constructed, in that it does not follow a clear logic and path. This is particularly evident in the Introduction, where a series of statements more or less true (with some issues with interpretation and English language) are made. However, the order of these statements is not very clear or functional in terms of logic.

Again, all of this requires not simply changing a few typos but extensive re-thinking and rewriting in a simple and intelligible fashion (and following a clear logic when moving from one section to another) of what the Authors want to say, so that it is clear to the reader.


Instruction: You are one of the reviewers, your task is to write a review for the article. You will be given the title of the article, the number of the round in which the article is located, and your order among the reviewers.

Input:
Title: IMPROVING PARALLEL EXECUTIONS BY INCREASING TASK GRANULARITY IN TASK-BASED RUNTIME SYSTEMS USING ACYCLIC DAG CLUSTERING
Review round: 1
Reviewer: 1

Output:
Basic reporting: Problem is well defined and understood, references to other articles are relevant. Structure is conform to professional article.
Experimental design: This work is one step further in study of "acyclic DAG clustering" which permit better performance in case of fine-grain parallelism. To prove validity of this work, they use both simulated and real execution and compare tendencies, which is a technique used in numerous articles. Multiple type of graphs have been used for simulating a wide range of problems.
Validity of the findings: The new clustering algorithm is an enhancement of clustering algorithm describe in another article, additions are well identify and explanations of them are correct.
There is a clear advantage for this new algorithm, this is the key of this article. Results seem reliable and clear to look at.
Unfortunately, simulated results don't match real execution results but it's not a big deal because it give us a good approximation of algorithm efficiency. Authors don't hide this fact and try to explain it.
Additional comments: Good job for finding these new heuristics !


Instruction: You are one of the reviewers, your task is to write a review for the article. You will be given the title of the article, the number of the round in which the article is located, and your order among the reviewers.

Input:
Title: IMPROVING PARALLEL EXECUTIONS BY INCREASING TASK GRANULARITY IN TASK-BASED RUNTIME SYSTEMS USING ACYCLIC DAG CLUSTERING
Review round: 1
Reviewer: 2

Output:
Basic reporting: This paper describes a strategy to partition a task graph, with the aim to increase task granularity and speed up the parallel execution time of applications.

The authors provide a good overview of the research topic, and the manuscript and sound and clear. However, I have some major concerns which are detailed in the following comments.

MAJOR REMARKS

Lines 43-45: "When the granularity is too small, the overhead of task management, and the potential data movements, becomes dominant and can dramatically increase the execution time due to the use of synchronizations."
I suggest to include a reference to this work:
Tagliavini, Giuseppe, Daniele Cesarini, and Andrea Marongiu. "Unleashing fine-grained parallelism on embedded many-core accelerators with lightweight OpenMP tasking." IEEE Transactions on Parallel and Distributed Systems 29, no. 9 (2018): 2150-2163.

Line 57: "Hence, the usual graph partitioning methods do not work because they do not take into account the direction of the edges."
The expression "usual graph partitioning methods" is not self-explanatory, the authors should provide at least a reference. For instance:
Hendrickson, Bruce, and Tamara G. Kolda. "Graph partitioning models for parallel computing." Parallel computing 26, no. 12 (2000): 1519-1534.

Line 64: "We provide two variants of the GDCA".
For the sake of clarity, the authors should include a quick description of the two variants in the introduction.

Line 68/69: "DGCA and our approach"
It is not clear if the authors provide a comparison with a "vanilla" DGCA and their variants, or not. Please provide more details or rephrase this sentence.

Line 149: "Implicitly, the hardware or execution model should have some overheads, which could come from the management of the tasks for example, or the minimal execution time will always be obtained without clustering, i.e. any clustering of tasks will reduce the degree of parallelism without offering any advantages."
The idea is clear, but I disagree on the premise: real hardware platforms always suffer for overheads.

Line 170: "However, looking at the edge-cut to measure the quality is not relevant when the final and only objective is the parallel efficiency"
In this context, the adoption of the term "quality" is not clear.

Lines 224-240:
Considering the limited description provided in this paragraph, the contributions of this paper seem to be quite limited. The authors should provide more details to give strength to their contributions.

Lines 242-273:
This section looks disjoint by the rest of the paper, and it also lacks references to the SoA. I suggest a citation to the following paper:
Kestor, Gokcen, Roberto Gioiosa, and Daniel Chavarrıa-Miranda. "Prometheus: scalable and accurate emulation of task-based applications on many-core systems." In 2015 IEEE International Symposium on Performance Analysis of Systems and Software (ISPASS), pp. 308-317. IEEE, 2015.

MINOR REMARKS
The authors should double-check the manuscript for typos, e.g.::
- Line 61: "we use generic algorithm" -> "we use a generic algorithm"
- Line 193: "This strategy ensure" -> "This strategy ensures"
- Line 127: "While this targets the same problem, we arge that execution time is the only criteria that should evaluated."
This sentence contains at least two typos and a grammatical error.
Experimental design: - Lines 135-137: "The real application we used to assess our method solves the transport equation on unstructured meshes. Task-based implementations to solve the transport equation on a grid (i.e. structured and regular mesh) have already been proposed by Salli Moustafa et al. [10]."
Overall, the experimental section is missing a substantial comparison with the state-of-the-art.
Since this application is a potential baseline for the proposed approach, I think that it would be really interesting from a scientific point of view to provide a comparison in the experimental section.
Validity of the findings: no comment
Additional comments: No additional comments


Instruction: You are one of the reviewers, your task is to write a review for the article. You will be given the title of the article, the number of the round in which the article is located, and your order among the reviewers.

Input:
Title: IMPROVING PARALLEL EXECUTIONS BY INCREASING TASK GRANULARITY IN TASK-BASED RUNTIME SYSTEMS USING ACYCLIC DAG CLUSTERING
Review round: 2
Reviewer: 1

Output:
Basic reporting: no comment
Experimental design: no comment
Validity of the findings: Taking into account the additional explanations provided in the rebuttal letter, I gave a more in-depth look at the code base to better understand the experimental setup. Finally, I do not have further doubts about the validity of the findings.
Additional comments: The answers provided by the authors in the rebuttal letter are convincing. From my point of view, the manuscript is now in good shape for publication.


Instruction: You are one of the reviewers, your task is to write a review for the article. You will be given the title of the article, the number of the round in which the article is located, and your order among the reviewers.

Input:
Title: HACSIM: AN R PACKAGE TO ESTIMATE INTRASPECIFIC SAMPLE SIZES FOR GENETIC DIVERSITY ASSESSMENT USING HAPLOTYPE ACCUMULATION CURVES
Review round: 1
Reviewer: 1

Output:
Basic reporting: no comment
Experimental design: no comment
Validity of the findings: no comment
Additional comments: Based on previous series of work in related field(s), the authors have introduced a new simulation algorithm for calculating sample sized needed in DNA barcoding studies. The manuscript is well written with clear logic and enough cited literatures.

Considering the progresses in next-generation sequencing and species delimitation, I suggest that the authors consider more complex biological targets such as multiple loci or genomic data (e.g., SNP) rather than the DNA barcode region (i.e., 648 bp of COI gene) in future work.


Instruction: You are one of the reviewers, your task is to write a review for the article. You will be given the title of the article, the number of the round in which the article is located, and your order among the reviewers.

Input:
Title: HACSIM: AN R PACKAGE TO ESTIMATE INTRASPECIFIC SAMPLE SIZES FOR GENETIC DIVERSITY ASSESSMENT USING HAPLOTYPE ACCUMULATION CURVES
Review round: 1
Reviewer: 2

Output:
Basic reporting: Summary/gist: The authors' hypothesis stems from assessing the genetic variation using the molecular techniques such as DNA barcoding  which is widely employed in the 'Now Generation Sequencing' era. The authors give a strong emphasis on existing approaches and present a stochastic model using Monte Carlo simulations by employing an iterative algorithm.  The algorithm, as the authors state, appears to be a fine-tuned approach based on Philips et al., 2015.  Furthermore, they study the effect of haplotype frequences in cytochrome c oxidase subunit 1, a mitochondrial 5'UTR sparsed region.  Finally they present the benchmark/proof-of-concept in fishes  besides taking certain hypothetical examples.

Strengths:  The work is absolutely novel,  the need of the hour, and a subject of interest coalescing statistical geneticists and biologists towards development of barcoding markers.  The manuscript is written well with good review and statistical interpretation.

Weaknesses:  Sometimes, the flow is too much carried away and the flow could have been better with a methodological flowchart.

The R algorithm could be linked with github/bitbucket repositories.

Reinstantiation of codes, if any must be mentioned as per the Minimum Information About Bioinformatics Investigation ( MIABI) guidelines.
Experimental design: The methodology of Philips et al.,  brings an overestimates/overfitting of the samples and this brings up the rationale where the authors develop a more iterative/stochastic model in the form of HACSim.  The figure 1 represents the schematic representation of this.

The haplotype variation could be best set for  Metagenomic samples. For examples, binning the reads into haplotypes, and further into strains would allow us to define  the specific barcodes at species level. Although the authors have subtly mentioned the role of COI and employed certain frequencies, they could have considered 'metagenomes'  for that barcoding is a big problem for assessing the metagenomic samples a sthey are in turn bound by tags and primer sequences for identifying the accurate sequences.

Could this algorithm identify and recover such haplotypes?

Could the authors use a non-stochastic model to run their code and see how it behaves?  Although it sounds unorthodoxical, this would set an assumption to depict inherently stochastic model.

Assuming that the haplotype frequencies are not equal, how similar would be the haplotype frequencies at a certainty/random?
Validity of the findings: I am bit intrigued by the reason why the authors chose Lake whitefish excepting the reason de tre and the association with "species diversity."   Aren't there any other datasets to measure species diversity?  The gene flow hypothesis could be measured at microbial level as well.

How would this HACSim be associated, for example on rRNA reads which has distinct biological properties as well.
Additional comments: No additional comments


Instruction: You are one of the reviewers, your task is to write a review for the article. You will be given the title of the article, the number of the round in which the article is located, and your order among the reviewers.

Input:
Title: HACSIM: AN R PACKAGE TO ESTIMATE INTRASPECIFIC SAMPLE SIZES FOR GENETIC DIVERSITY ASSESSMENT USING HAPLOTYPE ACCUMULATION CURVES
Review round: 2
Reviewer: 1

Output:
Basic reporting: I am happy and convinced with teh changes rendered by the authors. Thank you
Experimental design: None
Validity of the findings: None
Additional comments: I am happy and convinced with teh changes rendered by the authors. Thank you


Instruction: You are one of the reviewers, your task is to write a review for the article. You will be given the title of the article, the number of the round in which the article is located, and your order among the reviewers.

Input:
Title: CONVOLUTIONAL NEURAL NETWORKS APPROACH FOR MULTIMODAL BIOMETRIC IDENTIFICATION SYSTEM USING THE FUSION OF FINGERPRINT, FINGER-VEIN AND FACE IMAGES
Review round: 1
Reviewer: 1

Output:
Basic reporting: Literature review is just a pile of information, lacking of analysis and induction.
Experimental design: Authors proposed an approach for multimodal biometric identification system using the fusion of fingerprint, finger-vein and face images to overcome the short comings of single biometric and achieve the high-performance accuracy, and explain the validity of the multimodal biometric system by conducting some experiments.
Validity of the findings: Only combining Fingerprint, Finger-Vein and Face Images lack of some innovation, since a lot of multimodal biometric identification system have been proposed (just as mentioned in "Literature review"), authors should state clearly the advantage of combining the three traits, not just pick up 3 traits randomly.
Additional comments: Authors proposed an approach for multimodal biometric identification system using the fusion of fingerprint, finger-vein and face images to overcome the short comings of single biometric and achieve the high-performance accuracy, and explain the validity of the multimodal biometric system by conducting some experiments.

But in fact, only combining Fingerprint, Finger-Vein and Face Images lack of some innovation, since a lot of multimodal biometric identification system have been proposed (just as mentioned in "Literature review"), authors should state clearly the advantage of combining the three traits, not just pick up 3 traits randomly.

Literature review is just a pile of information, lacking of analysis and induction.

in line 192-193, the author wrote " where k indicates the biometric matcher, w1 ,w2 ,……. wk are the weight value over a range of [0,1] and according to the condition and the sum of w1,w2,……,wk is always 1."
since K=3, authors should rewrite the sentence (use 3 replace K, and adjust the presentation of the sentence) to make it more accurate and clearer

in line 170-171, the author wrote "decision trees in order to get a single decision tree. Figure 4 presents an example of separation of classes with the Random Forest. The RF algorithm is given in Fig. 5."
in fact, it is superfluous to present an example of separation of classes with the Random Forest, since Random Forest is well known method. in addition, we cannot find the Fig. 5 the author mentioned.


Instruction: You are one of the reviewers, your task is to write a review for the article. You will be given the title of the article, the number of the round in which the article is located, and your order among the reviewers.

Input:
Title: CONVOLUTIONAL NEURAL NETWORKS APPROACH FOR MULTIMODAL BIOMETRIC IDENTIFICATION SYSTEM USING THE FUSION OF FINGERPRINT, FINGER-VEIN AND FACE IMAGES
Review round: 1
Reviewer: 2

Output:
Basic reporting: The paper address issues of unimodal biometric system by fusion of multiple modalities. Approach seems to be technically good. It is difficulty to forge multiple modalities and hence can be more secure. The literature review needs to be augmented with some of cancellable approach which also impart security of template. these papers needs to be added in literature review

1. "Secure multimodal biometric system based on diffused graphs and optimal score fusion", Volume 8, Issue 4, July 2019, p. 231 – 242, IET biometric.
2.  ‘A review of biometric technology along with trends and prospects’, Pattern Recognit., 2014, 47, (8), pp. 2673–2688.
3. ‘An adaptive bimodal recognition framework using sparse coding for face and ear’, Pattern Recognit. Lett., 2015, 53, pp. 69–76.

Results are sufficient to prove the efficiency of method. I suggest to include the recognition index in results section.

There are number of grammatical errors, Sentence format needs to be improved. Try to write short and concise sentence for better understanding of readers.
Experimental design: Comparison with other state of the art methods needs to included in this section.
Validity of the findings: Seems okay
Additional comments: No additional comments


Instruction: You are one of the reviewers, your task is to write a review for the article. You will be given the title of the article, the number of the round in which the article is located, and your order among the reviewers.

Input:
Title: CONVOLUTIONAL NEURAL NETWORKS APPROACH FOR MULTIMODAL BIOMETRIC IDENTIFICATION SYSTEM USING THE FUSION OF FINGERPRINT, FINGER-VEIN AND FACE IMAGES
Review round: 2
Reviewer: 1

Output:
Basic reporting: The authors has partially addressed my previous comments. Although approach seems to be good. But authors needs to addressed the comments before final acceptance of manuscripts. I strongly feel the comparison with state of the art score fusion should be included in Table 14 as in current submission compared multimodal papers are not very recent. I suggest to included score fusion comparison  in  table 14

Robust multimodal biometric system based on optimal score level fusion model, Expert Systems with Applications, Volume 116, February 2019, Pages 364-376
Experimental design: Experimentation with real multimodal dataset can be included rather that virtual multimodal database.
Some ablation analysis about the proposed method should further strengthen the claims.
Validity of the findings: Seems okay but experimentation with real multimodal database needs to be included.
Additional comments: Experiments with real multimodal database.
Comparison with state of the art score fusion method.


Instruction: You are one of the reviewers, your task is to write a review for the article. You will be given the title of the article, the number of the round in which the article is located, and your order among the reviewers.

Input:
Title: CONVOLUTIONAL NEURAL NETWORKS APPROACH FOR MULTIMODAL BIOMETRIC IDENTIFICATION SYSTEM USING THE FUSION OF FINGERPRINT, FINGER-VEIN AND FACE IMAGES
Review round: 3
Reviewer: 1

Output:
Basic reporting: The author has address most of my earlier comments. The approach can be accepted for publicatin
Experimental design: Sufficient
Validity of the findings: Can be further augmented by comparing with state of the art methods
Additional comments: No additional comments


Instruction: You are one of the reviewers, your task is to write a review for the article. You will be given the title of the article, the number of the round in which the article is located, and your order among the reviewers.

Input:
Title: ROLE CLARITY DEFICIENCIES CAN WRECK AGILE TEAMS
Review round: 1
Reviewer: 1

Output:
Basic reporting: There are reporting issues in terms of style, and presentation of the findings. Overall the text is actually very readable, but there are just a large number of issues that require clarification. The presentation of the results needs to be restructured. Diagrams and tables could help to clarify. Details below.
Experimental design: This is a grounded theory study, and more details are needed to explain the analysis. Details below.
Validity of the findings: There seems to be a lot of speculation based on individual/singular observations. The analysis needs to be elaborated, and the findings need to be restructured, which will help in following the argumentation of the authors. Details below.
Additional comments: This article presents a study of self organising agile teams with a specific focus on the dynamics that emerge as team members get to grips with the roles that they have within the team. The study presents a number of findings, for example, that there must be 'local role clarity' (referring to individuals to understand and accept their role) but also team-wide role clarity, meaning that very team member understands and accept their own roles. If these are not present, then agile teams could break down.

This is a very interesting study drawing on a significant amount of data collected, and the authors seem to have very good access to the teams. There is a very nice level of frankness in the findings, and the study could be a great contribution to the agile literature to understand what makes self-organising teams work.

There are, however, a number of issues in the current presentation. I have a list of issues below, and will summarise the key issues here:

1. The paper presentation, while seemingly a trivial detail, is bothering me quite a bit. The use of very short paragraphs and sometimes sections (as short as one or 2 lines) makes little sense in that it raises a lot of questions that remain unanswered. I have listed a number of comments below regarding this issue, but in short, whenever you make some kind of claim or mention some kind of concept, clarify it, don't leave it up to the reader to do the research, but be complete in your presentation. In addition the use of different fonts and colours to distinguish the concepts, and the very formal referencing structures used to refer to individuals (t2dev1) really break the flow of the paper completely.

2. The actual 'theory' or findings should be reorganised and restructured. I find it very hard to clearly pinpoint how to do that, but it seems to me that the various findings are scattered over a number of sections. I suggest that the authors focus on the key findings such as local role clarity and team-wide role clarity, and take these as the central concepts, and link everything else to those. At the moment there are so many observations that seemingly are somewhat unrelated, or at least that relation isn't very explicit. Would a diagram help perhaps?

3. Restructuring of the methodology section; details are scattered over too many subsections, and sometimes related things are not discussed 'together'. (Details below). Further clarity on the analysis is also needed. Suggesting to look at Hoda's studies and notation to identify concepts.

4. Tone down the conclusions that you draw which are not sufficiently supported by the findings. There seems to be a lot of speculation.

5. There are a number of style and presentation issues (besides the font/smallcaps/colour issues in item 1 above). There is a list of issues at the bottom of this review.

6. Related work: the authors state that there is a lot of work on team work, and then simply state that they only discuss a few. I think the whole related work section could be  reworked, perhaps some restructuring is needed. I would also strongly suggest to refrain from pointing at any of your findings in the related work section (I don't really like it in Section 1 either in the current form, but I can see why one would do that).

Detailed comments follow below. The review comments suggest I be very constructive and positive - please assume this is implicit. I think this study is very interesting, but have a good few issues, and haven't bothered to tell you how good it all is every time.

Some of the paragraphs are very short, and miss elaboration or explanation. For example, paragraph 3, 4, 5, 6, 7 in Section 1 are very short. It almost reads like a novel. Some of the sections area also very short, e.g. 1.2, 1.3. Why have them as separate sections? Suggest to remove those subsections headers and merge all text under a single header "Section 1". As I proceeded reading the paper, I found that short paragraphs is an issue throughout, some as short as a single line. Please don't do that too much.

Paragraph 3: the present work will characterise one major hurdle; please explain here what the focus is of the paper.

Paragraph 4 is unclear altogether. Not clear what 'intermediate outcomes' are, why abstraction is necessary, and how roles are a form of abstraction. Not sure why Hoda et al 2013 are cited here either.

I'm sure there is a formal definition of role in the management literature. Why not draw on that?

You state that roles are fluent and change over time, which is why you do not name or list specific roles. I can't follow that logic. Also, Scrum defines a number of roles. Why not simply refer to those?

You indicate there is a huge literature on team work from management and other fields, but that only a few are considered. Why? There are likely to be very useful insights in that related work, so skipping it simply because it's from other fields isn't really a valid argument in my view. This is a missed opportunity. Please don't use 'grounded theory' as an excuse.

Sec 2.1 already alludes to some of the findings. I suggest to remove those forward references.

Section 2.1 mentions Tuckman's model, but delays discussion of this to the Discussion section. Why?

Section 2.2 reviews some prior literature organised by author. A better way to conduct a literature review is to synthesise by theme. Instead of Author A found ... Author B found ..., it is better to distill the various themes. Authors may or may not be referenced by name as part of the text. Further, the reference to the authors' own work is too short. This article should stand on its own, so include more detail on that previous work, what you found, and how that led (exactly) to the current study.

Please explain what exactly you don't share in terms of the relevance of personality type. What is wrong with it, in your view? Convince me of your position.

You write that Hoda et al examine roles in agile teams such as Mentor, ... ; actually, those were her findings, she didn't go out to examine those roles; they didn't exist as concepts before her study.

You suggest that Hoda's roles are "fixed" but I don't think she actually claims that her roles are "fixed"; she calls them roles, because that's an easy label that people understand, but it is perfectly acceptable to interpret that as "something that people do" - a "role" that people adopt within the team. I don't think that she suggests that these are fixed roles assigned to people within a team. At least, you don't provide any justification for your claim that she considers them as 'fixed' roles (e.g. quotes from her studies).

Section 2.2 is labeled self-organisation, but section 2.3 also discussed self organisation (e.g. "our contribution is how to define and describe roles in general in self-organised teams"). Consider relabelling your sections or combine them.

Section 2.4 is also too short: just a paragraph. If, as you state, reflection is a central theme in your study, then it warrants more detailed discussion. For example, you use the term reflective practitioner as if the readers are supposed to know what that means, but it is well worth describing what this means, and how this has been considered within the SE literature. A few years ago there was a special issue in IEEE Software for example on the reflective practitioner.

Why did you select Charmazian GT? What were your assumptions and factors in the decision to adopt CGT, as opposed to Glaser or Straussian GT? Simply stating a preference that Charmaz's version of the method has your preference because it's more clear (if that is the case to you) will do. (Glaser is a bit ... vague sometimes, and perhaps you found Strauss & Corbin too detailed?).

Further, you state it is suitable to study agile contexts, citing Hoda and others. Why in particular? You seem to allude to it, but you use signalling words that contrast that (i.e. "Furthermore," as if it is another reason?)

Don't simply refer to Barke and Prechelt when referring to the data used in this study. Again, this paper should stand on its own. Pretend as if your 2018 paper doesn't exist from that perspective, though you probably should cite it as it seems relevant.

When would audio recordings be considered inappropriate? Do you mean inconvenient, or not preferred by the recorded participants? Please clarify.

What do you mean by "These written protocols contain as much verbatim language as possible."? What written protocols? What verbatim language? Is this a good thing? (You lost me there).

You mix the discussion of interviews and observations in the paragraph labeled "Observations". Suggest to not do that.

The paragraph on member check and feedback suggests you completed data collection before analysis; this seems to contradict or conflict the practice of theoretical sampling a bit, because the latter suggests that you keep collecting data to fill in the gaps that emerge during analysis. Please also elaborate what you mean by member check; the term hasn't been used in the main text, only the label.

It is not clear why you state that the expert interviews focused on 'concrete' teams; were the interviews with the teams not regarding 'concrete' teams (namely theirs)?

Suggest to summarise the list of teams in sec 3.3 in table.

Why do you state that you restricted yourselves to Scrum teams 'to limit complexity'? You suggest in parentheses that you did not study other agile methods. But why would that be more 'complex'? Sub-universe is a bit flowery language, in my view.

You mix up the data collection (discussion of teams that you interviewed) and GTM practices (section 3.4); please do that in one place (teams were chosen to cover a variety ...). Further you discuss data analysis (MAXQDA) as part of 3.4 as well.

You separate Data collection (3.2) from 3.4, the actual GT practices which include data collection. I suggest to consolidate that as well.

There should be more detail on the data analysis/coding process. Hoda (and others following her example) has done this well, illustrating with examples how concepts were identified. This helps to understand the process you followed. Further you refer to word-by-word coding but also mention incident by incident coding. Did you use both? More details please.

Section 3.5, first line as a separate paragraph. Following paragraphs are also too short. Please just combine 3.5 with 3.4, or at least present longer paragraphs. The one-liners are not always self-explanatory; what does it mean that "The appearance of a concept name indicates a statement is grounded in the data."? What concept name; what do you mean by appearance, even? I can't follow this.

You write "a quote is often (if not always) provided ..." But you know whether it's often or always, because you wrote this paper. Be more precise.

I don't understand the reference / metaphor in footnote 3 on page 6 to the Facade layer pattern. Suggest to simply remove.

Sec 4.1, 2nd para, "This narrative" - which narrative? "This" is now a dangling pointer, because it's a separate paragraph - that's why paragraphs should not be split up!

Sec 4.1, "So here is the set of concepts ..."; this isn't great style, to be honest. Don't just list the concepts and announce them as "here is the list of results": tell a story instead.

I can't follow what the finding is in Sec 4.1 You present S1, S2, as definitions, perhaps?
I suggest to explain and describe these concepts, indicating what they refer to  and illustrate them with events/quotes/other indicators/evidence that led you to the coining of these concepts. I also strongly suggest to remove the numbers/prefixes S1.2-, S1.1-, as it breaks the flow.

In sec 2.3 you suggested that you did not consider personality type research, but in the first quote of Sec 4.2 you refer to this very concept: "he has a different personality". This is confusing. Why do you not consider personality relevant then, but still use it in a quote to illustrate your findings?

Section 1 introduced the key findings as F1, F2, etc. These labels are also used in the findings section, but I think these labels should not be used in the presentation of the results, but could only be used when summarising the results. In other words, remove any reference to Fn, but theorise instead in the presentation of the result. The presentation of the key findings can then be done in a summarising section.

Some quotes are simply not very illustrative. Why did you select the quote on "meditation", for example? What does this illustrate?

I cannot follow why the quote by t2dev1, page 7, second last quote, is labeled "effective feedback". I think it's pretty bad, to state "therefore I cannot respect you as a developer". I'm surprised nobody quit yet.

Please refrain from speculation, e.g. page 8: Now that t2dev2's role is clear, knowledge transfer from t2dev1 will likely become a lot faster and the overall team capability may become better more quickly. You conclude very firmly that "the problem is a local role clarity" - but that's just your theory, really - a hypothesised mechanism at work. What evidence do you have for this at all?

A main problem I have while reading the results is that there is a lot of stuff going on, but it's really hard to maintain a clear (clean) record in my head of what it is you actually have learned/found through this study. No doubt that this is partly due to the presentation style (small caps, different colours), the use of too formal systematisation of references to individuals (t2dev1), and the very short paragraphs and section sometimes which break the flow and concentration, but there is a different issue. I appreciate the difficulty of writing grounded theory, but there seem to be so many findings that are not clearly discussed in separate sections. Some sections eg 4.4 are too short. One sentence and one quote, that's not enough to convince me of your argument that, 'once role clarity is achieved, technical expertise becomes less important'. That's a single observation, but there is no additional evidence presented for that. The labelling of that section (4.4) is ("having non-perfect technical expertise") also not very descriptive.

Language & Style

we need to abstract a bit --> rephrase "a bit".

I'm not convinced that typesetting the concepts identified through GT in small caps is a useful practice; it distracts from reading. When typesetting "Roles" for example, "Role" is in small caps/red, and the s is black in normal font; this also doesn't help. References to other roles like sm, po are also hard to interpret, breaking the flow of reading. Strongly suggest to just spell it out in normal font.

Amount of meetings --> Number of meetings. Use "amount" to refer to inseparable things like water and beer. Use Number to refer to distinguishable, concrete items.

The term "data" is plural.

Suggest to remove unnecessary words, e.g. "In total, the present research" --> This research. Simpler is better.

Sound recordings: probably audio recordings. Sound is what you hear in the disco, hopefully, and preferably not in an interview. You wouldn't be able to understand people.

The referencing of teams as t1, t2, etc.: I suggest to simply state Team 1. Further I think the mathematical labelling of t4dev2 etc. to refer to team 4, developer 2 is a bit much. Does it matter for the reader whether it was dev2 or dev3?

4.2, Comparatively obvious - do you mean relatively obvious? Even that is a bit vague. (Comparatively - with what?)

The use of "I" in Sec 4.2, you writing as a researcher to "quote" participants (?) is a bit vague. The changing of "I" and "we" (as authors) throughout the paper is confusing and should be removed.

Reduce informal language such a "a bit" (also listed above) and "So the findings are", "the point is" etc. Please also try to remove "glue" language like "now we will explain". Don't announce it mid-text, just do it.

"Indeed," cannot be used at the start of a new paragraph. Indeed, (no pun), the word is often overused (in general, not necessarily this paper), and is a signalling word that highlights a previously stated argument. (Top of page 8)

Please refrain from using "variables" in the text, such as "someone else's role R.", so hat you can refer to "R" as someone else's role later. This is not readable. (Same for "team member A").


Instruction: You are one of the reviewers, your task is to write a review for the article. You will be given the title of the article, the number of the round in which the article is located, and your order among the reviewers.

Input:
Title: ROLE CLARITY DEFICIENCIES CAN WRECK AGILE TEAMS
Review round: 1
Reviewer: 2

Output:
Basic reporting: This paper presents a theory on the role of clarity deficiencies to help the workings of agile teams. Using a Grounded Theory Methodology on five agile team the study concludes that lack of role clarity, both at an individual and team level, creates friction in team, hampers everyday work and can also lead to employee turnover. The study concludes a high need for role clarity by whatever means it takes.

Overall, the paper is easy to understand with great attention paid to details. There are some places though where additional information can help.

Related work section feels more like a mix of background and related work. In this case, it might be worth presenting it the same way. In related work section, the relevance of each sub-section isn’t quite clear. Subsection ‘Teamwork in general’ actually doesn’t talk about teamwork in general but about specific studies which lead us to current work. At some point why these subsections exist at this place is not clear. For instance, what is the purpose of subsection ‘reflection’? What is the bigger message that is intended to be conveyed here?

Similar issues exist in the results section. Each subsection presents a distinct aspect of the observation, but it is not evident what the big picture is. It might help to present an overview of the factors that play a role in getting role clarity before delving into details. For instance, subsection ‘having non-perfect technical expertise’ is actually a positive use-case of role clarity. I suggest revisiting the title of each subsection to better convey the story.

At some places, concepts from related studies are poured in without explanation. For example, Constructing Grounded Theory variant of Charmaz and MAXQDA. It will help the reader if a brief explanation is provided for the terms and concepts borrowed from the literature.
Experimental design: The objective to help teams self-organize in an agile set-up is a relevant problem and building a theory around it seems appropriate. The experimental design to construct theory is presented in great details and all necessary precautions seem to have been taken. There are a few details that raise doubt.

Section 3.2 reports that "small talks built trust and lowered the hurdle to...". How are such things substantiated? Was this a part of the plan?

I also miss details at places on reaching conclusion. For instance, in section 4.1 it is reported that “it appears that TEAM WIDE ROLE CLARITY is as important for a team as is a sufficient level of technical expertise.” Details on how this statement is reached and similar statements will greatly help in appreciating the results. In its current form, not all claims seem to be backed by evidence.
Validity of the findings: The paper builds a theory based on discussions and observations of teams. To what extent do these results hold? Do all teams agree to the theory? Do they have an alternate explanation?

Discussion section introduces new findings not presented in the results section. For example, section ‘How to do it?’. I am not sure how these results are derived or how generalizable these results are? It will help to either make these questions a part of study design or refrain from describing it.

The paper reports that no tangible positive outcomes were seen when the concepts were applied in practice. What potentially explains it? Does that mean that the proposed framework is of no use?
Additional comments: The paper aims to present a theory for helping teams self-organize. But it seems that the bigger message is lost in the great details presented in the paper. The paper talks about too many things but fails to organize it in a way that helps reader get a bigger and coherent message.


Instruction: You are one of the reviewers, your task is to write a review for the article. You will be given the title of the article, the number of the round in which the article is located, and your order among the reviewers.

Input:
Title: ROLE CLARITY DEFICIENCIES CAN WRECK AGILE TEAMS
Review round: 2
Reviewer: 1

Output:
Basic reporting: Please general comments; sometimes too much "casual language".
Experimental design: N/A.
Validity of the findings: No issues.
Additional comments: This revision is a big improvement on the initial submission. I think this paper present some interesting findings, and would support publication - I think it is converging to an acceptable state. I have a number of small issues, suggesting a minor revision.

In sum, my comments refer to style and presentation
1. Style: the paper still exhibits some casual use of language, or language that one would use in daily conversation rather than what one would expect from an academic paper.
2. Presentation: the paper still has some sections that are too heavy on bullet points, suggesting this is a slide deck; I think this really needs to be polished into proper text rather than trying to summarise everything in bullet points.

Abstract

"They should invest whatever it takes"

The "whatever it takes" phrase might be too open-ended...
Perhaps something more neutral (and less dramatic) might be more appropriate academic style.

Introduction
- why do you need "(supposedly)" ? Can it be removed?
- misplaced closing parenthesis in l.55
- paragraph l.56-59: short sentences that don't really flow. "There is research aboutself-organized teams in general." - this is a bland statement - link it to your study here.
- l. 60 since a few years, *we have worked* (not: we work)
- l.61 no need for "own" - you already state "our"
- If you state "in our first article", the reader might expect that you have a 2nd aritle (is that this one); it seems to signal something as earlier you stated "in our previous work". But "previous work" and "first article" are the same I think?
- Still think that subsections 1.1 and 1.2 don't need a subsection title - in other words, suggest to remove those headers (but keep the text). They seem tagged on to the main part of Sec 1.

Method
- picked -> selected
- happen to have different levels of maturity -> seems a "coincidence" and lack of deterministic research design. Suggest to remove "happen to".
- l.94 "will be required" -> suggest to remove future tense such as this throughout the paper; there's no need for it; instead, starting after the comma on l.94: ", which requires a thorough analysis so as to reveal recurring structures across teams."
- l. 95: "some longitudinal data" -> be more specific; this sounds a bit ad hoc and hesitant ("some data").
- suggest to replace Helena Barke with "the first author".
- l.101 -which we will call -> which we refer to as (again, no future tense); same sentence: period missing.

- Data collection WAS done. Missing period after Table 1^3
- this research -> this study

Sec 2.3
- we will now explain -> we now explain (no future tense)
- l.161 we will ... ->remove will. (I won't comment on this issue again, please check the whole paper for this)

l.169 - what do you mean by "phenomena"; why do you need to make explicit that you do not focus the discussion "around people or teams"?

Section 3
- what are the labels R1-Accept, R2-Expertise, and R3-Acting again?
Why you need these labels? Are these attributes of a "role"? Make this more clear.

- I'm visually oriented, but am aware not everybody is. Does it make sense to add a diagram that captures the overall results as presented in Sec 3.1? Concepts can be boxes, theorised links as arrows?

- What does "No. 2" and "No. 3" mean (l. 216 and later)

- language such as "The point is" (l. 252) sounds like daily, verbal language rather than written language that belongs in an academic paper. Suggest to rephrase. The use of "So" at line 347 ditto. The style has certainly improved w.r.t the first submission, but on these occasions the authors still write as one would talk in a conversation.

Section 4
Suggest to elaborate briefly on each of the 7 "findings" in 4.1
Don't link them together as in the case of 3 and 4 with the word "but"

4.2:
l. 433 "In the following Section"
- do you mean THIS section?
- don't capitalise Section if you just use the word "section"w without referring to a specific numbered section (cf. "in this section" vs. "in Section 3")
- as mentioned, no future tense (no "will")

I'm usually not a fan of delaying a presentation of related work until so late in the paper, but recognise cases where this can be useful; in such cases it needs to be contrasted very clearly with the current study's findings. Is it worth adding a table that juxtaposes related work with this study, if you do decide to keep the related work until after the presentation of the study instead of as section 2.

It seems to me that Section 4.3 would be better labeled as Implications for Practice. Again, a table might be useful that juxtaposes the specific findings and implications that result from each of them. Given your findings what should one now do in practice (and what might the implications for research be? Does your research open up new questions?)

Section 5
Strongly suggest to rid of the bullet points; make this normal text.
Don't use the term 'proof'; you provide evidence not proof.

Section 6
Don't use an enumeration of findings or conclusions. Write text. This is not a powerpoint presentation published in PeerJ. This is an academic paper.

Personnell is spelled as personnel


Instruction: You are one of the reviewers, your task is to write a review for the article. You will be given the title of the article, the number of the round in which the article is located, and your order among the reviewers.

Input:
Title: ROLE CLARITY DEFICIENCIES CAN WRECK AGILE TEAMS
Review round: 2
Reviewer: 2

Output:
Basic reporting: Paper is well articulated and precise. The message of the paper comes out quite well.
Experimental design: Paper is clear about the method and scope of the results is well defined.
Validity of the findings: Results seem sound and not overclaiming.
Additional comments: This version of the paper brings the core message clearly and with scientific accuracy. The outcome of this study is a useful contribution towards building a theory on the dynamics in self-organizing teams.

My only suggestion is correcting the typing errors in the paper.


Instruction: You are one of the reviewers, your task is to write a review for the article. You will be given the title of the article, the number of the round in which the article is located, and your order among the reviewers.

Input:
Title: MACHINE LEARNING OF SYMBOLIC COMPOSITIONAL RULES WITH GENETIC PROGRAMMING: DISSONANCE TREATMENT IN PALESTRINA
Review round: 1
Reviewer: 1

Output:
Basic reporting: On English:
Some typos in the abstract: “we us all other dissonances” -> “we use all other dissonances” and some shorthand also that could be made clearer.  Abstract ll 14-16 could be broken into multiple sentences; and “with each other and *with* manually programmed rules”  16-17: “We chose genetic programming for our machine learning technique”  I’m not sure what “as positive examples” means in l. 23 – “as examples used to train the algorithm in a particular dissonance category”?  This seems redundant.  The next lines 24-25 seem more important: “We also trained the algorithm in many types of negative examples such as…”  l. 27 “category *of* suspension”

The cumbersome reading of the abstract almost made me not agree to review the paper; I hope others don’t decide not to read it for the same reason.

The introduction’s writing and later is stronger and better edited.  But there are still errors that make reading hard – careful rereading by a native speaker is crucial:
35:	methods OF algorithmic composition
144-5:	“the algorithm better leaves”  -- “an algorithm that leaves a few complex dissonance categories undetected is stronger than one that wrongly marks notes…” or something like that?
181:  “To be at the save side” – “To be on the safe sade”

On Literature references, sufficient field background/context:
The relevant literature is addressed and cited well.  However, the original encoders of the Palestrina corpus should be mentioned as well as the editions from which they are encoded.  (There are errors in encoding and in parsing that are known to people involved in the original project and the music21 project; I doubt they are sufficient to affect the end results however)

On Professional article structure, figures.  Raw data shared.:
Raw data is shared -- in abundance.  Version of software used should be noted.  Some issues w/ figures (discussed below) but in general very professional.

Figure 1 includes in addition to “p.t.” “n.t.” and “s.” also “p.t.s”.  This could be read as “p.t.” followed by “s.” but the presumably correct reading as multiple passing tones is not included

On Clear defs of all terms, theorems and proofs:
Very clear in general.

A pass, but some revision needed
Experimental design: The research question is well defined, and how this work on genetic programming will advance the field of computational music research is clearly stated and well argued (well done!).

There are no ethical concerns raised by the paper.

38–42 are key and important parts of the paper (and might be highlighted in the abstract): rules that can be studied by humans are important for acceptance of this work in the discipline of music theory (as an interdisciplinary paper should aim for).  Cuthbert/Ariza/Friedland 2011 p. 391 (http://ismir2011.ismir.net/papers/PS3-6.pdf), among others, discuss the importance of understanding how a decision was made in the context of classification of musical features.  Very important and good point.

148 It is unclear what the custom algorithm (after chordification) does in cases where the voice leading of the chordified version is unclear.

153-159: this seems a VERY smart way of working.  Applause.

The description of handling of suspensions (160-162) is still unclear to me though.  After chordification, the dissonant pitches would already be tied to a previous pitch.

163-166: There are some cases in the Palestrina corpus of incorrect clefs (namely treble8vb as treble, causing fifths to become fourths).  The authors are cautioned to check their corpus manually against the original encodings; all of which are on IMSLP.  The authors should also be sure to be on the latest release of music21, as prior versions did not necessarily parse properly the (partially incorrectly encoded) krn files where the number of voices in Agnus II is fewer than in Agnus I, but the original (usually five-voice) structure is restored in Agnus III.  This can make for a large number of dissonances.

211: were melodic intervals given in semitones, as diatonic steps, or (perhaps best) in Hewlett’s base-40 interval encoding?  This information is needed to replicate.  (I’m guessing from Table 1 later that it’s diatonic steps)

253: are random values that match the positive examples excluded from set 3?  Or are they too few to care about? (ah, this is mentioned in 282-83.  I think they should be removed).

267: again the interval should be described as diatonic or semitonal?

285–325: my knowledge of this part of the paper is low.  It looks good to me, but please weigh other reviewers’ comments more highly for this section (Genetic Programming)

317-18: “The fitness assigned … is 10 times its accuracy…plus 0.001 times the size of the tree” – should this be minus 0.001 times the size of the tree?  It seems to reward bloated trees?
Validity of the findings: 132 – unclear why the further dissonance categories are irrelevant; they certainly do occur in the Palestrina Mass corpus, and at this point in the paper it is hard for the reader to know why rejection by the algorithms makes evidence irrelevant.  (ah, l. 168 gives the answer to what the corpus is; that such dissonances do not occur is possible, but not certain)

171-175: not enough information is given here to verify the authors’ claim that the detection works rather well.  Can this data be provided (or see next comment)

183–186: a reader would be interested in a list of these occurrences of dissonances that cannot be automatically classified by the custom algorithm.  They would be of interest for music theorists and Renaissance scholars in addition to future researchers who may want to improve the algorithm.

196–199 is the most problematic passage I have yet read: the “key” detection of a Palestrina mass movement is likely to be very faulty as the Krumhansl-Schmuckler key determination algorithm (which should get citation) and the Sapp weights were designed for tonal music of the common practice period (1650–1880) and not for this period.  I believe I have seen, in other forums, the authors’ search for an algorithm that would classify Renaissance mode better, and I believe they can say that no such algorithm yet exists, and they should be able to satisfy me that the KS algorithm is the best thing available.  But the argument needs to be made.

215 – what does it mean to normalize the pitch class of the dissonance?  does rotating pitch classes so, say G = 0, affect the results?

221 – the total number of points in each cluster are noted below, but the total number of points recognized as outliers should be noted.  (it’s at 246)

228-233 is very well explained.

Figure 2 is hard to read – the blue bars could be made more prominent and the dots bigger as well.  A note that each pink dot represents one run could be added.

The bigger issue though is that C5 and C6 are included in the diagrams but have not been explained in the text yet.  What are they?  [Ah!  From the scripts I discover they are the cases in l. 249; this should be explained]

Figure 3 – why not plot #s 0 0 0 0, 1 1 1 1, etc. instead of abstract figures that I need to return to the caption often to read?  The switch from 0-1 to 0-10 score is also confusing.

Figures 5 and 6 should explain again what cluster number they refer to – we’re going back and forth between cluster numbers and verbal descriptions.

Returning to 293–300 in light of Figure 5–6:  295: is the “dissonant” note necessarily dissonant?  l. 263 seems to imply it could be simply a middle note.  Figure 5 line 1: I read the right side as duration(i+3) and was wondering why the duration of the note three notes later was being taken into account.  More space or something to show that it is duration(i) + 3 would help.  It took me some time to note that “dissonance status” was NOT an input variable, and that we are only looking at linear (melodic) segments.  If this is true, then the results are even stronger, but either way, this point should be made very clear in the article.

355 – it is unclear here and in table 2, what “can be” means – does this mean that “the generated rule/solution allows the note to be an eighth note” or “the received rule allows the note to be an eighth note but the generated solution does not”?  The direction of the deviation is important.

The Qualitative evaluation (339-356) is quite cursory and there is not enough information given here to evaluate the accuracy of the results, I’m afraid.

374 – I’m not sure what an “easy beat” is.  Is this a mistranslation for “simple beat” (which would also need explanation), or a integer number beat?  It is not an English language term in UK or US.

381: “We leave that to the reader” – the reader does not know the grouping – does the ^ at the beginning of the line constrain the entire line or not?  So are lines 2-3 to be read as
“accentWeight >= 1 AND ( 2 < duration(i) OR duration(i-1) >= 2 )”
or “(accentWeight >= 1 AND 2 < duration(i)) OR duration(i-1) >= 2”?   When writing something that could be seen as arrogant such as “we leave that to the reader” then the reader needs to be given a fighting chance at figuring it out.

Discussion area is strong and well written.

After reading the paper I discovered the electronic Appendices, which give the paper a much more positive result  for reproducability, etc.  Very strong.  Should be referenced in the text where they appear (the “Resulting rules.pdf” is especially interesting).  The version of music21 used (and other software) should be mentioned in the interest of reproducing.
Additional comments: Paper needs some minor revisions noted above before accepting, but it is very close and I anticipate that the next version will be ready for publication.

The research area is of interest and the research design is well done, and well written.  Professional, solid, and will have impact.

I am choosing not to publish my review publicly, but authors may know my name: Michael Scott Cuthbert (MIT).


Instruction: You are one of the reviewers, your task is to write a review for the article. You will be given the title of the article, the number of the round in which the article is located, and your order among the reviewers.

Input:
Title: MACHINE LEARNING OF SYMBOLIC COMPOSITIONAL RULES WITH GENETIC PROGRAMMING: DISSONANCE TREATMENT IN PALESTRINA
Review round: 1
Reviewer: 2

Output:
Basic reporting: Good, but I have some misgivings about music terminology and some unusually translated terms: I'll paste a few paragraph from my letter here:

This paper should be accepted, but there are several minor revisions that should be undertaken. There are three main areas in which I would encourage revisions include: 1) tweaking some of the language usage such that it conforms to English musical terminology, 2) adding some citations and engagement with similar and related research, 3) making a few aspects of the presentation more clear.
	First, there are some terminological snafus in the article. The term “easy” beat is incorrect: I believe the author means “weak” beat. The phrase “without getting noticed much” in line 125 is awkward and conceptually suspect. I believe the author means that these dissonances are less structural insomuch as they are heard as elaborating the surrounding consonances. In lines 127-128, the author might cite Jeppesen (or even the venerable J.J. Fux, 1725) to illustrate these dissonances because there are, in fact, some disagreements about categories of dissonance in this musical style. (There’s no reason to get into this debate in this paper, and it can be sidestepped by simply citing these standard dissonances.) In line 165, I believe the author means “whole note” rather than “whole tone,” since they’re talking about duration at this point. Additionally, it should be recognized that this music was not in 4/2 in the original manuscripts, but rather was probably signed with either a half circle or a cut half circle to indicate a duple grouping of the constituent breves. In other words, the 4/2 barring – and therefore the music21 metric weights – are modern anachronisms. (This fact doesn’t change the study’s results: it just should be acknowledged.) On line 199, the author means “tonic” not “root”  (chords have roots, pieces and keys have tonics). Additionally, on this front, the abstract needs a careful edit. As it currently stands, it is difficult to read and uses unidiomatic words and turns of phrase.
Experimental design: It's good.
Validity of the findings: I would encourage the author to connect the findings to other existing research. Again, here are the relevant paragraphs:

The paper cites no other work machine learning on musical corpora or automated harmonic analysis. To my mind, even though no one has done this particular approach to dissonance treatment, the author should still cite other research into modeling musical harmonies. I’ll include a short bibliography below, which could provide a springboard for a more comprehensive literature review. Additionally, the author argues that their contribution is most valuable to automated musical composition research. I’d argue that this modeling is also valuable to research into statistical learning. You could argue that this article models the experience of a learner who tries to construct a series of cognitive templates through trial and error until settling on generalizable rules. The author might consider adding some discussion of these implications.
A few other discussions could be tightened, clarified, or removed. First, Table 2 doesn’t do a lot of work and takes up a lot of space. To my eye, all the table illustrates is that there exists very little variation within the clustered rules. I think the table should basically be deleted and perhaps replaced with another sentence (in the Quantitative Evaluation section) describing this lack of variation and the caveats present in the table’s footnotes. If the author believes the table to be important enough to stay in the article, more discussion is necessary to indicate why. Additionally, Figure 5 should be talked through step-by-step from top to bottom (see lines 360-380-ish) rather than the piecemeal approach used now, and the value i on which the dissonance actually occurs should be explicitly stated (I believe the dissonance occurs on i, and not i+1 or i-1, but I had to think about that for a bit).  Also, explanations of computational specifics were often not comprehensible to me, but that’s probably because I’m a music theorists and not a computer scientist– I’ll defer to the editors and other reviewers on the assessment of those topics.
Additional comments: No additional comments


Instruction: You are one of the reviewers, your task is to write a review for the article. You will be given the title of the article, the number of the round in which the article is located, and your order among the reviewers.

Input:
Title: MACHINE LEARNING OF SYMBOLIC COMPOSITIONAL RULES WITH GENETIC PROGRAMMING: DISSONANCE TREATMENT IN PALESTRINA
Review round: 2
Reviewer: 1

Output:
Basic reporting: My concerns have been addressed. I'm glad this paper is moving toward publication!
Experimental design: My concerns have been addressed
Validity of the findings: My concerns have been addressed
Additional comments: My concerns are addressed. I congratulate the authors on a job well done.


Instruction: You are one of the reviewers, your task is to write a review for the article. You will be given the title of the article, the number of the round in which the article is located, and your order among the reviewers.

Input:
Title: STUDYING THE IMPACT OF CI ON PULL REQUEST DELIVERY TIME IN OPEN SOURCE PROJECTS—A CONCEPTUAL REPLICATION
Review round: 1
Reviewer: 1

Output:
Basic reporting: Basic reporting of the submitted article is in general good. The text is clear and concise. One place where the authors can potentially improve the write-up is in the reference section. I am not sure if this formatting is supported by PeerJ but as a reader, it feels odd finding references work.

I see one typing error (see line number 258). Please revisit the document for typing errors.
Experimental design: The objective of the paper is presenting a conceptual replication on the effects of adopting continuous integration on pull request delivery time. This study builds on the original work by Bernardo et al. (2018). This paper replicates the original study and includes a factor that can potentially explain the observation. This paper further tests the observation on a new set of projects to test the usefulness of conclusions.

The study seems to be meticulously designed with potential threats sufficiently addressed in threats to validity. The choice of statistical tests seems appropriate too.
Validity of the findings: The study replicates the original study testing the findings of Bernardo et al (2018). The study furthers adds a factor that can potentially explain pull request delivery times. Using original projects as well as new projects (including controls) the study suggests that the results of the original study holds and more generally predicting PR delivery time is difficult.

As a conceptual replication, study does justice in stepwise replicating the original study but also providing deeper context into the relations.
Additional comments: No additional comments


Instruction: You are one of the reviewers, your task is to write a review for the article. You will be given the title of the article, the number of the round in which the article is located, and your order among the reviewers.

Input:
Title: STUDYING THE IMPACT OF CI ON PULL REQUEST DELIVERY TIME IN OPEN SOURCE PROJECTS—A CONCEPTUAL REPLICATION
Review round: 1
Reviewer: 2

Output:
Basic reporting: This paper presents a replication study analyzing the impact of continuous integration (CI) on pull request delivery time.
First, the paper presents an exact replication of previous work, and then it presents a conceptual replication as well as describing the generalizability of the results.
Overall, the paper asks and answers five research questions on this topic.
The paper does a good job of situating this study amid previous work, both in discussing the research under replication, as well as other related work.
The article was well structure, and contains links to the supporting data, which is available on GitHub.

Unfortunately the paper does have a few minor presentation flaws:
Page 1, Line 28 in the abstract: comparabe -> comparable
Page 5, Line 186, "and merge more PR" -> more PR's
Figure 10 is hard to read, and could be much easier to parse if it were expanded to fill the available space.
Experimental design: This work is a well designed replication experiment.  By producing an exact replication as well as a conceptual replication, there is much higher confidence in the findings.
The conceptual study also clearly address the concern of the previous work, which had not accounted for the potential that projects might naturally speed up at delivering over time, independent of CI.

One minor improvement would be if the paper were to describe the criteria used to determine if a project were worth being filtered out.  Currently the paper states that "we also discarded toy projects, tutorials, and other similar projects not intended to be deployed" It would be helpful if they stated how these projects were determined to be in this category.
Validity of the findings: This paper does a great job of answering the research questions posed in a clear and data-driven way, while also acknowledging the potential threats to validity.
The findings are backed up by the data, and the data is readily available to be examined and used by future researchers.
Additionally, this paper provides a valuable replication of exiting work, and by mostly validating the original findings, they provide much higher confidence.
Additional comments: Overall this paper provides a valuable replication study to evaluate the impact of CI on code delivery.


Instruction: You are one of the reviewers, your task is to write a review for the article. You will be given the title of the article, the number of the round in which the article is located, and your order among the reviewers.

Input:
Title: STUDYING THE IMPACT OF CI ON PULL REQUEST DELIVERY TIME IN OPEN SOURCE PROJECTS—A CONCEPTUAL REPLICATION
Review round: 1
Reviewer: 3

Output:
Basic reporting: The paper is very well written and structured. The provided background
and used terminology are sound. Finally, the tables and pictures are very
instructive and help to convey the ideas discussed in the paper.
Experimental design: The paper aims to perform a conceptual replication of the work performed by
Bernardo et al, 2018. The identified gaps are (i) the need for replication
studies in empirical software engineering and (ii) the opportunities for
improving the methodology employed by us (i.e., Bernardo et al. 2018).
The methodology applied in the paper is very well motivated and sound.
Validity of the findings: The overall validity of the paper is sound and contains some opportunities for improvement (as expected in a scientific paper).
Additional comments: I am glad with the opportunity to review this paper as it presents a conceptual
replication of our work (Bernardo et al, 2018). The analyses presented in this
work are not only reassuring for me, but also provide learning opportunities
for future research. I thank the authors for that.

Regarding the design choices of the study, I think the paper can be mainly improved in two aspects:
1) the collection of the NOCI projects and 2) the correlation & redundant analyses before
introducing the 'come-in-time' metric.

- NOCI projects) Sizílio et al (2019) observed that some projects which do not
  use Travis-CI do actually use other tools for implementing CI. However, the
  CI implementation of these projects is not as easily identified as checking
  whether a project has builds on the Travis-CI server. Indeed, Sizílio et al (2019)
  sent a little survey for GitHub developers asking  whether they have used other
  CI implementations (i.e., other than Travis-CI) to confirm that a given
  project was indeed NOCI. I would suggest the paper to discuss
  this issue, probably in the threats to validity section.

- New metric) I think the paper should address the possibility of correlation between the
  'come-in-time' metric (i.e., the new metric) and the existing metrics. For example,
  the newly proposed metric is, to some extent, similar to the queue-rank metric.
  I wonder whether if these metrics happen to share a correlation. If
  that was indeed the case, the explanatory power of the metrics in the
  models may change.

Minor comments:

- Ln 363, the paper states that 17 out of 86 projects have acceptable
  R-squares. However, it should be 17 out of 87, no?

- It would be nice if the newly collected dataset (i.e., new CI
  projects, NOCI projects, and new metrics) could be shared with the community.

References:

Gustavo Sizílio, Daniel Alencar da Costa, and Uirá Kulesza.
An Empirical Study of the Relationship between Continuous Integration and Test Code Evolution. In Proceedings of the 35th IEEE International Conference on Software Maintenance and Evolution (ICSME 2019).


Instruction: You are one of the reviewers, your task is to write a review for the article. You will be given the title of the article, the number of the round in which the article is located, and your order among the reviewers.

Input:
Title: MAKING SIMULATION RESULTS REPRODUCIBLE—SURVEY, GUIDELINES, AND EXAMPLES BASED ON GRADLE AND DOCKER
Review round: 1
Reviewer: 1

Output:
Basic reporting: The paper is well-written. The general structure is fine as well, only within certain sections I would prefer more structure. All the survey results are discussed in Section 2.3, which is a massive wall of text. I would recommend adding some subsections, for example, a separate section for the results related to peer reviewing.

Intro and related work are ok. I would add a brief summary about the main findings at the end of the introduction.

Figures are relevant and also well described, but in my opinion, the figures are way too small. When printed the labels are hard or near impossible to read. Make sure to use proper font sizes.

Raw data was supplied and seems proper.
Experimental design: The research conducted is in scope of the journal, the research questions are relevant and meaningful. The survey part including the methodology definitely improved compared to the previous version. However, there is still missing information:
-) How did the authors come up with both the survey questions and the possible responses for single-choice and multi-choice questions? Predefined answers increase the chance for introducing researcher bias, which is also not reported as a potential threat in the threats to validity section. The added threats to validity section needs to be more thorough. I would recommend the authors to check out related literature. The already mentioned Runeson et al. “Case study research in software engineering: guidelines and examples” is a good starting point.
-) There is no information how open / free-form questions were analyzed. Was this based on open coding?
-) There is no information whether all questions were mandatory for survey participants. We do not learn how many participants completed the survey, for example, did all 125 participants complete the survey, or are there some partial results as well?
-) We do not learn how the authors treated numerical inputs. Were some ranges (e.g., in years) provided upfront? Or were these ranges and categories introduced later on?

Learning about subpopulations is helpful, I really appreciate that we now have this information. However, I’m wondering how ‘relevant’ those 11 bachelor and master students are for a survey on the reproducibility of research results, especially for those parts covering reproducibility in the context of peer review.
Validity of the findings: What I’m missing throughout the entire paper (it is only mentioned on the last page) is a discussion about the newly introduced artifact evaluations as part of some computer science conferences/journals. The goal is to make research results reproducible and badges awarded for papers also provide some sort of credibility / recognition. So, the question is whether these new processes are enough, or do we still need to think about ways for including artifact evaluation during peer review. I would have appreciated to have some discussion when reflecting on the survey results and proposing the general guidelines in Section 3.

Regarding the third guideline in Section 3, even though source code is provided and not just binaries, the code can still contain malicious parts.

“The average amount of time participants would invest in making software of others to reproduce results was 15.94 hours, neglecting 5 outliers who would spend 100 or more hours.” I’m a bit confused why those 5 outliers are excluded from the average?
Especially, since the average time spent for making the own research results reproducible is 24.4 hours and 6 participants that invest between 100-250 hours seem to be included? Without having the exact numbers, it seems that both averages could be on a very similar level? I recommend the authors to clarify why they exclude / include certain data points.

I appreciate that the authors summarized the comparison of their examples in a table. However, I have the impression that categories and ratings are very subjective. Compared to the other approaches, why is the effort for sharing Docker images ‘moderate’? Once you have the image (and we are only considering the sharing effort) it is just a single command to upload it to Dockerhub. How is this more effort than pushing for example a Node.js package to NPM?
Furthermore, based on the description of the categories it is hard to tell the difference between “Required software” and “Effort for installation and execution”. I recommend the authors to go over their categories again, make them clearer and not overlapping, and then do the ratings/assessments again and try to justify ratings/assessments.
Additional comments: The authors conducted a study among researchers with the goal to identify the importance of reproducibility of research results and current issues for making software artifacts available and reusing software artifacts provided by others. Further, the authors provided a set of guidelines for reproducibility and discussed and compared existing tools for sharing and reproducing software artifacts.

Even though the paper improved compared to the previous version, there are still many major flaws detailed in the comments that need to be addressed before it can be considered to be published in this journal.

Further comments:
Related work improved, but there is no statement on how this paper goes beyond the findings reported by others. What is the research gap the authors are trying to fill?

The authors need to be careful with the Docker terminology. There is a difference between Docker images and Docker containers. You do not share containers, you share images. Images become containers when they run on the Docker engine. Furthermore, also when using Dockerfiles to specify the ingredients and the recipe on how to build an image, the source code and all the other artifacts that are needed for building an image need to be available, either as part of the host image or (finally) on the machine where the image is built. Consequently, when “sharing” artifacts with Dockerfiles, it is (in most cases) not just the Dockerfile that needs to be shared.

Gradle is not necessarily bound to specific project structures. This is only the case when using the default settings, but Gradle is highly customizable and thus the configuration can be adapted to support building projects with different structures. The authors should consider that when assessing Gradle.

Section 2.3: “These responses indicate the need to provide open source methods of sharing software artifacts free of charge”. Maybe this goes down to some lack of rigor in the survey design, but there are various platforms out there that allow sharing artifacts completely free of charge and also supporting large file sizes. For example,  Zenodo (https://zenodo.org/) and figshare (https://figshare.com/).

Minor:
First sentence sounds a bit clumsy. Maybe “The paper addresses two [research] questions related to reproducibility within the context computer science research”?
Line 33, “paper addresses two questions related to”, maybe “…two research questions…”?
Line 34, “… with the the aspects…”


Instruction: You are one of the reviewers, your task is to write a review for the article. You will be given the title of the article, the number of the round in which the article is located, and your order among the reviewers.

Input:
Title: MAKING SIMULATION RESULTS REPRODUCIBLE—SURVEY, GUIDELINES, AND EXAMPLES BASED ON GRADLE AND DOCKER
Review round: 1
Reviewer: 2

Output:
Basic reporting: - The language is ok, and the article flows without any specific issues. The correct use of the Saxon genitive should be revised (e.g., Docker's vs. Docker concepts)

-  The references to the literature are sufficient. The authors identify the work of Cito et al. as very closely related to theirs. Accordingly, I suggest them to better pinpoint the differences between the two.

- Please, check the reference style. Avoid brackets in (Author (Year)) format.

-  I was not able to find a link to the raw data from the survey.

- The introduction is quite weak and requires more context before introducing the research questions.

- In the Introduction the authors state "Reproducing results of published experiments, however, is often a cumbersome and unrewarding task. The reason for this is twofold" But then only one reason is reported, "First, some fields ..."
Experimental design: The survey design is detailed enough but suffers from a few shortcomings.

RQs
- The first research question implies that for the sample reproducibility may not be important. - Is this a good sample for surveying best practices and tools (i.e., second RQ)?
- Please, specify what are the "related research areas of computer science."

Sample
- Please, report (even a ballpark) of the total amount of people you may have reached in your survey (not only the ones who answered).

- The characterization of the respondents does not take into account overlaps (e.g., one can be a Ph.D. student and work as a researcher within a project).
- What is the area of research of the remaining 13% of the sample not reported in Section 2.2?
- It would be beneficial to provide a geographical characterization to check whether cultural factors could impact the findings.

- The authors later report that 35 out of 125 respondents never published a software artifact. This information should be reported earlier. Most importantly, this should be elaborated in reference to my previous comment regarding the validity of the sample.

Results
- Notice that time one is willing to invest in reproducibility of a project depends on the size of the said project. Accordingly, a characterization of the projects the respondents work on would be beneficial to interpret this result.

- The language is "leading." Avoid adjectives as "interestingly" or "only 28 out of 125." Let the reader decide what is interesting and whether 28 is "only" a small proportion of 125 (for me, it is not).

- No statistical analysis was performed. Hence, the deducing facts about the world based on these numbers only is quite subjective. I recommend performing statistical analysis (at least correlations) to support some of the claims. Moreover, avoid statements like "There was no signiﬁcant difference determined between different research environments." as no significance test was performed.
Validity of the findings: - Data from the survey is not provided (or at least I could not find it) and no statistical test is performed.

- I am not sure how the guidelines and tools follow from the results of the survey. The former are quite general (which is fine), the latter should be better contextualized in specific usage scenarios (and if possible, discipline related to CS)

- Please explain what is a Dockerfile.
Additional comments: - Consider moving inline links to footnotes for better readability.


Instruction: You are one of the reviewers, your task is to write a review for the article. You will be given the title of the article, the number of the round in which the article is located, and your order among the reviewers.

Input:
Title: MAKING SIMULATION RESULTS REPRODUCIBLE—SURVEY, GUIDELINES, AND EXAMPLES BASED ON GRADLE AND DOCKER
Review round: 2
Reviewer: 1

Output:
Basic reporting: The paper is well-written. Introduction and related work definitely improved compared to the previous version. The motivation of the study and how the conducted research differs from related work is now much clearer.

Figures are relevant and also well described, but in my opinion, some figures are still too small (e.g., Fig 1, 3, 6, 7). The labels are easier to read compared to the previous version, but there is some space for improvement. Consider an increased font size.

The structure mainly stayed the same. I still have the impression that Section 3.3 (formerly 2.3) requires some sub-sections or paragraphs emphasizing the covered topics. I’m also not a fan of the title “What Researchers Want”. Why not having some smaller paragraphs such as relevance, effort, peer review, publication & security, or similar? This would break this wall of text and provide more structure.

Raw data was supplied and seems proper.
Experimental design: The research conducted is in scope of the journal, the research questions are relevant and meaningful. The entire experimental design and the methodology of the survey part improved and threats to validity were updated accordingly. The authors addressed most of the comments sufficiently.

I’m not overly convinced on the relevance of bachelor or master students on a survey regarding the reproducibility of research results. The argument provided in the author’s response letter “we expect the opinion and knowledge of the subject of today’s students being relevant for the next generation of researchers” does not fully convince me, especially considering the fact that only a small percentage of students conduct PhD studies afterwards. For me it would be ok to include these students as long as there is somehow an indication that they are or were involved in research projects. However, answering questions on reproducibility and how much time and effort you would invest in making your own results reproducible and (re-)using the tools of others without having first-hand research experience seems problematic.
Validity of the findings: The authors clarified and addressed most of the raised concerns appropriately.

I’m only missing information on how the authors decided on cluster sizes (e.g., page 5: “Summarizing the results in clusters…” and “Summarizing through clusters…”) and why do for example the cluster sizes on the time willing to invest for reproducing results of others and on the time for making own results reproducible differ?
Additional comments: The authors conducted a study among researchers with the goal to identify the importance of reproducibility of research results and current issues for making software artifacts available and reusing software artifacts provided by others. Further, the authors provided a set of guidelines for reproducibility and discussed and compared existing tools for sharing and reproducing software artifacts.

The paper definitely improved, most of the issues are addressed. As described, there are some minor comments I would like to see addressed before publishing the paper in the journal.

Further comments:
For the previous revision I pointed the authors to the correct usage of Docker terminology. Most of this image vs. container mix-ups have been fixed, but there are some more occurrences, e.g., page 10 lines 364 and 373, page 13 lines 517 and 519, page 15 line 607. I recommend the authors to conduct thorough proof-reading to fix these as well.

I don’t see much value in the added correlation analysis. I have the impression that it is pure number reporting and it is not really used to support some of the author’s claims such as the mentioned “We could not identify significant differences between different research environments”. I would just recommend the authors to be careful with the language and avoid the usage of terms such as “significant” as long as there is no underlying statistical test.

Minor:
Page 7, line 262: “A multiple-choice question…”, this sentence seems incomplete.
Page 7: I would recommend merging the pre-defined answers (starting at line 265) with the frequencies they were selected (line 272). Further, I would remove the listing of the additional five “other” answers in detail and just provide a categorization (e.g., dependencies).


Instruction: You are one of the reviewers, your task is to write a review for the article. You will be given the title of the article, the number of the round in which the article is located, and your order among the reviewers.

Input:
Title: MAKING SIMULATION RESULTS REPRODUCIBLE—SURVEY, GUIDELINES, AND EXAMPLES BASED ON GRADLE AND DOCKER
Review round: 2
Reviewer: 2

Output:
Basic reporting: The language was improved and, overall, the text flows better.
See my annotated PDF for language and comprehensibility improvements.

The introduction section has been improved considerably, which improves further understandability and scoping of the paper.
Experimental design: My previous request to perform statistical analysis was carried out to a good extent and with sufficient information to fairly evaluate the validity of the results.
Validity of the findings: Data is provided. The findings are used to answer the research question (although not formally, see PDF annotations).
Additional comments: I would like to thank the authors for their work.
Please, see my comments in the PDF for further, minor, improvements.


Instruction: You are one of the reviewers, your task is to write a review for the article. You will be given the title of the article, the number of the round in which the article is located, and your order among the reviewers.

Input:
Title: 3D TEXTURE-BASED FACE RECOGNITION SYSTEM USING FINE-TUNED DEEP RESIDUAL NETWORKS
Review round: 1
Reviewer: 1

Output:
Basic reporting: - English needs to be improved. Some examples where language could be improved are as follows:
    - 188: First letters of terms with abbreviation  should be capital letters
    - 207: Gamma to be written as a symbol
    - 268: Singular verb with plural subject
    - 58, 243, 273-275, 304: Inappropriate usage of capital letters
    - 292-300: Inappropriate usage of past tense
    - 416, 487-489: Inappropriate usage of punctation mark
    - 229-235: Need to use inline latex scripts

- Figure 2 is not cited in the main text.
- Figure 2 caption refers to the term "Face Recognition Grand Challenge" first time, hence its abbreviation (FRGC) must appear in parenthesis near from it.
- Figures 8, 9 and 10 are parts of a multi-part figure. Please arrange them as they are to be published, label each part with an uppercase letter and submit them in one file.
- When starting a sentence with a citation, use 'Figure 1'.  For example in line 166, 258.
- Please include figure numbers in filenames, for example 'Figure 1.png'

- Thanks for providing all figures with a high resolution however your figure titles need to be more descriptive. For example:
    - Figure 1 title is "3D face pre-processing.png".  Using file name with its extension makes comprehension difficult. In addition preprocessing is only one phase of this figure which causes ambiguity.
    - Figure 4 title is: "The HOG features of various person samples". Please consider to use "3D face images" instead of "person samples".
    - Figure 5 title is: "The structure of ResNet in different layers". I suggest to use the term "architecture" instead of "structure". Also mentioning different layers is unnecessary.
    - Figure 6 title is: "The Fine-Tuned Resnet Neural Network". Inappropriate usage of capital letters.
    - Figure 8, 9 and 10's title is: "The accuracy rate of different layer numbers in the fine-tuning ResNet architecture." Please indicate the number of layers explicitly; 50, 101 or 152. Reader can not understand which figure part is for which number of layers.

- Please include table numbers in filenames, for example 'Table 1.png'.
- Table 1 is redundant of the information included in Figure 8, 9 and 10.
- The "Method" column in Table 2 contains reference numbers but these reference numbers  can not be interpreted.
- Authors provided the source code however raw dataset, preprocessing code and pre-trained models are missing.
Experimental design: - Research question is defined as achieving high recognition accuracy on a large scale 3D face image dataset in the presence of gradient vanishing problem. However your experiments are on FRGC dataset which contains 4000 images and is not a large scale dataset. On the other hand you use ResNet which has no gradient vanishing problem. I suggest that the knowledge gap being filled in your paper is the usage of HOG features as the input representation for 3d image recognition.

- I could not grasp the parts of method through B3 to B4. For example line 230 says "we evenly divide the vectors into nine intervals"  then line 247 talks about small and large intervals. What does "category information" refer to?

- Method's reproducibility requires the followings:
    - Gamma value in equation 1 is not defined.
    - Which pre-trained weights did you use for fine tuning? From which link can we download the pre-trained weights? Are they pretrained on HOG features of 3D face images?
    - Line 400 says that fully connected layer can classify more than 1000 types. But line 355 says that there are 466 people in the dataset.
Validity of the findings: No comment
Additional comments: No additional comments