Instruction: You are one of the reviewers, your task is to write a review for the article. You will be given the title of the article, the number of the round in which the article is located, and your order among the reviewers.

Input:
Title: IDENTIFICATION OF HIGH-EFFICIENCY 3′GG GRNA MOTIFS IN INDEXED FASTA FILES WITH NGG2
Review round: 1
Reviewer: 1

Output:
Basic reporting: No comments.
Experimental design: Lines 20 and 68-69.    The Farboud & Meyer paper only worked with nematodes so it is not yet clear the 3’GG motif will be as effective in other systems.

Lines 40-46 .     The commonly used CRISPR/Cas system, i.e. Cas9,   is a Type II system.  The Types I and III have different components.

Line 54-55.   It is not correct to say “the PAM site for a Type I CRISPR is an NGG”.  First Cas9 is Type II, not Type I.  Second, Type II CRISPR systems (Cas9) are found in different species of bacteria, and the Cas9 protein in different species has evolved different PAM motif requirements.  The commonly used S. pyogenes Cas9 has the NGG preference.  Other species, i.e. N. meningitidis, use different PAMs.   (GATT in that species).

Line 85.  Wouldn't it be pretty easy to upgrade the tool to report uniqueness of the gRNA?  at least report yes or no?

Line 87-88 and 170.    BLAT is not very useful for checking off-targets, since off-targets can contain mismatches and BLAT does not work well for detecting CRISPR targets with even one or a few mismatches.   They are too short and BLAT was designed to detect near-perfect matches of this length or longer.  BLAST is appropriate as long as the parameters are set to allow mismatches to a certain degree within the target.

Line 160.   How was gene overlap defined - is this within the coding regions of exons, or merely within the entire transcription unit?  In other words does the tool return gRNA sites in both exons and introns of the gene?  This was not clear at all, but it is very important, as large genes may have many intronic matches but few exonic matches and I am not sure how this was performed.  This needs to be made very explicit.
Validity of the findings: No comments
Additional comments: This computational tool is fine and conceptually very simple.    I agree that is may be of good use to researchers who are dealing with model organism sequences/genomes for which CRISPR targets have not been pre-cataloged.

I do think it may be premature to regard the “3’GG” motif as a guarantee of high efficiency targets, as the Farboud & Meyer paper only involved nematodes.   This motif has not yet been shown to be consistently valuable in other organisms or systems.   Although it may well be, I don’t think the Farboud paper really well explained how this motif was shown to be efficient in previous papers, although they do make this claim. Therefore I believe the author should introduce this note of caution into the manuscript.

Lines 20 and 68-69.    The Farboud & Meyer paper only worked with nematodes so it is not yet clear the 3’GG motif will be as effective in other systems.

Lines 40-46 .     The commonly used CRISPR/Cas system, i.e. Cas9,   is a Type II system.  The Types I and III have different components.

Line 54-55.   It is not correct to say “the PAM site for a Type I CRISPR is an NGG”.  First Cas9 is Type II, not Type I.  Second, Type II CRISPR systems (Cas9) are found in different species of bacteria, and the Cas9 protein in different species has evolved different PAM motif requirements.  The commonly used S. pyogenes Cas9 has the NGG preference.  Other species, i.e. N. meningitidis, use different PAMs.   (GATT in that species).

Line 85.  Wouldn't it be pretty easy to upgrade the tool to report uniqueness of the gRNA?  at least report yes or no?

Line 87-88 and 170.    BLAT is not very useful for checking off-targets, since off-targets can contain mismatches and BLAT does not work well for detecting CRISPR targets with even one or a few mismatches.   They are too short and BLAT was designed to detect near-perfect matches of this length or longer.  BLAST is appropriate as long as the parameters are set to allow mismatches to a certain degree within the target.

Line 160.   How was gene overlap defined - is this within the coding regions of exons, or merely within the entire transcription unit?  In other words does the tool return gRNA sites in both exons and introns of the gene?  This was not clear at all, but it is very important, as large genes may have many intronic matches but few exonic matches and I am not sure how this was performed.  This needs to be made very explicit.


Instruction: You are one of the reviewers, your task is to write a review for the article. You will be given the title of the article, the number of the round in which the article is located, and your order among the reviewers.

Input:
Title: IDENTIFICATION OF HIGH-EFFICIENCY 3′GG GRNA MOTIFS IN INDEXED FASTA FILES WITH NGG2
Review round: 1
Reviewer: 2

Output:
Basic reporting: No comments
Experimental design: This work to identify unique 3'GG motif sites to use in targeted genome editing is of clear utility and value to the community.  The ngg2 tool, if improved to address the issues below, would be a good contribution in this area.

1.  Comparison to existing tools
  - lacks specific references, tool descriptions
  - uninformative explanation of "less common, non-model" organism issue which defines the novelty of the current work (ngg2)
  - no clear evidence / discussion of how ngg2 provably addresses issues with non-model genomes where others did not

2.  Uniqueness of sites
  - not clear why ngg2 omits this, given the emphasized critical importance of identifying unique sites?  This step is more work than the what ngg2 currently performs
  - manuscript includes results for uniqueness, possibly using blast/blat; no details / parameters provided
  - genome sequence is already available and of interest in most applications, and redundant sites are easily collated (hash, map data structures)
  - since no results are presented for the "allowN" option, very difficult to gauge its utility and impact on uniqueness and runtime

3.  [Python] code design / implementation / evaluation
  - relies on FASTA sequence data, previously indexed in a specific (samtools) format
  - an undo limitation:  (a) requires tool/pre-processing dependencies, and (b) does not allow pipelining
  - simply using community library (biopython) or even a discussion of why this was not done, would be expected
  - no runtime results/discussion presented; important given application / evaluation on whole genomes
Validity of the findings: How would 3'GG sites disrupt a protein-coding gene if they are located are in introns/UTRs?  Needs clearer description of "overlap" criteria
Additional comments: One additional minor suggestion for the manuscript :  Figures 1 and 3 seem extraneous and less informative versions of the data in tables - might move Suppl. Table 3 to main text as it is an important use-case for using the tool


Instruction: You are one of the reviewers, your task is to write a review for the article. You will be given the title of the article, the number of the round in which the article is located, and your order among the reviewers.

Input:
Title: AN ALGORITHM FOR DISCOVERING LAGRANGIANS AUTOMATICALLY FROM DATA
Review round: 1
Reviewer: 1

Output:
Basic reporting: No comments
Experimental design: No comments
Validity of the findings: A formatting suggestion: Summarize results for test systems pertaining to final model (the (m,p) parameters, number of free parameters, computational time & number of iterations required) in a table.
On that note, there doesn't seem to be much analysis on the differing convergence rates for different systems - what conditions cause this-or-that test system to converge quicker?
Would also like to see some comments on how the computation time scales for 2-dimensional or higher dimensional systems (since every test system presented has been 1-dimensional) - is this approach for finding the Lagrangian computationally feasible for larger systems?
Additional comments: No additional comments


Instruction: You are one of the reviewers, your task is to write a review for the article. You will be given the title of the article, the number of the round in which the article is located, and your order among the reviewers.

Input:
Title: AN ALGORITHM FOR DISCOVERING LAGRANGIANS AUTOMATICALLY FROM DATA
Review round: 1
Reviewer: 2

Output:
Basic reporting: No Comments
Experimental design: No Comments
Validity of the findings: No Comments
Additional comments: The idea of using the principle of least action seems novel and is interesting. The paper is well written, structured and complicated physics and algorithms are explained well albeit at a high level of abstraction.

I was left wondering a little about the scalability of the first approach using the Nelder-Mead optimisation. The examples, and the excellent results even with noise, seem to be dealing with situations with very small values of m and p. Could you say something about how this method would scale to more complicated examples? Also, what sorts of values of m and p might be found in typical real-world examples?

In the second, GA, approach the use of syntactic expressions as the genetic material is appropriate but I do not think that that is novel, as such. Perhaps some more references to related previous work is needed here.

Minor points... Some of the specifics of the modifications to EL_D to construct the score function seem a little arbitrary.  For the non-physicist reader the discounting of certain terms from the Lagrangian because they can have "no physical significance" could be explained.


Instruction: You are one of the reviewers, your task is to write a review for the article. You will be given the title of the article, the number of the round in which the article is located, and your order among the reviewers.

Input:
Title: A FRAMEWORK FOR CUT-OVER MANAGEMENT
Review round: 1
Reviewer: 1

Output:
Basic reporting: This paper provides a governance structure for IT-related projects, especially when the projects are ready to roll out to production. This paper is based on the author’s consulting experiences in the Department of Testing and Release Management at the Otto Group, which is headquartered in Hamburg, Germany.

My impression of the paper is that the author has done enough work to warrant a publication, especially the author does provide a methodology for rolling out projects so that the transition to new systems can be done smoothly.

However, I do have one complaint. The paper does not include a real-world case. (Maybe I missed it, I do not know.) For papers of this sort, real-world cases are important because the theory proposed by the paper can only be tested empirically, not mathematically. Hence, a successful application of the methodology is important to convince readers that the methodology is valid.
Experimental design: See previous comments.
Validity of the findings: Can only be validated by real-world cases.
Additional comments: Please provide a real-world case of successful application of your methodology.


Instruction: You are one of the reviewers, your task is to write a review for the article. You will be given the title of the article, the number of the round in which the article is located, and your order among the reviewers.

Input:
Title: A FRAMEWORK FOR CUT-OVER MANAGEMENT
Review round: 1
Reviewer: 2

Output:
Basic reporting: The paper clearly defines a method for live deployment of a project. It does a good job of emphasizing the need for applying an exact, formal methodology as opposed to simply using best practices, so that the scientific method can be applied and and data can be used to back up our idea that the method is "better".
Experimental design: The paper demonstrates an adequate case study that applies this methodology and the ideas are well supported. Because each project is unique, it would be difficult or impossible to exactly reproduce the results. However, it could be tested on other projects to see if inclusion of cut-over requirements in the ITIL management model show more timely delivery.
Validity of the findings: The paper used good statistical methods that support the validity of the papers findings
Additional comments: No Comment


Instruction: You are one of the reviewers, your task is to write a review for the article. You will be given the title of the article, the number of the round in which the article is located, and your order among the reviewers.

Input:
Title: THE RISE OF CHROME
Review round: 1
Reviewer: 1

Output:
Basic reporting: The paper reports a study for the comparison of three of the most popular browsers for desktop PCs: Google Chrome, Internet Explorer and Mozilla Firefox by considering the versions released in the last 6 years. Three factors have been considered in three separated parts of the sudy:
- The scores obtained by executing well-known benchmarks;
- The measured start times;
- The number of features provided by each browser, in a set of 25 relevant features selected by the authors. To measure the relative importance of these features, the authors proposed a complex statistic approach based on the judgements of 254 users.
 A brief discussion that tries to obtain some lesson learned is placed in the last paragraphs.
Experimental design: The authors measured the benchmark scores for any release of any of the three browsers and reported the results in figures. No further comparative analysis on these data are reported.
The complete set of parameters considered by these benchmarks is not reported, so I am not able to know, in example, if and how parameters such as the memory usage has been taken into account and in what measure. The confidence in the experimental data depends on the validity of the benchmarks, that is not discussed in this paper.
As regards the third part of the experimentation, the selected features are analyzed in order to evaluate only wins (i.e. features that a browser provided early with respect to the other ones) and losses (i.e. features that a browser provided later with respect to the other ones): it seems a quite reductive analysis. On the other hand, a complex statistical approach has been adopted to weigh the importance of the features for the users. The adopted methodology could be described in a more formalized and detailed way, giving space to the threats to construct validity related to this methodology.
In conclusion, the experimental design lacks of:
-	A more detailed description of the set of aspects that the benchmark takes into account;
-	A discussion of the threats to the validity;
-	A more convincing explanation of why the set of three experiments that have been carried out is able to provide a complete judgement regarding the user perceived quality of the three browsers.
Validity of the findings: As pointed out in the Experimental Design section, the lack of a Threats to Validity subsection is a great limitation with respect to the assessment of the validity of the findings.
The insight provided in the discussion sections (5.2 and 5.3) are intuitively acceptable but they are not inferred from the data.
I think that the experiment is interesting for historical reasons but its contribution in terms of lesson learned is very uncertain and limited.
Additional comments: Section 2.2 is not of great interest: in this section data from StatCounter.com are cited to show the use of its browser in recent years, but without relevance with respect to the other analysis reported in the rest of the paper.
I do not find the “Insight for R” software package: please give a reference for it.
The Opera browser is introduced only in section 5.2. Why it has not been considered in the rest of the analysis?
As regards the third part of the experimentation, a historical evaluation of the score of the features provided by the browsers over time could be reported, on the basis of the evaluated importance scores.
Modern browsers offer the possibility of being integrated with plug-ins, which often are widespread, as in the case of AdBlock. The presence of these plug-ins is able to threat the validity of some of the “wins” and “losses”, since in many cases you consider a browser as not having a feature when it was actually available even before it was natively incorporated into the browser.
The study is quite interesting but it is only able to give a view of the history of three of the most popular browsers, but if fails in terms of lesson learned that can be useful for the future. The findings provided in the discussion sections (5.2 and 5.3) are intuitively acceptable but they are not inferred from the described experimentation.


Instruction: You are one of the reviewers, your task is to write a review for the article. You will be given the title of the article, the number of the round in which the article is located, and your order among the reviewers.

Input:
Title: THE RISE OF CHROME
Review round: 1
Reviewer: 2

Output:
Basic reporting: This paper proposes a study of performances and feature from most widespread browsers. The performances are evaluated by means of available benchmarks whereas
the feature set was based on a survey on the importance of 25 major features.
The structure of the paper is appropriate.
Experimental design: From the point of view of experimental design i think that both section 3 (technical performance assessment) and 4 ( feature selection) should be improved.

In section 3 the performance assessment is performed only using javascript benchmarks and none is said for a lot of important aspects (machine setups, configuration of nodes with respect to Operating System running the benchmark and so on).
There are a lot of aspect that are not even taken into account:
- caching efficiency
- page reloads
just to name few. My opinion moreover is that this section needs an improvement to study the influence on the proposed results of external factors that influence the provided results.

For what regards section 4 the feature selection i think that the classification of feature is too coarse. The survey it should be structured taking into account kind of user and applications in order to better qualify the scopes of browsers usage by classes of users.
Validity of the findings: In general i think that the overall structure of the paper is poor from a technical point of view: there are no research questions that are formulated and the analysis is only performed to gather data without any research statement behnind. My hint to authors is to improve on this: what research questions they are trying to answer? what is the final goal of the paper ?   This is not clear, if the goal is to assess that chrome has become the most widespread browser due to technical superiority on both performance and feature sides the paper  should be better restructured highlighting adequately the research questions and taking into account marketing issues (that are informally and qualitatively discussed but have a great impact on the final browser adoption statistics)
Additional comments: No comment.


Instruction: You are one of the reviewers, your task is to write a review for the article. You will be given the title of the article, the number of the round in which the article is located, and your order among the reviewers.

Input:
Title: APPLICATION OF GRAPH THEORY TO THE ELABORATION OF PERSONAL GENOMIC DATA FOR GENEALOGICAL RESEARCH
Review round: 1
Reviewer: 1

Output:
Basic reporting: Supplementary Table 1 is vital for conclusion of this paper. But there is no description of how to derive this table within the paper. Could the author explain more about it?
Experimental design: Is this a robust method to analyze another dataset? Multiple testing data is required.
Validity of the findings: 1. How does the proposed method deal with potential confounders? data fuzziness and dentical by state (IBS).
2. The design itself make sure that the TU is linked to all the other individuals. But is it true from the observed data? Does TU have shared haplotype larger than 2Mbp with everyone?
3. Could the author compare predictions of the proposed method to that of Y chromosome based method? At least they can compare the paternal grandfather lineage predictions.
Additional comments: This paper is  target specific, concise and clear. It solves an interesting problem with simple Graph Theory. However, the author should do some comparison with previous methods to emphasize its unique advantages and apply it onto multiple datasets to prove its general applicability.


Instruction: You are one of the reviewers, your task is to write a review for the article. You will be given the title of the article, the number of the round in which the article is located, and your order among the reviewers.

Input:
Title: APPLICATION OF GRAPH THEORY TO THE ELABORATION OF PERSONAL GENOMIC DATA FOR GENEALOGICAL RESEARCH
Review round: 1
Reviewer: 2

Output:
Basic reporting: The paper proposed a method to extract genealogical information at different levels by adjusting the threshold of shared genome. The paper presented an interesting topic, but failed to provide and compare with state-of-art research and advancement in genealogical study.

Section 3 claims that the analysis must be performed using statistical techniques robust enough to sustain unavoidable uncertainties. But the rest of paper does not apply or prove such techniques.
Experimental design: The analysis starts from 8 generation distance, and then loose the condition to 9 and 10 generation distances. Why choose 8 as the starting point? Is this choice applicable to all cases?
Validity of the findings: No Comments.
Additional comments: The method used in the paper is threshold-based filtering on similarity matrix, without serious links to graph theory.


Instruction: You are one of the reviewers, your task is to write a review for the article. You will be given the title of the article, the number of the round in which the article is located, and your order among the reviewers.

Input:
Title: APPLICATION OF GRAPH THEORY TO THE ELABORATION OF PERSONAL GENOMIC DATA FOR GENEALOGICAL RESEARCH
Review round: 2
Reviewer: 1

Output:
Basic reporting: No Comments
Experimental design: No Comments
Validity of the findings: Due to the unavailability of data, the author can not independently validate their methods. This is the major concern.

The choice of threshold on the links between the different individuals is very important. But there is no clear standard to choose it.
Additional comments: No Comments


Instruction: You are one of the reviewers, your task is to write a review for the article. You will be given the title of the article, the number of the round in which the article is located, and your order among the reviewers.

Input:
Title: APPLICATION OF GRAPH THEORY TO THE ELABORATION OF PERSONAL GENOMIC DATA FOR GENEALOGICAL RESEARCH
Review round: 2
Reviewer: 2

Output:
Basic reporting: The majority of my concerns have been addressed except the use of term Graph Theory. However, this is a debatable issue and should not affect the acceptance.
Experimental design: No Comments
Validity of the findings: No Comments
Additional comments: No additional comments


Instruction: You are one of the reviewers, your task is to write a review for the article. You will be given the title of the article, the number of the round in which the article is located, and your order among the reviewers.

Input:
Title: A REDUNDANCY-REMOVING FEATURE SELECTION ALGORITHM FOR NOMINAL DATA
Review round: 1
Reviewer: 1

Output:
Basic reporting: No Comments
Experimental design: No Comments
Validity of the findings: No Comments
Additional comments: In this paper, the author proposed an algorithm named as RedremovingMRLR to select features for nominal data. My main concerns include:
(1) I think the novelty of this algorithm is the improvement of traditional NMIFS. Please highlight your unique contributions.
(2) In experiments, the computational time of the algorithms should also be compared. The author should analyze the computational complexity at least.
(3) There are some typos in this paper. For example Ref. [16]. Besides, there are also some related works should be cited. For example, for feature extraction, “Stable local dimensionality reduction approaches” and “Multiple view semi supervised dimensionality reduction” are two important literatures. For feature selection, I suggest the author to refer “Joint embedding learning and sparse regression: A framework for unsupervised feature selection”


Instruction: You are one of the reviewers, your task is to write a review for the article. You will be given the title of the article, the number of the round in which the article is located, and your order among the reviewers.

Input:
Title: A REDUNDANCY-REMOVING FEATURE SELECTION ALGORITHM FOR NOMINAL DATA
Review round: 1
Reviewer: 2

Output:
Basic reporting: No Comments.
Experimental design: No Comments.
Validity of the findings: No Comments.
Additional comments: This paper proposes a nominal-data feature selection method based on mutual information . The writting of the paper is very clear.  The paper is well organized, and is very easy to follow. The idea in the paper is nice, and it could be accepted with minor revision.

Minor comments:
1. The definition of || in Eq.1  and Eq.4 should be clarified and the * in  Eq.2 should be deleted.
2  The font size of data set in table 1should be the same.
3  The size of ...(Y_1, Y_2, ... Y_9) used in Experiment 1.
4  Fig. 1a, 1b shows the classification results .... should be Fig. 1a, and Fig. 1b. show (page 11).


Instruction: You are one of the reviewers, your task is to write a review for the article. You will be given the title of the article, the number of the round in which the article is located, and your order among the reviewers.

Input:
Title: A REDUNDANCY-REMOVING FEATURE SELECTION ALGORITHM FOR NOMINAL DATA
Review round: 2
Reviewer: 1

Output:
Basic reporting: No Comments
Experimental design: No Comments
Validity of the findings: No Comments
Additional comments: The authors have addressed all my comments and I think it is ready for publication.


Instruction: You are one of the reviewers, your task is to write a review for the article. You will be given the title of the article, the number of the round in which the article is located, and your order among the reviewers.

Input:
Title: A REDUNDANCY-REMOVING FEATURE SELECTION ALGORITHM FOR NOMINAL DATA
Review round: 2
Reviewer: 2

Output:
Basic reporting: No Comments
Experimental design: No Comments
Validity of the findings: No Comments
Additional comments: No Comments


Instruction: You are one of the reviewers, your task is to write a review for the article. You will be given the title of the article, the number of the round in which the article is located, and your order among the reviewers.

Input:
Title: MINING KNOWN ATTACK PATTERNS FROM SECURITY-RELATED EVENTS
Review round: 1
Reviewer: 1

Output:
Basic reporting: The article deals with a really interesting problem. Authors propose a new data mining system that employs text mining techniques to reduce the analysis time, increase the quality of the reports, and automatically build correlated scenarios.

I have some concerns about the introduction section. From my personal point of view, this is quite extensive. At the same time, it does not deal with some important points like Data Mining. Thus, it is my understanding that the whole introduction section should be rewritten, describing the problem, describing data mining, denoting how data mining could be useful for solving the problem, etc. As for Figure 1 and its description, it could be more appropriate for the background section. Introduction section should introduces the problem and how to solve it.

I do not really understand what [Mitre] means (line 38 and 104).

As for the related work section (and also the introduction section), it is quite large. Noted that the problem description does not start till page 8 (from a total of 21). All of this could cause a lack of clarity in what the goal is and how it is solved.

References need to be revised. Most of them are from 90s. The reader could also find some references from 1986, or even from 1960. A research work should deal with a current problem, so references 50 years ago cannot be included.

Definitions 2 and 3 should be improved. Starting from Definition 2, the "set of words" needs to be defined. Is it contained in D? According to Definition 3, the distance function requires a better description. Maybe an example could help since it is not clear how one could measure the distancia between and attack event and an attack step. Finally, the inverse of a distance should be defined similarly to "distance function" does.

Line 308, authors state that the stemming process is achieved by using optionally the Porter algorithm, the Lancaster algorithm and the Regex-driven algorithm. Nevertheless, no citations are available for these algorithms. The reader feels confused in this line and has to go back discovering that it was not defined previously.

For mining the maximal frequent sequences, the authors state that they have chosen an algorithm coded by following the Apriori-like method. At this point, I wonder whether this algorithm is the best option for a text mining problem. Apriori-like algorithms suffer from both computational and memory requirements. Since the goal if to deal with a hihg number of words in an efficient way, the reader needs to know more about it and how the authors have solved this issue. Besides, there is no execution time analysis within the manuscript.

Line 393, there is a "¿". The same occurs in line 395 and 396.

Line 396 and 397, theres is a "Error: Reference source not found"
Experimental design: The experimental design seems to be well described, but it lacks of clarity in some aspects. For instance, since the authors use an Apriori-like algorithm, the hardware used for the experimental study is not appropriate (to the best of my knowledge). From my background in Pattern Mining and Associaiton Rule Mining, an Intel Core i3 with 4GB RAM could be enough just for some specific datasets (small ones).

An analysis of the computational time is required, specially if the authors state that the aim of the proposed system is to "reduce the analysis time"

Datasets should be publicly available, or at least, they should be described well in terms of complexity.
Validity of the findings: No commets
Additional comments: The work is interesting but it requires to be significantly improved. I expected much more from the experimental study, which is shorter than the related work section.


Instruction: You are one of the reviewers, your task is to write a review for the article. You will be given the title of the article, the number of the round in which the article is located, and your order among the reviewers.

Input:
Title: MINING KNOWN ATTACK PATTERNS FROM SECURITY-RELATED EVENTS
Review round: 1
Reviewer: 2

Output:
Basic reporting: The paper proposes a process to relate attack patterns with security events that are triggered by specific software.

The topic is interesting. However, it is necessary that the understanding the paper are improved:

- Authors should specify clearly the work carried out.  Time and again, they say that “they propose a new data mining system”, what  a new data mining system is? This is a very broad definition to define their work and it is used in abstract, introduction, conclusion, ....

- There are several comments that cause confusion. For example, authors comment that they use data mining and text mining, the main difference between both process is the structure of patterns. However, the process/steps are similar. Then, they comment that use KNN classification algorithm, but without determining the class, they only want to find the most related events according to one distance. In this case, why do they talk of classification method?. In general, authors describe different general areas, but their work is not limited in no of them. Finally, if the finality is to show the user the top-K related events, in this case, they could revise the area of recommend systems.

It would be necessary that authors restrict well the work that they carry out and focus the rest of paper in this area (related work, proposal, experimentation, ….).
Experimental design: In experimentation section, concretely section 6.2, authors comment that increasing the maximum distance, the number of neighbours increases sensibly. However, if the number of neighbours is delimited to 10, this restriction would select only  the 10 most closeness and there have been no difference although the maximum distance was largest.

It would be convenient that authors concrete this information.
Validity of the findings: No comments
Additional comments: Authors should specify if they have developed an new application or they use a specific software. Really, there is not a novel proposal in the different steps of the process and there are a lot of software that it is possible to use. It would be convenient that authors should add information about this topic.

In section 4.3, both section 4.3.2 and 4.3.3 specify that it is used a binary vector, but in this case it would be numerical, would not it?.

It would be necessary to review the symbols in equations from line 391 to 397.


Instruction: You are one of the reviewers, your task is to write a review for the article. You will be given the title of the article, the number of the round in which the article is located, and your order among the reviewers.

Input:
Title: MINING KNOWN ATTACK PATTERNS FROM SECURITY-RELATED EVENTS
Review round: 2
Reviewer: 1

Output:
Basic reporting: In the new version of the paper, all of my previous comments have been considered carefully.  I think the paper can be accepted
Experimental design: No Comments
Validity of the findings: No Comments
Additional comments: In the new version of the paper, all of my previous comments have been considered carefully.  I think the paper can be accepted


Instruction: You are one of the reviewers, your task is to write a review for the article. You will be given the title of the article, the number of the round in which the article is located, and your order among the reviewers.

Input:
Title: MINING KNOWN ATTACK PATTERNS FROM SECURITY-RELATED EVENTS
Review round: 2
Reviewer: 2

Output:
Basic reporting: In general, I am satisfied with the review of paper in many points. However, there is one point that it would be convenient to review.

Authors go on combining different concepts to classify its paper. For example: “new data mining system that employs text mining techniques”, this description is confused and authors combine both definitions in the paper. I recommend authors that they focus directly on Text Mining to describe their work.

I recommend the next references to authors. These references show the description of text mining, its relation with data mining and its main applications:

Gupta, V., & Lehal, G. S. (2009). A survey of text mining techniques and applications. Journal of emerging technologies in web intelligence, 1(1), 60-76.

Patel, F. N., & Soni, N. R. (2012). Text mining: A Brief survey. International Journal of Advanced Computer Research, 2(4), 243-248.
Experimental design: No Comments
Validity of the findings: No Comments
Additional comments: No additional comments


Instruction: You are one of the reviewers, your task is to write a review for the article. You will be given the title of the article, the number of the round in which the article is located, and your order among the reviewers.

Input:
Title: SHARING ANALYSIS IN THE PAWNS COMPILER
Review round: 1
Reviewer: 1

Output:
Basic reporting: Some notations in the article are not defined:

- In the definition below Line 421 you use the notation ":c" which has not been defined before. Moreover, what is the difference between using square brackets and parentheses, such as "v1.[Ref.1]" and "v1.(Ref.1 :c)"?

- In the definition below Line 426 you use the notation "++c" which has not been defined before. Is it the same as ":c" above? Please define both and explain the difference.

- In the definition below Line 435 you use both the notations "fc(v.[dc.i])" and "fc(v.(dc.i:c))": explain the difference.
Experimental design: No Comments.
Validity of the findings: No Comments.
Additional comments: The paper describes the sharing analysis in the Pawns compiler. The author briefly describes the language and then defines an abstract domain and a sharing algorithm. Since the language contains some extra annotation on the variables which are allowed to share before and after a function call, sharing analysis is used to verify that the annotations cover all the possible sharing.

The author describes the algorithm without a formal proof of correctness (which would help in clarifying some steps of the algorithm but could be mostly case-based and maybe boring).

The results presented in the article are new and correct.

The article is very difficult to read, also because Pawns is a new language. I would strongly suggest to add some figures such as that below Line 132, at least in the example at Line 355 and in the three examples in Section 6, to help the reader.

Moreover, some notations are not defined and should be fixed (see "Basic Reporting" area).

Typos:
- Line 54 mutablity -> mutability
- Line 62: I do not understand the sentence: ".. which source programs are translated into". Please rephrase it.
- Line 443 our the -> our


Instruction: You are one of the reviewers, your task is to write a review for the article. You will be given the title of the article, the number of the round in which the article is located, and your order among the reviewers.

Input:
Title: SHARING ANALYSIS IN THE PAWNS COMPILER
Review round: 1
Reviewer: 2

Output:
Basic reporting: The article describes the sharing analysis used in the Pawns compiler to validate the correctness of the sharing and mutability annotations. The presentation is mostly self-contained. There are some sentences which are difficult to grasp without having read paper [1] in the bibliography. I have detailed these cases in the "General Comments for the Author" section, but overall they do not affect very much the readability of the paper.

Bibliographic references could be improved. For example, there is a vast literature on the static analysis of sharing for logic programs, but the author only cites the ones which follow the approach based on type folding. A good overview of other approaches to sharing analysis may be found in "Bagnara R, Zaffanella E, Hill PM. Enhanced Sharing Analysis Techniques: A Comprehensive Evaluation. TPLP 5 - 1&2". Some of these analysis have been ported to object oriented languages with destructive updates, such as "Mario Méndez-Lojo, Manuel V. Hermenegildo. Precise set sharing analysis for Java-style programs. Proceedings of VMCAI'08".
Experimental design: No comments.
Validity of the findings: While the general approach followed by the analysis is sound, there is no proof of correctness of the results. This makes very difficult to judge the validity of the work, and increases the possibility that analysis has hidden flaws. I understand that a full correctness proof would be very long and tedious, but some attempts to justify at least the most interesting and tricky constructs (namely, function application and destructive update) should be provided.

In any case, I think that the analysis is mostly correct, but I found some soundness issues  which I am going to detail below. The issue with function application is, in my view, the most serious, since it is not clear how to fix it without causing an high loss of precision.

- page 10 line 385-387. I do not think condition 6 as stated here is enough to ensure type safety. Consider the following program

*xsp = []
ysp = xsp::Ref [Int]
zsp = zsp::Ref [Bool]
(assign !ysp [3]) !zsp
a1 = and *zsp

In this case xsp is not alive in line four, hence we do not need to annotate the assign statement with !xsp. The assignment indirectly modifies *zsp to be a list of integers, clearly violating well-typedness. However, condition 6 is respected since [Int] is not a less general type that [Bool]: they are actually incomparable. Therefore, I think that point 6 should be changed in "any live variable that may be indirectly update at that point only shares with variables which have a more general type".

- page 11, alias definition for Assign. Some sharing information is missing on the result, concerning the component v1.[Ref.1]. Consider the following type information:

data Box3 = B3 Int
data Box2 = B2 Box3

and the program fragment:

b3 = B3 0
b3bis = B3 1
b2 = B2 b3
case b2 of
  (B2 b3p) ->
     *!b3p := b3bis (!b2)

This is the same program annotated with sharing information computed by the analysis. Note that I use the same abbreviations introduced in line 496 of the paper.

b3 = B3 0
< { b3.[B3.1] } >
b3bis = B3 1
< { b3.[B3.1] }, { b3bis.[B3.1] } >
b2 = B2 b3
< { b2.[B2.1] }, { b2.[B2.1,B3.1], b3.[B3.1] }, { b3bis.[B3.1] } >
case b2 of
  (B2 b3p) ->
     < { b2.[B2.1], b3p.[Ref.1] }, { b2.[B2.1,B3.1], b3.[B3.1], b3p.[Ref.1, B3.1] }, { b3bis.[B3.1] } >
     *!b3p := b3bis (!b2)

For the assignment we have:

self1 = < (b3p.[Ref.1], b3p.[Ref.1]), (b3p.[Ref.1,B3.1], b3p.[Ref.1,B3.1]) >
share1 = < (b3p.[Ref.1, B3.1], b3bis.[B3.1]) >
al = { b2.[B2.1] }
selfal = < (b2.[B2.1, B3.1], b2.[B2.1, B3.1]) >
shareal = < (b2.[B2.1, B3.1], b3bis.[B3.1]), (b2.[B2.1, B3.1], b3p.[Ref.1,B3.1]) >
old1 = < (b3p.[Ref.1], b3p.[Ref.1]), (b3p.[Ref.1, B3.1], b3p.[Ref.1, B3.1]), (b2.[B2.1,B3.1],  b3p.[Ref.1, B3.1]), (b3.[B3.1], b3p.[Ref.1, B3.1) >
a0 \ old1 = < (b2.[B2.1], b2.[B2.1]),  (b2.[B2.1,B3.1], b2.[B2.1, B3.1]), (b3.[B3.1],b3.[B3.1]), (b2.[B2.1,B3.1], b3.[B3.1]), (b3bis.[B3.1], b3bis.[B3.1]) >

result = < { b2.[B2.1] }, { b3p.[Ref.1] }, { b2.[B2.1,B3.1],  b3bis.[B3.1], b3p.[Ref.1,B3.1] }, { b2.[B2.1,B3.1], b3.[B3.1] } >

Note that the result does not contain the pair (b2.[B2.1], b3p.[Ref.1]), while these two components share the same location on the heap. In order to fix the definition, it could be enough to add to the result all the pairs between al and v1.[Ref.1], but this requires further investigation.

- page 12, alias definition for DC. I think it is wrong and some pairs are missing from the result, namely:

\bigcup_{i,j \leq 1 \leq N} { {fc(v.(dc.i:c_1)), fc(v.(dc.j:c_2))} | {vi.c1, vj.c2} \in a_0, i != j }

Otherwise, sharing among variables v1, ... vN in not propagated to sharing among the components of v.

- page 13, alias definition for App. I think this is wrong, since posttt does not include all sharing informations which may be created during the execution of f. The problem is that posttt only updates sharing informations on the base of the situation at the end of the function application. However, during the execution, more sharing may be created which is not exposed in the formal arguments at the end.

Consider the following program fragment:

data Box3 = B3 Int
data Box2 = B2 Box3
data Box1 = B1 Box2

bad: Box1 -> Box3 -> ()
   sharing bad !b1 b3 = _
   pre nosharing
   post nosharing

bad !b1 b3 =
   case b1 of
     (B1 b2p) ->
        b2 = *b2p
        case b2 of
	    (B2 b3p) ->
	       *!b3p := b3 (!b2, !b2p, !b1)
        newb3 = B3 0
	newb2 = B2 newb3
	*!b2p := newb2 (!b1)

Note that the post-condition of the "bad" function is correct, although it cannot be verified by the current analysis algorithm due to loss of precision. Now assume bad is called in the following context.

b3 = B3 0
b3bis = B3 1
b2 = B2 b3
b1 = B1 b2
bad !b1 b3bis

Before the call to "bad", we have the sharing < { b1.[B1.1] }, { b1.[B1.1, B2.1], b2.[B2.1] } , { b1.[B1.1, B2.1, B3.1], b2.[B.2.1, B3.1], b3.[B3.1] }, { b3bis.[B3.1] } >. There is no sharing between b1 and b3bis, hence "bad" may be called. Since the postcondition for bad is "nosharing", the posttt variable does not add any new pair. Therefore, according to the analysis, after we return from "bad" the variables b2 and b3bis do not share. However, this is not true, since inside the execution of "bad" b2 is changed to (B2 b3bis).

It could be possible to argue that this is not a problem because "bad" is not accepted with the given postcondition, but in this way the correctness of the analysis would be dependent of the fact it is not precise enough. This is not a solid ground on which to build the analyzer: any improvement on the analysis could break its correctness. Moreover, I am not sure that it is not possible to provide an example like "bad" which is accepted by the analysis.

This might be fixed by either changing the definition of "alias" for the function application case, or by changing the meaning of the post-condition, so that it records not only the last sharing situation but also (at some extent which should be investigated) the intermediate situations during the execution of the function.
Additional comments: ==> Presentation

- page 2 line 50-51. In the sentence "may lead to a proliferation of different versions of data structures" it is not clear what are the different versions you are talking about. I think you are referring to the situation described in page 8 of [*], where different versions of the same function only differ for the set of mutable arguments. This needs to be stated more explicitly.

- page 2 line 52. It is not clear what "destructive update requiring an extra level of indirection" means. I think you are referring to the extra level of indirection you get when you replace every constructor argument with references, as discussed in page 9 of [*]. However, you should be more explicit, because in most languages you may destructively update a non-reference type, and therefore you do not need references everywhere.

- page 6. Somewhere in this page you should state that for those variables which share with abstract you do not keep or track further sharing information. Moreover, if I am not wrong, when you call a function f and an actual parameter is abstract, the corresponding formal parameter should be abstract too. I see this is a consequence of respecting the preconditions for f, however explicitly recognizing this property helps understanding what abstract really means.

- page 8 lines 285-294. Not everyone is familiar with polymorphic types: I would devote more space to explain the problem with destructive updates of polymorphic types. The discussion and the examples in Section 10 of [*] are quite good for this purpose.

- page 10 line 376-387. I would add another point to the list: "for all function calls, the sharing information among the actual parameters is a subset of the sharing information among formal parameters as declared in the pre-condition, modulo variable renaming". The fact that checking pre-conditions is needed is acknowledged later, in line 391, but I do not see any reason why not put it in the list.

- page 11 lines 408-410. There are several things I do not understand here. First of all, what do you mean by "type assignment"? I think it is the correspondence between variables and types computed by the type-checking phase, but I am not sure. Furthermore, you say that "sharing information is given for all type instances of all defined functions". But sharing information should be computed by the analyzer, so what is this sharing information which is given a priori? Are you referring to sharing information in the pre- and post-conditions? Or to sharing information attached to closure types? Moreover, it seems that for polymorphic functions you have many sharing informations for different instances of the type variables. How is it possible?

- page 11 line 410. Here it seems you allow to declare sharing among parameters of different type variables, while in [*] this is explicitly disallowed. A comment would be welcome.

- page 11, alias definition for DerefEq. You introduce the notation (Ref.1 :c) without defining it.

- page 12, alias definition for Assign. You introduce the notation c_a ++ c without defining it.

- page 12, alias definition for Assign. Please, explain why we need to keep old1 when the assignment might create a cyclic structure, since I was not able to come up with an example where it was really needed.

- page 13, alias definition for App, definitions of sharec and self. In order to be consistent with the reverse ordering of closure arguments, I think it is better to replace the components v.[Cl.i] and v.(Cl.i :c) with v[Cl.N-i+1] and v(Cl.N-i+1 :c) respectively.

- page 13 lines 453-454. I think to understand the role of closure arguments, but I cannot grasp the meaning of this sentence.

- page 14 lines 458-461. This sentence puzzled me until I realized that abstract variables and non-mutable variables are two different things. Probably you should stress out this fact somewhere.

- page 14 aliasCase definition. To be consistent (merely from a stylistic point of view) with the alias function for DC, the definitions for avdc, vishare and share should be preceded by \bigcup_{1 \leq i \leq N}. (and also \bigcup_{1 \leq j \leq N} for vishare). Alternatively, the union may be removed in the alias definition of DC.

==> Typos:

- page 7 line 244. The entire line should be removed.

- page 12, alias definition for DC. In the first line of self1, dc is in italic font but it should be in typewriter.

- page 13, alias definition for App. The definitions for selfc and sharec are full of spurious ")"

- page 13 line 443. "our the implementation": either "our" or "the" should be removed.

- page 13 line 454: prepresentation => representation

- page 14 line 461: transisitivity => transitivity

- page 14 aliasCase definition. In the definition of vishare and share the indexes i and j are in typewriter font while they should be in italic.


Instruction: You are one of the reviewers, your task is to write a review for the article. You will be given the title of the article, the number of the round in which the article is located, and your order among the reviewers.

Input:
Title: SHARING ANALYSIS IN THE PAWNS COMPILER
Review round: 2
Reviewer: 1

Output:
Basic reporting: No Comments
Experimental design: No Comments
Validity of the findings: No Comments
Additional comments: I have very appreciated the effort of the author to improve readability of the paper with the added examples and explanations. Also the use of "aliasing" instead of "sharing" seems more appropriate. I think that the article can be accepted without further revisions.

Minor points:

Line 478: the definition of the function "take" should be in a separate line
Line 502: write explicitly that in old1 (in Assign) :d:c_1  means that the length is at least one.
Line 547: in selfc (in App) remove the extra parenthesis after v_i.c_1,v_j.c_2
Line 832: I disagree that set-sharing is usually augmented with groundness. Even if [11] provides some evidence that groundness information is useful, often sharing is augmented with linearity or freeness information ([11] is a work of 10 years ago, there has been much work on this topic, such as finding improved algorithms for exploiting linearity and/or freeness in sharing).
Line 840: This claim is too strong, I think that Java has sum through subclasses.


Instruction: You are one of the reviewers, your task is to write a review for the article. You will be given the title of the article, the number of the round in which the article is located, and your order among the reviewers.

Input:
Title: SHARING ANALYSIS IN THE PAWNS COMPILER
Review round: 2
Reviewer: 2

Output:
Basic reporting: The paper has been vastly improved w.r.t the previous version: not only it is much more readable, but also many bugs have been fixed. I also like the fact that the term "sharing pair" has been replaced with "aliasing pair": I find the latter much more precise. However, I think there are still some problems from the point of view of correctness which deserve a minor revision.
Experimental design: No comments.
Validity of the findings: --> Algorithm for "assign"

There is a correctness issue I overlooked the first time I reviewed the paper. If "v2.c1" and "v2.c2" may share, after *v1:=v2 the same should hold for "v1.(Ref 1 : c1)" and "v1.(Ref.1 : c2)". However, this does not happen with the current analysis algorithm. It should be easy to fix.

--> General correctness of the analysis

Apart from the bug described above, I think the analysis algorithm is correct. However, condition C on page 11, which is the key inductive hypotheses for a possible correctness proof, is not complete. The problem lies at point 4 of C: "the union of pre- and post-conditions of f abstracts the state when f returns plus the values of mutable parameters in all states during the execution of f". This condition is not strong enough to ensure correctness of the algorithm for App, and the evidence is the program f2 on page 19. During the execution of this program, v1 and v2 never share. Therefore, if we only require that the post-condition abstracts the values of the parameters in all the states during the execution, a valid post-condition for f2 would be "nosharing". But this would make the App algorithm incorrect.

Nonetheless, the analysis works because "nosharing" is not accepted as the post-condition of f2, and the reason is that the invariant which is maintained by the analysis is more complex than what is stated at point 4. In lines 582-584 the paper says that parameters are actually "proxies for the argument variables as well as any variable components they may alias". This fact is essential to prove correctness, and should be part, in some way or another, of the condition C. One possibility could be to add something like this to C: "If v1 and v2 are mutable parameters, before the execution va.ca is an alias for v1.c1 and vb.cb is an alias for v2.c2, and the execution of f may make va.ca and vb.cb aliases, then (v1.c1, v2.c2) is in the union of pre- and post-condition". This is enough to prove correctness of App, and I think is preserved by all the other operations (although I suspect Assign is probably too conservative, but this is not a correctness problem).
Additional comments: First of all, I am sorry that my previous review did not contain the bibliography data for [*]: it was a reference to "An informal introduction to Pawns: a declarative/imperative language", which you have cited as reference [1].

--> Expressing sharing properties

It is not always easy to express pre- and post-condition using Pawns' syntax. For example, assume you want to define the function zip with the following pre- and post-conditions:

zip:: [a] -> [a] -> [Pair a a]
sharing zip xs ys = zs
pre ... elements of xs and ys may share, but the "spine" cannot ...
post ... first and second elements of each pair in zs may share ....

Is it possible to express this properties concisely? For the pre-condition, something like this could work:

Case xs of
   Cons(*x, *xs1) ->
      Case ys of ->
         Cons(*y, *ys1) -> *x = *y; ()
         Nil -> ()
   Nil -> ()

But it is not compact at all (and maybe is also incorrect). I think you should add (not necessarily for this paper) the possibility of expressing sharing as a set of pairs, i.e., with the same representation used by the alias algorithm.

--> Minor points

- page 7 line 252. This sentence is confusing. A concrete tree is returned for any post-condition different than abstract, not just for nosharing.

- page 8 line 313. I would move the first sentence of this paragraph either at the end of the paragraph (replacing InsType with ::), or at the end of the previous paragraph (where InsType is defined).

- page 11 point 5. You have added this point, following my suggestion. Actually, now that I have read the answer in your rebuttal letter, I see the reason why this point was not there in the first place. So, I would not object to the removal of point 5 (but I think it is fine to keep it, if you want).

- page 12 line 459. Why you feel the need to explicitly state that Instype is in the core language?

- page 12 lines 478-479. Please, reformat the "take" function on separate lines, as for all the other examples.

- page 12 lines 480-483. It is still not clear how pre- and post-conditions for the application are generated by pre- and post-conditions in the definition of the function. For example, assume that v has type [ Pair (Ref Int) (Ref Int) ] ( a list of pairs of references to int ). Assume you know that v.[Cons.1, Pair.1, Ref.1] and v.[Cons.1, Pair.2, Ref.1] do not share. Is  it possible to infer that, after "y = take 5 v", also y.[Cons.1, Pair.1, Ref.1] and y.[Cons.1, Pair.2, Ref.1] do not share. I would say yes, because since "take" is polymorphic, it cannot "open" or "generate" new pairs. Is it true or there are corner cases I am overlooking?

- page 13 lines 484-486. This is related to the question about line 410 on my previous review. In the answer to that question, you say that type checking prevents sharing between things whose types are distinct variables. I think this means that the following declaration would not type check since xs=ys is not allowed in the pre-condition section.

zip:: [a] -> [b] -> [Pair a b]
sharing zip xs ys = zs
pre xs = ys
post ...whatever works...

However, the sentence "can be shared to represent possible sharing of arbitrary components of an arbitrary type" suggests the opposite. I think you should clearly state in the paper what you wrote in the rebuttal letter, i.e., that type checking prevents sharing between things whose types are distinct variables.

- page 14, algorithm for Assign. It should be possible to simplify the algorithm by removing self1 and share1. Since Ref's cannot be null, { v1.[Ref.1], v1.[Ref.1] } is always in a0, hence almost all the aliasing pairs in self1 and share1 also appear in selfal and shareal. The only exception is { v1.[Ref.1], v1.[Ref.1] } which is in self1 but might not be in selfal. However, { v1.[Ref.1], v1.[Ref.1] } is in a0 \setminus old1, hence it also appears in the result.

- page 14, algorithm for Assign. Please, make it clear that ":d :c1" in the definition of "old1" is not a typo, but a way to ensure that all the components in old1 have at least one element after Ref.1.

- page 14, line 515. I would start another paragraph just before "Consider the example below".

- page 15, line 535. Is "modula" correct in English ?

- page 16, algorithm for App. There is a spurious ")"  in the third line of the definition for "selfc".

- page 17 line 553. I would start a new paragraph just before "Analysis of function calls". As an alternative, a preposition such as "On the contrary" or "Instead" should separate this part from the previous one devoted to the case when application generates a closure.

 - page 23, end of section 7. You are making comparisons among different compilation and execution times, but you do not give any concrete value. Are the times involved big enough that the differences are significant (in the sense, for example, that results are not biased by the start-up delay of different compilers and runtime systems) ?

- page 24 lines 835-854. I do not fully agree with your statement that Java and other object oriented languages do not have "sum of products". Subtyping through inheritance is not very dissimilar. It may be used to simulate abstract data types, although standard OO languages do not have convenient syntax constructs to work with them (but others, such as Scala, do). In any case, my point is that your abstract domain, with the appropriate changes, could be useful even with standard OO languages.


Instruction: You are one of the reviewers, your task is to write a review for the article. You will be given the title of the article, the number of the round in which the article is located, and your order among the reviewers.

Input:
Title: CFSAN SNP PIPELINE: AN AUTOMATED METHOD FOR CONSTRUCTING SNP MATRICES FROM NEXT-GENERATION SEQUENCE DATA
Review round: 1
Reviewer: 1

Output:
Basic reporting: No comments.
Experimental design: This manuscript describes a custom pipeline designed to take fastq files through the mapping and variant-calling phases with one program. The concept behind the pipeline is of high importance as the analysis of next-gen datasets is currently cumbersome and requires multiple pieces of software. Additionally, the pipeline would be useful as additional samples became available in order to standardize the overall analysis.

Although the basic steps of the SNP pipeline are well-outlined and rely, for the most part, on currently available software (Bowtie2, samtools, Varscan), some custom scripts are included and therefore a validation of this pipeline on a publically-available SNP dataset is required.  Although it is understood that different variant callers have various strengths and weaknesses, without a "gold standard", there is really no evidence that this pipeline will give accurate variant calls.

How will this pipeline be updated to reflect newer versions of the software used?
Validity of the findings: See comments above.
Additional comments: Although implied, it needs to be explicitly stated that this pipeline will remain freely available.


Instruction: You are one of the reviewers, your task is to write a review for the article. You will be given the title of the article, the number of the round in which the article is located, and your order among the reviewers.

Input:
Title: CFSAN SNP PIPELINE: AN AUTOMATED METHOD FOR CONSTRUCTING SNP MATRICES FROM NEXT-GENERATION SEQUENCE DATA
Review round: 1
Reviewer: 2

Output:
Basic reporting: The paper and the software package feel a bit disconnected. The software package is everything I would like to see in an academic pipeline: source code available on GitHub, installation and usage well documented on readthedocs.org, and test cases distributed with it. However, the paper describing the software does not contain enough information to help the reader decide whether or not to use it. As the pipeline uses currently available tools and the current landscape of short-read aligners and variant callers is large and in flux, not enough discussion is devoted to the rationale behind the use of those tools.
Experimental design: Quality trimming: an obvious omission from the pipeline is trimming of the raw fastq reads for a quality cutoff. While [recent studies](http://journal.frontiersin.org/article/10.3389/fgene.2014.00013/) have shown that most data may in fact be over-trimmed, the same study shows no trimming leads to a large number of spurious k-mers. Was quality trimming omitted for the sake of speed? Was it found to have little effect on the final SNP pileup?

Explain the reasoning behind the "custom" parts of the pipeline. It is mentioned that Varscan's consensus caller is not used. Why not? What does this custom one do? I tried digging through the snppipeline.py and other code, but was unable to work out the logic in a timely fashion.

Explain the reasoning behind the choice of various tools. SAMtools and sra-toolkit are essentially without peer, but Bowtie2 is one of half a dozen commonly used short read aligners. Were any other tools (bwa-mem, novoalign, SOAP) tried? Is the emphasis on speed? Same for the choice of SNP callers. Was FreeBayes, SOAPsnp or something else considered? Why or why not?

While it is not free for all users, the GATK variant calling pipeline is widely used. How does this pipeline compare, both in terms of accuracy and run time?
Validity of the findings: A possible use of the pipeline mentioned is phylogenetic analysis of food borne outbreaks. To that end, the software seems to assume small, single chromosome organisms will be reference mapped. In the resulting snpma file, samples are ID'd by sample name only. It is possible to reconstruct which SNP's are in which position/chromosome using the snplist.txt file, but it is cumbersome.

The snpma file is referred to as a "SNP matrix" but its format appears to be a compressed multifasta .aln file containing only the positions that vary between samples. A search of the literature produced no standard for file formats relating to SNP matrices, but the most common is samples are rows and the called positions are columns labeled by position. As this is not the format of the snpma file, it should be described/explained. Also, an example pylogenetic treatment of the snpma test data in PHYLIP or some other package could be helpful.
Additional comments: I installed and ran the software on both the test lambda data and 7 strains of C. neoformans (SRR1063249, SRR1063250, SRR1063251, SRR1063252, SRR1063259, SRR1063260, SRR1063281) I am currently analyzing. The install process was relatively smooth, above average for a typical bioinformatics software package.

I may have misread the instructions, but it appears that the pipeline does not support multiple read files per sample. One of the reasons I only tested 7 C. neoformans data sets is that the rest have multiple read files per strain.

The name "SNP Pipeline" is too general. However, I am loathe to suggest yet another clever name for a bioinformatics tool. The name on the github and readthedocs pages is "CFSAN SNP Pipeline," I would suggest calling it that in the paper for the sake of people finding it.

Conclusion, line 132: "robust and accurate" is not supported by the manuscript as written. "robust and reproducible" is certainly true, however. Claims to accuracy require comparison to other workflows and pipelines, which are currently not presented.


Instruction: You are one of the reviewers, your task is to write a review for the article. You will be given the title of the article, the number of the round in which the article is located, and your order among the reviewers.

Input:
Title: CFSAN SNP PIPELINE: AN AUTOMATED METHOD FOR CONSTRUCTING SNP MATRICES FROM NEXT-GENERATION SEQUENCE DATA
Review round: 2
Reviewer: 1

Output:
Basic reporting: The addition of validated variant calls has greatly added to this manuscript.
Experimental design: As above
Validity of the findings: Findings appear valid on this revision.
Additional comments: Minor comments:

(1)	Line 53: “…that, when given a reference genome, will generate…”
(2)	Line 221-222: “…within which…”; awkward, re-word
(3)	Line 240-242: reword “See supplemental data S1…”
(4)	Line 278: “CFSAN SNP Pipeline”
(5)	Line 332-334: Based on these results, should there be a minimum depth of coverage to use this program? I agree and appreciate that future developments should include a depth of coverage threshold
(6)	Line 354: “and also”
(7)	Line 355; reword sentence “Such a dataset….”
(8)	Some discussion/re-iteration regarding the lack of detection of insertions and deletions is warranted as the program was not designed specifically to detect them.
(9)	Table 2, “False positives”.  It may be worth noting the total number of SNPs, insertions and deletions created by CFSAN SNP mutator.
(10)	Table 3 can be supplementary.


Instruction: You are one of the reviewers, your task is to write a review for the article. You will be given the title of the article, the number of the round in which the article is located, and your order among the reviewers.

Input:
Title: CFSAN SNP PIPELINE: AN AUTOMATED METHOD FOR CONSTRUCTING SNP MATRICES FROM NEXT-GENERATION SEQUENCE DATA
Review round: 2
Reviewer: 2

Output:
Basic reporting: The authors have sufficiently addressed my previous concerns, and written a new package to produce randomly mutated genomes.
Experimental design: Testing of the accuracy via randomly mutated genomes provides a measure of error, and source of the error is further characterized. CFSAN SNP mutator is a tool others could use to assess their own pipelines.
Validity of the findings: No further comment.
Additional comments: Table 2, check spelling of False Positive, rendered "P seositives" in my copy.
Consider deleting "simple" from the description of CFSAN SNP Mutator
Add the github information and availability information for CFSAN SNP Mutator to the end of the manuscript.


Instruction: You are one of the reviewers, your task is to write a review for the article. You will be given the title of the article, the number of the round in which the article is located, and your order among the reviewers.

Input:
Title: PAME: PLASMONIC ASSAY MODELING ENVIRONMENT
Review round: 1
Reviewer: 1

Output:
Basic reporting: In the abstract, the author stated that the plasmonic assays own the advantages such as labeling free and measurement in real time.  The combination of nanoparticles and fiberoptics show promise in in-situ probing and implantable devices.  But the author did not provide information to support those advantages in the article.  It will be better if the advantages of plasmonic assays and combination of nanoparticles and fiberoptics can be moved to the "Introduction" Chapter.

The abstract should focus on building, verify the model with PAME platform.
Experimental design: In Figure 4, the horizontal coordinate seems represent the wave length. What is the meaning of “counts” in vertical coordinate?  How the normalized counts in Figure 6c and 6d were obtained?  Provide detail information will make those clearer.
Validity of the findings: No Comments.
Additional comments: No additional comments


Instruction: You are one of the reviewers, your task is to write a review for the article. You will be given the title of the article, the number of the round in which the article is located, and your order among the reviewers.

Input:
Title: PAME: PLASMONIC ASSAY MODELING ENVIRONMENT
Review round: 1
Reviewer: 2

Output:
Basic reporting: This article meet all standards of PeerJ with a very high quality.
For example, the structure of the article are well organized, content are self-sustained with appropriate introduction and background information. Data and source are openly shared online.
Experimental design: Several experimental examples are given in this article to show how to use the open-sourced tool PAME.
Experiments are clear described and results are well discussed. A very high technical standard are maintained in the experiment design.
Meanwhile, supplemental examples are also abundant and very helpful for new users.
Validity of the findings: The software introduced in this article shows a very convenient interface, comprehensive functionalities for modeling plasmonic systems, as well as many previous modeling examples in literature.

With the authors open-sourcing the software, it will be easily accessible to researchers in the area, and bring a great contribution to the community.
Additional comments: The submitted paper discuss the plasmonic assay modeling and simulation, and introduced an open-sourcePython application for modeling plasmic systems of bulk and nanoparticle-embedded metallic films.

The article is of very high quality in terms of basic structure, scientific reporting, theory and experiment design, and final validation.

Very minor issues are: 1. a few ? marks in Page 1( in Introduction), and Page 11 (2 paragraph before section 4 IMPLEMENTATION AND PERFORMANCE).
2. Format issue, Page 2, COMSOL Multiphysics is extending to margin area.


Instruction: You are one of the reviewers, your task is to write a review for the article. You will be given the title of the article, the number of the round in which the article is located, and your order among the reviewers.

Input:
Title: PAME: PLASMONIC ASSAY MODELING ENVIRONMENT
Review round: 1
Reviewer: 3

Output:
Basic reporting: Please give more introduction and analysis for the section 4 implementation and performance
Experimental design: The experimental design is well organized and reasonable
Validity of the findings: The data can support authors’ ideas and conclusions
Additional comments: This paper has innovated idea and good simulation; it is also well organized. If the authors can provide more introduction and analysis for section 4 implementation and performance, it is much better.


Instruction: You are one of the reviewers, your task is to write a review for the article. You will be given the title of the article, the number of the round in which the article is located, and your order among the reviewers.

Input:
Title: PAME: PLASMONIC ASSAY MODELING ENVIRONMENT
Review round: 2
Reviewer: 1

Output:
Basic reporting: The author introduced open-source Python application for modeling plasmonic systems of bulk and nanoparticle-embedded metallic films, and also present the PAME’s theory and design. The paper was well organized and the examples are suitable.
Experimental design: After building the model, the author gave two examples for the application, one is Refractometer and another on e is Multiplexed Ag-Au Sensor.  Those two examples are sufficiently support the author's ideas.
Validity of the findings: The examples in this paper are simulations of a fiberoptic refractometer,and protein binding to a multiplexed sensor composed of a mixed layer of gold and silver colloids.
The examples, data and citation can support the author's view points in modeling the plasmonic system.
Additional comments: The author introduced a method to model the plasmonic systems  based on open-source Python, and also provide some examples and data to support the view points.  This modeling method is meaningful in analyze the plasmonic systems.

.


Instruction: You are one of the reviewers, your task is to write a review for the article. You will be given the title of the article, the number of the round in which the article is located, and your order among the reviewers.

Input:
Title: PAME: PLASMONIC ASSAY MODELING ENVIRONMENT
Review round: 2
Reviewer: 2

Output:
Basic reporting: This article meet all standards of PeerJ with a very high quality.
Experimental design: Experimental design are carefully described and results are well discussed.
Validity of the findings: No additional comments.
Additional comments: Again, this article is of very high quality in terms of basic structure, scientific reporting, theory and experiment design. The authors had fixed previous issues and now is fully ready for publication.


Instruction: You are one of the reviewers, your task is to write a review for the article. You will be given the title of the article, the number of the round in which the article is located, and your order among the reviewers.

Input:
Title: PAME: PLASMONIC ASSAY MODELING ENVIRONMENT
Review round: 2
Reviewer: 3

Output:
Basic reporting: No comments
Experimental design: No comments
Validity of the findings: No comments
Additional comments: No additional comments


Instruction: You are one of the reviewers, your task is to write a review for the article. You will be given the title of the article, the number of the round in which the article is located, and your order among the reviewers.

Input:
Title: PAME: PLASMONIC ASSAY MODELING ENVIRONMENT
Review round: 2
Reviewer: 4

Output:
Basic reporting: The authors reported a python-based software platform that could model and simulate the behavior of plasmonic assays assuming certain experimental conditions. PAGE also provides an user friendly interface for modeling and simulation.
Experimental design: The authors reported three examples of how PAGE could be utilized. In the first example, PAGE was used to simulate the sensor response to increasing concentrations of glycerine for an axial fiber. It was  shown that there was a strong agreement of between measured and PAGE simulated response. In the second example, PAGE was used to model a multiplexed dip sensor containing gold and silver nanoparticle species. Despite several approximations, PAGE was able to provide several insights into the multiplexed sensor.
Validity of the findings: In the first example, PAGE’s simulated results aligned well with the experimental results. In the second example, PAGE’s simulated results were supported by several lines of evidence from previous publications.
Additional comments: Page 2, change “PAME if a fully graphical application that …” to “PAME is a fully graphical application that …”


Instruction: You are one of the reviewers, your task is to write a review for the article. You will be given the title of the article, the number of the round in which the article is located, and your order among the reviewers.

Input:
Title: HOW DO YOU FEEL, DEVELOPER? AN EXPLANATORY THEORY OF THE IMPACT OF AFFECTS ON PROGRAMMING PERFORMANCE
Review round: 1
Reviewer: 1

Output:
Basic reporting: 1) I could not find where the authors have made available the intermediate results, such as the codes and other data. As it is presented, the "chain of evidence" for interpretive purposes is not very explicit. I understand that is the case for many IS research papers, but I would like to rise the bar and have a bit more concrete description of the intermediate results during the analysis/interpretation stage.

2) Some terms should be clarified, and some expressions need to be re-formulated:
Examples:
Page 2, line 64: what you mean by "removed treatment"?
Page 3, line 123: please explain what you mean by "variance-based"
Page 3, line 125: "Our theoretical framework was primarily *based* on.."
Page 11, line 503: "*A* more tangible focus.."
Experimental design: I am very curious about the four-staged coding process. In particular, I am missing some middle-level results when going from focused coding to axial coding. In particular, why are no examples of categories extracted via axial coding? you jump directly from 308 codes to 6 themes, which is a quite big jump. In addition, I am interested on if there was any cross-validation of the coding (in particular during axial coding) across multiple researchers, or if it was only one researcher who did the entire interpretation. Also, can you elaborate further on the difference between initial and focused coding? - maybe with some examples?

When it comes to conflicting results between the manikin questionnaire and the qualitative data (here I am assuming the open ended interviews, the regular meetings and the observational sessions), how and when did you performed the clarifications when conflicting answers were identified? - an example could be very illustrative.
Validity of the findings: A major aspect that I am not extremely convinced in the proposed theory is the distinction made between Affects and Attractors. I believe the authors need to present a better case to justify the need of having two separate constructs, which appear to be difficult to clearly separate.

Also, I would suggest the authors to touch more actively upon how the specifics particular to the software engineering context affect the applicability of more general theories explaining the relationship of affect and performance. Basically, a more active comparison between the existing, more general theories with respect to the theory presented is advisable.

In the limitations of the work, the authors need to discuss their choice of having only males in an inductive study. It is well known that gender plays a major role on outcomes and behavior related to affect, and they should at least discuss how their sample may have affect the dynamics of the project, and thus the narrative from which the theory is derived.
Additional comments: This paper is well written, provides a good line of argumentation in relation to the relevance of the topic, it presents a sound methodology, and the results are presented in a clear, concise manner. The implications of affect on programming performance is a topic that it has caught up attention recently in SE, and the authors make a clear case for the "knowledge gap" they are trying to address. It was a pleasent experience to review this paper.


Instruction: You are one of the reviewers, your task is to write a review for the article. You will be given the title of the article, the number of the round in which the article is located, and your order among the reviewers.

Input:
Title: HOW DO YOU FEEL, DEVELOPER? AN EXPLANATORY THEORY OF THE IMPACT OF AFFECTS ON PROGRAMMING PERFORMANCE
Review round: 1
Reviewer: 2

Output:
Basic reporting: This paper presents a theory that explains the affects that software developers have and how these affects impact their performance while programming. To come up with the theory, the authors conducted a qualitative study and collected data through observations and various kinds of interviews. Using grounded theory techniques, the authors then established an explanatory theory.

The authors do a very good job in motivating why their research and in particular a theory of the impact of affects on software developers is necessary.  In the paper, it is clearly explained what the implications of such a theory are, why a better understanding is necessary, and how the theory can help to get a better understanding. The authors also provide reasons why such a theory was not yet established, and why in general researchers have not yet focused much on software developers’ affects. Furthermore, the authors also put a lot of effort into explaining the theoretical background of their theory construction approach.

The paper is also well structured and written in a way that makes it easy to follow. However, in some cases, important details and explanations are missing. For example, the authors state that there were three discrepancies between the qualitative data and the SAM data, but all of these three discrepancies could be resolved. It would be interesting to know how these discrepancies were resolved and why there were discrepancies in the first place. Another aspect that needs clarification is the questionnaires that were used directly after a developer made a commit. What was the reason to let the participants fill out a questionnaire exactly at this point in time and what happened if a developer could not make any commit for a whole week (as mentioned in quote 16)? Didn’t that lead to a loss of data for this participant? How did the authors handle this situation? Furthermore, for me it is unclear how the paragraph from line 379 to 391 fits into the section about developers’ affects.
Experimental design: The experimental design that the authors have chosen is a good fit for their research. It allows them to observe developers during their work, and using grounded theory techniques, come up with a theory that explains the observations. However, there are two issues with the experimental design.

The authors should clarify what exactly they consider to be an emotion. There is already a lot of research in this direction, and different authors have a different opinion on how to distinguish between emotions, feelings, moods, etc. The authors state that in their work, they use the term affect to include emotions and moods and that’s fine, but I doubt that, for example, “feeling motivated”, or “feeling alone” would be considered neither an emotion nor mood by a lot of researchers in this area.

The other issue is about the sample size of the study and the environment in which the study took place. The authors state that theorist argue that theories should be simple in the beginning and by studying only two developers, they could keep the complex situation simple. Still, I’m a little bit concerned about the very small sample size and that the study took place in a separated work space. The two study participants did not interact with other developers, neither formally nor informally.

In general, the authors describe their experimental design in a very detailed why that would make it possible for other researchers to reproduce it.  However, there are also some minor open questions that would benefit from clarification. First, it would be interesting to know whether the interviews were conducted in English or if the authors translated the participants’ answers. Second, the authors state that they conducted 5 impromptu meetings. Could the authors clarify which events or circumstances lead to calling an impromptu meeting? Third, for the affects, the authors performed a quantitative triangulation. Was there also such a triangulation done for the performance? If not, why? Fourth, in the section “Theory construction and representation”, the authors discuss the issues that come with the development of a theory and state that their study design has taken these issues into account. Could the authors elaborate how exactly their study design has tackled these issues?
Validity of the findings: The findings that the authors present are interesting and appropriately stated. The authors also discuss some implications of their findings for research but also for practitioners.

However, there are also some open questions that the authors should address. First of all, the authors state that affects and focus are in general positively correlated. Did the authors consider that there might also be negative affects that have a positive influence on the focus? For example, a developer might get very frustrated because s/he can’t fix a difficult bug, so that s/he puts a lot of effort in it, leading to a state of high focus that eventually helps to solve the difficult problem. Similarly, in Figure 1, it looks like events always trigger affects, but couldn’t it also be the other way round?

Furthermore, the category “Interventions” in Figure 1 seems to be very specific to this particular study setting, where a researcher could intervene when two bachelor students working on a software project couldn’t cope with each other. What would be the real world analog?
Additional comments: I only have two minor comments: i) in the enumeration starting on line 158, number 3) is missing, and ii) the authors grouped affects in two different categories: positive and negative affects. Later they refer to these two categories as two different dimensions of affects. Is this really correct? For me it looks more like two different values or categories of affects. Wouldn’t dimensions of affects be something like valence or arousal, i.e. different dimensions to measure an affect?


Instruction: You are one of the reviewers, your task is to write a review for the article. You will be given the title of the article, the number of the round in which the article is located, and your order among the reviewers.

Input:
Title: AUDITORY INTERFACES IN AUTOMATED DRIVING: AN INTERNATIONAL SURVEY
Review round: 1
Reviewer: 1

Output:
Basic reporting: The paper contains clear background material that serves as a readable introduction to someone unfamiliar with the field. The English language contains a few minor grammatical errors but is perfectly understandable throughout. The structure of the paper is logical and well organised. The figures and tables included are clearly presented and necessary to the arguement of the paper.

The aims of the survey are clearly defined.
The case for the use of audio as a communications mode for in-car systems is well made.
The coverage of the advantages of auditory displays is not comprehensive, but adequate.
 Some coverage of the downsides of auditory displays should be added, such as sounds being masked by other sounds, the fact that they are transient leading to them possibly being missed by the driver ...

The description of an earcon is not in line with established auditory display literature (1), and appears somewhat confused with the concept of an auditory icon (1).

In general, a more detailed coverage of relevant previous research on auditory displays (1) would strengthen the theoretical background presented and lay the ground for a more insightful discussion of the options for the design of audio Parking Assistance (PA) and Forward Collision Warning Systems (FCWS).

The paper provides a clear introduction to PA and FCWS and their current limitations.

A figure is given for the number of PA systems currently available but no equivalent is given for FCWSs, though some figures are quoted for a survey of Volvo users.

The details of the survey questionnaire and the results and analysis of these are clearly presented (but see comments on experimental design and validity below).

The references appear correct and complete.
Experimental design: The methods employed would be completely reproducible if desired.
 Appropriate ethical clearance has been obtained for the work.

As the authors acknowledge, a driving simulator based examination of the issues covered in this paper would provide more ecologically valid results, though with the disadvantages of substantial extra cost and having a much smaller number of participants.

Participants were asked about preference for the gender of the voice supporting driver takeover messages. The gender is of course only one characteristic of the voice: natural vs. synthesised, quality of reproduction, level of prosedy, speed of delivery and level of driver configurability are at least some other important parameters to be taken into account. Again these could be tested through simulation.

It seems strange that illustrations were used to represent the 3 scenarios, PA, FCWS and Future System (FS), whereas there sole mode of communication would be audio. This seems particularly flawed in the case of FS, as participants are unlikely to have any previous relevant auditory experience to draw upon. This limitation is acknowledged in the Discussion section of the paper. There may have been technical issues in delivering a realistic auditory representation of FS to participants via the Internet, but there doesn't seem to be any good reason why audio samples of typical PA and FCWS systems could not have been provided. I don't know whetehr CrowdFlower supports audio, but if not, this shortfall in the study design surely made it worth considering alternative means of delivering the survey.

The survey questionnaire employed appears generally clear and unambiguous.
It is however hard to know how respondents could answer the question adequately concerning the surround sound of the FS system without having an auditory example to go on.

In the survey, interpreting the scale for frequency of driving (question 4) does not seem to be entirely straightforward, for example, two participants giving the same values of 2, 3 and 4 are likely to interpret what is meant by those figures somewhat differently. Specifying what is meant by more of the values in the scale could have reduced the scope for different interpretations of those values.

The authors  analysed for English and non-English speaking countries, it would be interesting to know if there was a deliberate decision to analyse in that way as opposed for example to asking a question about the participants first language, or nationality, as clearly someone's location does not necessarily correlate with their nationality or first language.

The authors have gone to some trouble to remove completed questionnaires they had reason to be doubtful about, i.e. where participants had not read instructions, were under driving age etc. This still left a healthy number of completed surveys (1205).

The large number of neutral responses concerning the FS could be down to the lack of a realistic representation and lack of previous experience of such a system on the part of survey participants.

The evidence of the preference for a female voice, both among male and female participants is compelling.

The lack of support for vibrations as  a warning mechanism is interesting.
Validity of the findings: The results of the survey are clearly stated and in general are justified by the data presented.

The conclusions and reflections on the results ar generally  reasonable and the limitations of the survey appropriately acknowledged. There is some speculation in the discussion, but it is clear when this occurs and seems completely reasonable given the nature of the survey and its findings.

The contrast between the lack of enthusiasm for vibrations as a message carrier in the survey results contrasts with the driving simulation study by Adell,  et al mentioned in the discussion. This could be due to the lack of experience of such systems among the survey participants.

The relatively positive attitudes in the survey to the PA andFCWS audio systems (in comparison with other studies) may in part be due to the lack of audio indicatinghow these would actually sound, but negativity inspired by the sounds of current systems can surely be addressed by better sound design, which in turn could be evaluated using simulations.

The preferences for multimodal feedback in PA and FCWS systems is a valuable finding. I do not  agree with the authors that the existence of visual feedbac in existing PAs makes the comments from survey respondents about audio only PAs irrelevant. Considering the arguement in favour of multimodality in this context, there is surely an arguement for PA systems with vibratory output so that hearing impaired drivers benefit from multimodality, particularly in view of the directional nature of the visual sense.

The authors rightly acknowledge the shortcomings of self reporting surveys, and the likely mismatch in demographics between Crowd Flower respondens and the early adopters of Fully Automated Driving (FAD) vehicles. Nevertheless I agree with their overall conclusions about the value of their findings  in supporting the near future iterations of PA and FCWS designs.
Additional comments: Overall this is a highly readable and interesting contribution. I do have some significant reservations about some of the methods employed in administering the survey, particularly the use of illustrations and no use of sound samples to represent the PA and FCWS systems. Overall however I believe the findings of the study to be valid and a useful contribution to the design of future PA and FCWS systems.

(1) Hermann, T., Hunt, A., Neuhoff, J. G., editors (2011). The
Sonification Handbook. Logos Publishing House, Berlin,
Germany, chapter 2, Theory of Sonification, available online at http://sonification.de/handbook


Instruction: You are one of the reviewers, your task is to write a review for the article. You will be given the title of the article, the number of the round in which the article is located, and your order among the reviewers.

Input:
Title: AUDITORY INTERFACES IN AUTOMATED DRIVING: AN INTERNATIONAL SURVEY
Review round: 1
Reviewer: 2

Output:
Basic reporting: I will provide my review as a separate document.
Experimental design: I will provide my review as a separate document.
Validity of the findings: I will provide my review as a separate document.
Additional comments: No additional comments


Instruction: You are one of the reviewers, your task is to write a review for the article. You will be given the title of the article, the number of the round in which the article is located, and your order among the reviewers.

Input:
Title: AUDITORY INTERFACES IN AUTOMATED DRIVING: AN INTERNATIONAL SURVEY
Review round: 1
Reviewer: 3

Output:
Basic reporting: This article presents the results of an international survey about auditory interfaces in automated driving.

It is well written and easy to read.
There is however one sentence which does not seem clear to me, lines 169 to 172 : why such system might lead to changes in traffic regulation and warning emergency systems? What might be these changes?

The authors followed a clearly organized structure, with a complete introduction and a description of the method, before presenting the findings and discuss them. However, the paper lacks a proper conclusion to make the finding more visible for the reader.

This article includes a large number of relevant references. The reference format should be modified to respect the style used in PeerJ Computer Science ((Name,  year) for in-text citations and beginning with the last name in the reference section).  Also, there may be a mistake line 538, supporting that “existing PAs already provide visual feedback” with references 55, 56, 60 dealing with FCWS.

Figures are relevant and well described. Figures 4 and 6 may, however, be difficult to read when printed in black and white. More significantly, some values seem inconsistent between tables 2 and 3. For instance for PAhelp, values are said to be included between 1 and 5 on table 2 but there are only frequencies for answers 1, 2 and 3 on table 3. It is also the case for PApast, FCWSpast and FCWShelp.

Most data have been made available. Only textual comments are lacking, but well described in the document.
Results are clearly described and surely useful for designers or manufacturers of automated vehicles.
Experimental design: Experimental design is clearly described and the investigation rigorously conducted, respecting ethical standards. According to me, it just lacks the description of the FS which was given to the respondents to imagine this auditory display.

The large number of respondents and their diversity provides a broad perspective on the opinion of potential clients of such auditory displays. CrowdFlower respondents might not be representative of the entire population and it could be interesting to compare these findings to wealthier population, maybe more used to such technology (PA and FCWS) and more likely to purchase these vehicles. Nonetheless, this point is well discussed in this paper. It is not compulsory for this study and could be considered as a future development.

Moreover, it might seem delicate to ask people to give their point of view about a technology they only imagine. But this point is also well discussed and the vision of potential users is always instructive. It might have been interesting to compare the findings on PA and FCWS between drivers who are used to these systems and those who are not. Actual users might find it more helpful and, thus, less annoying than non-users.
Validity of the findings: Concerning the data analysis, I wonder if some analyses shouldnot have been extended.
The Spearman correlations given lines 335 and 336 between age and annoyance or helpfulness seem very weak, and might not be statistically significant. Further analyses, like confidence interval or p-values can answer this point.
Similarly, comparisons between male and female respondents or English speakers vs. non-English speakers’ preferences about male or female voice should need further analysis to determine their significance. Confidence intervals or chi square test might answer this point.

Concerning the conclusions, I wonder if the textual comments given lines 426 and 427 reflect more a fear than a real dissatisfaction. Were these respondents common users of FCWS or just imagined than this warning could rapidly become annoying? Are there any changes in the judgments of annoyance and helpfulness between people who use these systems and those who do not?
Additional comments: No additional comments