Instruction: You are one of the reviewers, your task is to write a review for the article. You will be given the title of the article, the number of the round in which the article is located, and your order among the reviewers.

Input:
Title: FUZZY BASED BINARY FEATURE PROFILING FOR MODUS OPERANDI ANALYSIS
Review round: 1
Reviewer: 1

Output:
Basic reporting: The author presents a model for making profiling for modus operandi analysis in criminal investigations. The authors present an automatic model for this problem and then also consider it to address commonly used UCI datasets.
The idea presented by the authors is interesting. They properly motivate their work. They are presenting an automatic system to find similarities between crimes. What I miss from this paper is the lack of comparison against the profiling given by humans, that is, that from the police.
Overall, the method is properly explained and easy to follow. However, it is not clear how they fine-tune the fuzzy inference system. Could the authors provide more insights on this?
It is also not clear how the authors labeled the classification data-set. One of the problems with the paper is that it is not clear whether the data used is enough so as to obtain meaningful conclusions and to be rally useful. They consider only 67 instances with 20 classes (suspects) and one can imagine that the total number of crimes is much higher. As a result, there are classes with only one instance, which are impossible to learn/predict. Does this make sense? I am not sure of that, and even if the paper makes sense the experimental part is quite weak due to these problems.
Moreover, it is not clear how their model is applied in the rest of common data-sets. Given that it is a classification model, why not to try with a greater variety of data-sets? Likewise, why not to test their model against commonly used classifiers? From my point of view, this last part of the paper is poor and a superficial analysis of the results is carried out. Furthermore, I would say that the statistical analysis is not properly carried out, and ranks are not properly interpreted.
The English of the paper should also be revised.
Overall, the paper is interesting for its application, but there is a lot of work to be done on it.
Experimental design: The authors data may be meaninigless so as to achieve meaninful conclusions. The experimental framework and analysis of UCI data-sets is not properly carried out and the statistical tests are misinterpreted.
Validity of the findings: I am not sure of the real validity of their findings due to the low number of instances considered (67, with some classes having only 1 instance).
Additional comments: No additional comments


Instruction: You are one of the reviewers, your task is to write a review for the article. You will be given the title of the article, the number of the round in which the article is located, and your order among the reviewers.

Input:
Title: FUZZY BASED BINARY FEATURE PROFILING FOR MODUS OPERANDI ANALYSIS
Review round: 1
Reviewer: 2

Output:
Basic reporting: Section "Related work": -
 - clearly separate (in several paragraphs) data mining techniques: Association rules , Link analysis, Classification
 - mention in particularly the distance metrics between binary vectors proposed in the literature and cite them when introducing DP and CP

Rows 196-197: replace "materials" with "systems"

Section "Generating the dynamic MOs.." :
 - if DMO is related to a particular criminal, then replace in row 292 "second criminal" with "same criminal"
 - no need of equation (2) - it's sufficient to mention that M_D is the median of D sequence

Section "Finding deviation ...":
 - change in Eq (6) "x_i - y_i" with (x_i = ) AND (y_i = 0)
 - row 350 - there is no table 7 here
 - row 355 - replace "Equation" with "expression"

Sometimes the information is repeated, as in the rows 549 - 552.
Experimental design: No comments
Validity of the findings: No comments
Additional comments: Re-read carefully the text. Sometimes the predicate is missing.


Instruction: You are one of the reviewers, your task is to write a review for the article. You will be given the title of the article, the number of the round in which the article is located, and your order among the reviewers.

Input:
Title: FUZZY BASED BINARY FEATURE PROFILING FOR MODUS OPERANDI ANALYSIS
Review round: 1
Reviewer: 3

Output:
Basic reporting: The paper is well written and structured. The abstract and introduction reflects clearly the aim of this practical paper.

The bibliography must be updated, only one submitted reference in 2015 is referenced. Most of references are to dates earlier than 2011. Reference 49 must include the year of publication.
Experimental design: No comments
Validity of the findings: No comments
Additional comments: In this paper, a method for identifying modus operandi of criminals is proposed. The method is based on a "binary feature vector profiling" where relationships between criminal and conducted crimes are analyzed by means of a fuzzy inference system. A crime data set was used in testing and the results show that the proposed method provide good results.

In general, the idea presented in this paper is very interesting and well developed.


Instruction: You are one of the reviewers, your task is to write a review for the article. You will be given the title of the article, the number of the round in which the article is located, and your order among the reviewers.

Input:
Title: FUZZY BASED BINARY FEATURE PROFILING FOR MODUS OPERANDI ANALYSIS
Review round: 2
Reviewer: 1

Output:
Basic reporting: The authors have made an effort to improve the paper. I think that it is much clearer now.
However, I have to object that I cannot really see the necessity of this new method, mainly looking at the results on the general data-sets. They are always worse than previous ones when looking at the statistical comparison. Hence, why not to apply previous methods to their problem instead of inventing some new complex algorithm?  I cannot really see this point. I would expect an application paper where they consider different existing algorithms, and in case that the new one is better in that scenario, which would make much more sense.
Moreover, why are they considering the RMSE? Being a classification model, it does not make much sense …
With respect to the problem of classes with only one instance, I continue not understanding why this is not a problem. I cannot understand how this can work in a classification scenario where cross-validation is used to validate the models.
Experimental design: It is much better now, but I do not understand why they considered the RMSE. Also, they did not show the p-values of the statistical tests. The problem with classes with one instance still continues.
Validity of the findings: Not clear due to the data-set used.
Additional comments: No additional comments


Instruction: You are one of the reviewers, your task is to write a review for the article. You will be given the title of the article, the number of the round in which the article is located, and your order among the reviewers.

Input:
Title: FUZZY BASED BINARY FEATURE PROFILING FOR MODUS OPERANDI ANALYSIS
Review round: 2
Reviewer: 2

Output:
Basic reporting: No Comments
Experimental design: No Comments
Validity of the findings: No Comments
Additional comments: No additional comments


Instruction: You are one of the reviewers, your task is to write a review for the article. You will be given the title of the article, the number of the round in which the article is located, and your order among the reviewers.

Input:
Title: FUZZY BASED BINARY FEATURE PROFILING FOR MODUS OPERANDI ANALYSIS
Review round: 2
Reviewer: 3

Output:
Basic reporting: No Comments
Experimental design: No Comments
Validity of the findings: No Comments
Additional comments: All suggestions mentioned in the previous revision have been correctly addressed by the authors.


Instruction: You are one of the reviewers, your task is to write a review for the article. You will be given the title of the article, the number of the round in which the article is located, and your order among the reviewers.

Input:
Title: SOFTWARE PROCESS IMPROVEMENT: A SYSTEMATIC MAPPING STUDY ON THE STATE OF THE ART
Review round: 1
Reviewer: 1

Output:
Basic reporting: The paper is excellently written, clear and unambiguous.
Experimental design: The research design is rigorous, conforming to all guidelines for systematic mapping. The method is extensively described, research questions clearly identified and subsequently addressed.
Validity of the findings: The findings are valid. More specifically the findings are useful to any researcher in the general area of software process improvement.
Additional comments: There have been so many poorly done systematic literature reviews that it was refreshing to find one that was done well, rigorously and produced valid and useful findings.


Instruction: You are one of the reviewers, your task is to write a review for the article. You will be given the title of the article, the number of the round in which the article is located, and your order among the reviewers.

Input:
Title: SOFTWARE PROCESS IMPROVEMENT: A SYSTEMATIC MAPPING STUDY ON THE STATE OF THE ART
Review round: 1
Reviewer: 2

Output:
Basic reporting: This paper is an extension of the authors' earlier paper providing a systematic mapping of Software Process Improvement updating their previous study findings.
The paper follows the guidelines of SLRs by Kitchenham and Charters as well as SMS by Petersen et al. The authors propose 3 research questions and describe their data collection and analysis procedures in great detail.
Experimental design: The motivation of the study is "to shed light on" the field of SPI and to present the state of the art of SPI. I would have preferred the motivation being related to findings or hypothesis of the previous studies by the authors.

Despite the fact that this paper is an extension of the authors' earlier mapping study of SPI, I find it seriously disturbing that the authors have identical abstract, introduction, related works section, research design, data collection and analysis sections to their earlier paper that was published at ICSSP conference in 2015. First, after having conducted such an extensive literature review, surely the authors can come up with a different abstract and introduction sections with citing different articles. Secondly, the extended paper could focus considerably more on the discussion if it wasn't repeating the first sections of the paper - a reference to the previous paper would have sufficed.
Validity of the findings: The authors describe their findings clearly even though it is a very technical and long section. The Discussion section is particularly valuable to this paper. At the same time, there is a lot of repetition between Discussion and Conclusion sections which the authors should revise.
Additional comments: There are some comments that the authors could take into account in improving their paper:
1. When a research topic is as applied as SPI and has been around for a long time, there could be little novelty in it for research to investigate much further (in Discussion/Limitation sections).
2. In Introduction, the authors say that "we still struggle to answer the question like: What is out there?..." Could you please refer to papers that would support such a claim - who is struggling?
3. About agile SPI, mentioned in Introduction, there should also be a reference to Salo and Abrahamsson whose agile SPI paper was one of the first ones and is heavily cited.
4. In Section 3.5 the authors describe how they called further researchers to confirm their publication classification. How did you select these researchers? How many were there? Did you have anybody from ISO/IEC JTC1 SC7 WG10 that develops ISO/IEC 15504/33000 standards? Both space and nuclear SPI fields are completely missing from the classification.
5. Table 11 - search strings. In the opinion of this reviewer, the authors have mixed up process models with methods, and assessment methods with improvement methods.
SPICE/ISO/IEC 15504 should have been in the S4 string rather than in S8 as it is, first and foremost, a process model (having both a process reference and a process assessment model in it) rather than a method. SPICE has a 7-step improvement cycle in ISO/IEC 15504 Part 7. CMMI improvement method is called IDEAL while SCAMPI is the assessment method for CMMI.


Instruction: You are one of the reviewers, your task is to write a review for the article. You will be given the title of the article, the number of the round in which the article is located, and your order among the reviewers.

Input:
Title: TOWARD PHYSIOLOGICAL INDICES OF EMOTIONAL STATE DRIVING FUTURE EBOOK INTERACTIVITY
Review round: 1
Reviewer: 1

Output:
Basic reporting: In this study the authors attempt to determine the affective or emotional state of a well know writer, engaged in original composition, from a range of physiological and neurophysiological signals. The authors attempt to induce and classify emotional states of the writer in response to an IAPS picture set. While successful at separating calibration from writing sessions, the authors were not able to isolate individual emotional states. The research presented in this manuscript is the beginning of an ambitions study to identify/classify the emotional state of readers in responses a written (emotional) narrative. The ultimate goal being an ebook-like system that can adapted to the mental state of the individual reader.
Experimental design: The work described in this manuscript was conducted with an appropriate amount of scientific rigor, given the real-world task and environment. As such, it will contribute to the growing field of applied neuroscience and real-world neuroimaging. However, there are several revisions that would significantly improve this manuscript (see below).
Validity of the findings: The title and abstract are misleading and imply that this study develops/uses an interactive ebook system – this text must be revised. The final use of this technology can be discussed in the intro, or more appropriately, the discussion section. Related to this comment is the structure of the manuscript, which at first reads like a review/theory article. In the introduction alone there are sections that cover ebooks, art and neuroscience, psychology and neurophysiology of emotions (valence and arousal), as well as cognitive processes that underlie creativity. This is too verbose an introduction for a study that only resulted in a differentiation between calibration and writing sessions. While optional, I suggest the authors split this manuscript into a review article and a shorter case study.

The other major concern is related to the neurophysiological analysis described in the manuscript. While the ultimate goal of the study was to classify emotions, the manuscript would be improved by adding a figure that outlines the EEG components that were most discriminative between calibration and writing sessions. What were the most discriminative features/electrodes? This is also related to my concern with the artifact processing steps described in the manuscript. While individual electrodes with high variance (e.g. Oz) were excluded, there appears to be no attempt at removing 1) EoG components and 2) epochs of EMG or other noise artifacts. How can the authors be sure that the difference between calibration and writing sessions was not due to eye or muscle movements (e.g. picture scanning or jaw clenching)? Eye movement metrics especially have been shown to differentiate between even relatively similar tasks, with blinks and saccades often reflected the Alpha spectrum of frontal channels. The manuscript would be significantly improved if these concerns were addressed with the addition of EEG spectral and/or summary figure(s).

Given the results, it would seem prudent for the next study to have a control component/condition that includes non-emotional writing or non-cognitive writing (e.g. copying) to see if these sessions can be differentiated from the emotional ones.

The authors should address the relationship of the present study to Onton & Makeig 2009 (“High-frequency Broadband Modulations of Electroencephalographic Spectra”). Would the author’s classification approach have been more successful if they had used methods similar to Onton & Makeig

The subjective ratings (figure 4) and the participant’s commentary about the comfort and obtrusiveness of the monitoring systems was very informative and relevant for future research in this area.
Additional comments: Minor Points:

The majority of the (vertical) figure axes are unlabeled or without units. Please include labels and units in future revisions.

Typos and Grammar:

Line 36:  change to "Sales of ebooks are..."
Line 175: add commas "...through, for instance, skin..."
Lines 173 through 179: Sentence is unwieldy. Please break into parts or reduce the examples.
Line 188: Begin sentence with "An important step..."
Line 189: either remove the word "potential" or the word "added"
Line 190: add a comma "...is new, and entertainment..."
Line 192: change "artefacts" to "artifacts"
Line 196: remove the word "use"
Line 216: change to "This is a powerful mechanism..."
Line 256: change to "Arnon Grunberg was..."
Line 483: "scientist" should be plural


Instruction: You are one of the reviewers, your task is to write a review for the article. You will be given the title of the article, the number of the round in which the article is located, and your order among the reviewers.

Input:
Title: TOWARD PHYSIOLOGICAL INDICES OF EMOTIONAL STATE DRIVING FUTURE EBOOK INTERACTIVITY
Review round: 1
Reviewer: 2

Output:
Basic reporting: van Erp et al. suggest an interactive form of ebooks. The story would take into account physiological signals of the reader and thereby create an interactive format. Tests of the concept on the author during writing were partly successful, differentiating writing phases from non writing phases. But it was not possible yet to differentiate parts of the story with different emotional content. The manuscript presents an intriguing idea and is well written.

Introduction
line 51 - include an up to date authoritative review, e.g. Dehaene et al. Nature Review Neuroscience 2015
line 74 - recording EEG and further physiological measures of several hundreds of subjects is a lot of work. Make such a promise only if you really follow up on it.
line 100 - this section is devoid of references. Add at least one, e.g. Dalgleish Nature Review Neuroscience 2004 or Tovote et al 2015
line 129 - a side remark: contentment was not listed as a primary emotion.
line 171 - Arousal and emotions also have a significant influence on eye movements. This in turn is important for EEG measurements e.g. Kaspar and König 2012 Front Hum Neurosci
Experimental design: Methods
line 269 - the setup is nice, I like it.
line 298 - You should separate eye movement induced artefacts in the EEG recordings by ICA based methods as in Plöchl et al 2012 Front Hum Neurosci. In principle you can live with indirect measurement of eye movements in the EEG signal, but then you lose most of the information in the primary EEG. In the present case the number of electrodes is moderate and it has to be tested whether enough channels remain to make statements on e.g. DLPFC. At the very least, this has to be discussed. It might contribute to the high performance of EEG features in line 377 or low performance in line 386.
Validity of the findings: line 351 - The statistical methods are valid

Results
ok
Additional comments: Discussion
line 429 - The EEG recordings might be reliable, but wether it is really EEG is a different question. Same issue as above, automated cleaning/separating algorithms are available.
line 462 - “i.e. the writer does not experience a particular emotion himself when writing about it. “ is this so?
The discussion could address methodological aspects. For example increasing the number of EEG channels could be really helpful.

signed Peter König


Instruction: You are one of the reviewers, your task is to write a review for the article. You will be given the title of the article, the number of the round in which the article is located, and your order among the reviewers.

Input:
Title: TOWARD PHYSIOLOGICAL INDICES OF EMOTIONAL STATE DRIVING FUTURE EBOOK INTERACTIVITY
Review round: 2
Reviewer: 1

Output:
Basic reporting: In this study the authors attempt to determine the affective or emotional state of a well know writer, engaged in original composition, from a range of physiological and neurophysiological signals. The authors attempt to induce and classify emotional states of the writer in response to an IAPS picture set. While successful at separating calibration from writing sessions, the authors were not able to isolate individual emotional states. The research presented in this (revised) manuscript is the beginning of an ambitions study to identify/classify the emotional state of readers in responses a written (emotional) narrative. The ultimate goal being an ebook-like system that can adapted to the mental state of the individual reader.

The manuscript meets all the PeerJ basic reporting standards.
Experimental design: The work described in this manuscript was conducted with an appropriate amount of scientific rigor, given the real-world task and environment. As such, it will contribute to the growing field of applied neuroscience and real-world neuroimaging.
Validity of the findings: Minor Concerns:

Some aspects of the introduction present neuroscientific theories or frameworks (e.g. the emotional brain) as more established than they currently are. Many of the components of the introduction (emotion, beauty, creativity) describe research at the forefront of cognitive neuroscience and are, at present, vigorously debated.

The authors are appropriately guarded and circumspect in their interpretation of the frontal delta and theta components. Indeed, these are likely to be eye movement related features. However, the gamma components are also likely to be (at least partially) driven by EMG. Even though the occipital channels were removed to mitigate neck muscle artifacts, jaw clenching and forehead movements can also be reflected in the gamma bands across the scalp.

The embedded image quality was very low making some of the figures very hard to interpret (especially new figure 4). Hopefully, this will be resolved during the publication process.
Additional comments: The revisions in response to the reviewers concerns have significantly improved the manuscript. The (substantial) introduction now provides an appropriate background and literature review for this very ambitious study. The abstract, introduction, results and discussion now flow and provide a more coherent narrative. The expanded results likewise provide more insight into the EEG classification component. My previous concerns have been address and I now support publication of this manuscript.


Instruction: You are one of the reviewers, your task is to write a review for the article. You will be given the title of the article, the number of the round in which the article is located, and your order among the reviewers.

Input:
Title: TOWARD PHYSIOLOGICAL INDICES OF EMOTIONAL STATE DRIVING FUTURE EBOOK INTERACTIVITY
Review round: 2
Reviewer: 2

Output:
Basic reporting: In the first round i gave a rather positive evaluation. With the revision, however, I'm not impressed. The authors were very reluctant to include the suggstions and the manuscript has not been imporved.

The relation to the existing literature is not adequat. Yet, the authors step up their claim by including the claim that "We describe the theoretical foundation of the emotional and creative brain and review the neurophysiological indices that can be used to drive future ebook interactivity in a real life situation."  If this has aspects of a review, then it is mandatory that the current state of the art is referenced and the claims made adapted to what is delivered in the present manuscript.

The nature of the signal measured is a central question. In EEG measurements many artifactual sources are known. Eye movements are a prominent example.  To state (line 406) "Initialy, the EEG data of the remaining channels were used in an Independent Component Analysis (ICA) to identify and remove potential artifacts. However, the ICA revealed that potential artifacts were non-stationary (i.e. changing over time) and therefore difficult to identify and thus no more data were removed. " or in short, there are so many artifacts, we leave them in, is not good enough. I'd expect cleaning the EEG data of artifacts, or in the very least a thorough discussion of the issue with appropriate references to state of the art techniques.

Otherwise it is not clear whether the recordings refer to brain activity (as implied by EEG and BCI) or to eye movements or muscle activity.
Experimental design: ...
Validity of the findings: As long as the issues above are not adequatly addressed, i.e. wether brain activity has been recorded, the validity of the report is questionable.
Additional comments: ...


Instruction: You are one of the reviewers, your task is to write a review for the article. You will be given the title of the article, the number of the round in which the article is located, and your order among the reviewers.

Input:
Title: BIOASSAY TEMPLATES FOR THE SEMANTIC WEB
Review round: 1
Reviewer: 1

Output:
Basic reporting: This paper describes the use of the BioAssay Ontology for describing screening assays. The authors make good points about the difficulty in using and applying biomedical ontologies to scientific workflows. To address this, they developed a software tool which contains a common assay template and is modifiable to suit user's needs. This type of software is valuable for researchers and will help aid in structuring scientific data and ensuring reproducibility.

Figures 1 and 2 are cumbersome to view. I think these could be removed and the authors could point to BioPortal or the OWL file to convey this information to the reader.

The screenshots are low resolution.
Experimental design: For your bioassay template, you need to include detailed instructions on your GitHub page or in the paper on how to access and use the CAT, it was not obvious to me how to open and use the software, and I don't find it very intuitive to use.
Validity of the findings: The goal of this software application is to enable researchers to capture semantic data about their research findings. However, there needs to be detailed instructions on how to use this software tool, as most basic science researchers are not familiar with how to access software via GitHub. I think this could be addressed in the paper and/or the ReadMe file in GitHub.
Additional comments: Add URL in abstract

Line 34: Should define what you mean by screening biologists

Line 38: Do you mean in publications or in the lab notebook, or both?

Line 41: Please clarify here - you are referring to the Materials and Methods section of a publication?

Line 58: Add link to BAO url

Line 92: The OWL file can also be viewed in Protege

Line 103: BioPortal says you have 3340 classes

Line 106: what is the BARD effort?

Line 143: Provide a URL for the GitHub site

Line 254: Should be 'detailed', not 'detail' in "The detail view has a placeholder..."

General comments:
In the BAO, there is a typo in the comment in the class: BAO_0003110 HMS, I think it should be 'Harvard' instead of 'Harward'. Seems like this should be an 'alternative term' instead of a comment, as well.

Where can users make term requests or submit issues to the BAO?


Instruction: You are one of the reviewers, your task is to write a review for the article. You will be given the title of the article, the number of the round in which the article is located, and your order among the reviewers.

Input:
Title: BIOASSAY TEMPLATES FOR THE SEMANTIC WEB
Review round: 1
Reviewer: 2

Output:
Basic reporting: This report is well written and addresses an important bottleneck in the use of domain ontologies to annotate biological data. The manuscript is well organized with clear introduction, methods, results and conclusions. Figure are adequate (some content needs to be improved / corrected). The developed BAT, CAT and software code with examples are made available open source.
Experimental design: The development of annotation software tools is an important area of research and development in the larger area of Semantic Web Technologies and Linked Data in particular in biomedical basic research where complex experiments, technologies and datatypes  are common place. Authors report a simple approach to enable scientists who perform experiments and have little or knowledge of ontologies to make at least some basic use of the BioAssay Ontology to annotate biological assays (here primarily biochemical and cell-based assays)
Validity of the findings: The and methods approach are straight-forward, although I have not reviewed the code. I am very familiar with the BAO (disclosed above) and can therefore judge that the selection of the major category and their implementation are useful and cover basic annotations of straight-forward assays. It does not address a number of more complex annotations as the authors themselves indicate; these could be added in later as the tools evolve. BAO also provides the terminology and modeling to relate assays to each other (for example to describe a screening campaign), for example confirmatory, counter, alternate confirmatory, etc; this an important consideration outside the current solution. It could be mentioned. For reference, we have explained such assay relations and how they can be used to aggregate data in PMID 24078711.
Additional comments: While the content appears overwhelmingly correct, it could be pointed out that the majority over one million assays in PubChem are imported from other resources. This is relevant, because these are not strictly assays, but (for example in case of ChEMBL records) curated descriptions from the literature; they would typically describe the final results / outcome of a study, but not each step on the way (e.g. the screening campaign). While it does make sense to annotate these records using formal semantics, BAO allows to describe screening assays technologically and to some extent operationally, e.g. screening pipelines, multiplexes and parallelized assays. The best example therefore to point out in PubChem are the ones generated in the NIH funded Molecular Libraries Program, specifically MLSCN and MLPCN programs.
In the introduction the complexity of BAO is pointed out and exemplified with figures 1 and 2. Figure 1 appears to include many object properties that are not part of BAO and that are also not used in any assay descriptions in BAO. Many of those appear RO and BFO2.0 relations; some others appear to be made up, e.g.  "has other".
BAO imports a module of relationships from RO / BFO, which can be viewed in an ontology editor. This has to be corrected and I would also recommend to indicate the source of the relations (BAO, RO, OBI, etc), because this is an important feature of  compatibility. The same should be done for figure 2 (class hierarchy). BAO imports modules from many other ontologies (GO, ChEBI, CLO, EFI, DO, etc) and this should mentioned; perhaps where it is suggested that GO could be used to further annotate biological processes (74-79).


Instruction: You are one of the reviewers, your task is to write a review for the article. You will be given the title of the article, the number of the round in which the article is located, and your order among the reviewers.

Input:
Title: BIOASSAY TEMPLATES FOR THE SEMANTIC WEB
Review round: 1
Reviewer: 3

Output:
Basic reporting: The manuscript presents a common assay template (CAT) based on the previously developed ontology BAO and software (the BioAssay Editor) to ease the annotation of bioassays with formally defined BAO labels.

While the presented work on CAT is solid, with the potential users being involved into the development of CAT, there is no sufficient information on how the user requirements for the developed softwarewere identified and how the Editor was evaluated.

The overall logic of the submitted manuscript is not as crisp as one would expect:
-	The authors state that there is a problem (“the major problem”) – a comparison of protocols.
-	They claim that BAO can solve this problem – “in order to address this problem, the BAO was devised.
-	“the BAO remains largely unused”.
-	The presented in the manuscript work will make it easier to use BAO for the annotation of bioassays.

My concerns are:
-	In my view, BAO is not sufficient to solve the problem of accurate comparison of the protocols. The granularity for the representation of experiments is not detailed enough, with some essential for the comparison information still missing in BAO. It can contribute to the solution of the problem, but no one evaluated to what degree. The authors need to make this point clear (since it is about the main problem discussed in the manuscript): provide convincing evidence that bioassay protocols annotated with BAO terms can be accurately identified as “identical, similar, or completely different”, or that BAO offers a partial solution to the problem, that BAO annotation can help with such identification.
-	The manuscript gives an impression (possibly unintentional) that the main drive for the reported work is to make BAO more usable. Instead the motivation should be to solve the stated problem (or contribute to its solution).
-	The conclusions of the manuscript do not refer to the stated problem, how the reported results solved the problem. Instead the conclusion section reports on the solution of another (even if related) problem.

The suggestion is to make the flow of the manuscript more logically coherent with a problem clearly stated, solutions to this problem described and evaluated, and conclusions about the considered problem made.
Experimental design: The process for the development of CAT is well described. 16 CAT were annotated in 81% of the 1066 PubMed assays.
Unfortunately there is no sufficient description of what methodology was used for the software development and how the quality and usability of the Editor was evaluated.
Validity of the findings: The findings are valid and of value to the research community. However it has not been estimated  how the findings contribute to the solution of the stated problem.
Additional comments: The reported work is important and deserves to be published, but it needs to be  better presented.


Instruction: You are one of the reviewers, your task is to write a review for the article. You will be given the title of the article, the number of the round in which the article is located, and your order among the reviewers.

Input:
Title: CAPTURING THE INTERPLAY OF DYNAMICS AND NETWORKS THROUGH PARAMETERIZATIONS OF LAPLACIAN OPERATORS
Review round: 1
Reviewer: 1

Output:
Basic reporting: The theoretical novelty in this paper is to extend the ideas of network centrality, set conductance and its relation to the spectral gap, as defined under the basic random walk on the network, to a more general family of stochastic dynamic processes on the network. These processes are characterized via a generalization of the Laplacian, parametrized by three quantities - scaling and reweighting matrices $T$ and $W$ (which correspond to biasing the walk by increasing the self-loop probability/delay of individual states, or adding edge-weights), and an exponent $\rho$ which allows one to view different dynamics (in particular, 'epidemic' models), as similarity transformations of the random walk transition matrix. The paper discusses how existing models fit within this framework, derive an analog of Cheeger's inequality for these settings, and then applies the clustering algorithms to various networks.

The paper was well written, and enjoyable to read. However I have a few issues with this work, in particular with respect to its relation to some existing works, which I would like the authors to address.
Major issues:
* The main ideas in this paper seem to be to extend the ideas of conductance and Cheerger's inequality to stochastic dynamic processes on graphs. However, there is a very well established line of work on Markov chain mixing times, starting from the seminal papers by Jerrum and Sinclair (M. Jerrum and A. Sinclair. Conductance and the rapid mixing property for Markov chains: the approximation of the permanent resolved, 1988) and Lawler and Sokal (G. Lawler and A. Sokal. Bounds on the $\ell_2$ spectrum for Markov chains and Markov processes: a generalization of Cheeger’s inequality, 1988), which establish such results for any general Markov Chain (also see the book by Aldous and Fill on reversible Markov Chains - all the Markov chains considered in this work appear to be reversible, in particular as the underlying graphs are undirected). I am unsure how this paper's contributions go beyond these known results, and would like the authors to add some references to this area, and clarify the differences.
* Also, the authors reference an earlier work of theirs (Ghosh, R., Teng, S.-h., Lerman, K., and Yan, X. (2014). The Interplay Between Dynamics and Networks: Centrality, Communities, and Cheeger Inequality. ) which introduces a parametrized Laplacian framework - although they claim the new framework is different, I could not figure out the differences, albeit after a quick look at the older work. Any difference should be clarified in more detail.

Minor comments:
* In Markov chain literature, P_{ij} is conventionally used to denote the transition probability from $i$ to $j$, and correspondingly, all state probability vectors (usually denote $\pi_t$) are taken to be row vectors (so $\pi_t = \pi_{t-1}P$). It was very confusing for me to read everything transposed, especially as I had not paid close attention to the definitions at the start - it would be very useful if the authors add a footnote early on warning readers of this difference in notational conventions.
* The set of models referred to as epidemic models appear to be very similar to the de Groot consensus process (see the Wikipedia entry, and references therein) - more generally, such processes are typically referred to as consensus, and the term epidemic is used for percolation-style dynamics. It would be useful if the authors clarify this.
* The authors use $\theta$ to denote very different objects - in case of random walks, it is used to denote the probability vector, while in case of consensus processes, it is used to denote 'belief' vectors. I understand that the authors do so to show the similarities between the two - however, I feel this is more confusing, in particular for readers not familiar with these processes.
* There are some small errors and inconsistencies in the writing. A few examples:
   - (Pg 4) "the entries of diagonal matrix V is the components of the eigenvector V"
   - (Pg 5) "linearized approach to synchronization of different variants of the Kuramoto model" (the Kuramoto model is not defined here)
   - (Pg 5) "Let vol(S) = ∑i∈ di"
   - (Pg 6) "The smallest achievable such ratio is also known as..." - smallest over subsets
   - (Pg 6) "one can then perform a partitioning-based algorithm" - is this a standard term?
   - (Pg 6) "the presence of a good cluster implies " - 'good' is undefined at the moment
   - (Pg 8) "In fact, we can manipulate the adjacency matrix in any way (we can even come up completely new matrices) as long as the result is still an positive semi-definite and symmetric matrix, for any perceived “interaction graph” of dynamics."
   - (Pg 11) "However, the uniform value associated with each vertex is" - this is very non-standard, and seems somewhat forced as a definition.
  - (Pg 12) The summation limits are not clearly specified (for i)
  - (Pg 12) "any symmetric real matrix has an orthonormal basis V which consists of its eigenvectors" The matrix need not be full rank - this can be more correctly stated.
* The PeerJ instructions indicate that acknowledgements should not include any funding aknowledgement.
Experimental design: Not applicable, as the work does not have any experiments.
Validity of the findings: Not applicable, as the work does not present any new data.
Additional comments: No additional comments


Instruction: You are one of the reviewers, your task is to write a review for the article. You will be given the title of the article, the number of the round in which the article is located, and your order among the reviewers.

Input:
Title: CAPTURING THE INTERPLAY OF DYNAMICS AND NETWORKS THROUGH PARAMETERIZATIONS OF LAPLACIAN OPERATORS
Review round: 1
Reviewer: 2

Output:
Basic reporting: I will write the comments on the criteria as stated on the PeerJ website, in the same order.

There are many grammatical errors and typos. Some typos result in mathematically incorrect statements. For example:
- on page 8 \Theta(0) is left-multiply with a matrix, while throughout the paper it is a column-vector and is right-multiplied with a matrix.
 - p. 12 `with a random walk as a standard basis' is mathematically incorrect
- p. 19: `eigenvalue vector'
- p.4: in the second displayed formula \theta_j should be \theta_j(t). A similar typo appears elsewhere as well.
- p.18: `The right hand side of the theorem'

I personally did not like how the citations looked because the authors use a author-year style, but also use the names in the text, so the names are printed double: for example on p.20: `used by Lescovec et al. (Lescovec et al., 2008)'

Introduction implies much greater generaliy of the results than are actually in the paper.

Avrachenkov et al. in a series of papers considered similar parametrization and also discussed \rho=-1/2, 0, 1/2. They also used this approach for community detection, but in a setting of semi-supervised learning, when some representatives of clusters are known. The conductance approach in this paper could be complementary to the approach by Avrachemkov et al. It would be also interesting to compare the results. Definitely this highly relevant work must be properly referenced. E.g.
@inproceedings{avrachenkov2012generalized,
  title={Generalized optimization framework for graph-based semi-supervised learning},
  author={Avrachenkov, Konstantin and Gon{\c{c}}alves, Paulo and Mishenin, Alexey and Sokol, Marina},
  booktitle={Proceedings of SIAM Conference on Data Mining (SDM 2012)},
  volume={9},
  year={2012},
  organization={SIAM}
}
@incollection{avrachenkov2013choice,
  title={On the choice of kernel and labelled data in semi-supervised learning methods},
  author={Avrachenkov, Konstantin and Gon{\c{c}}alves, Paulo and Sokol, Marina},
  booktitle={Algorithms and Models for the Web Graph},
  pages={56--67},
  year={2013},
  publisher={Springer}
}

I find strange the reference to Lambiotte et al. 2011 on page 4. The described process is just a continuous time Markov chain. You can refer to a standard textbook, such as one by Ross or Karlin&Taylor.

I think that on page 10 the authors should not include a citation to their article next to Katz and Brin and Page, because the reference to their own work is rather specific, while the others are more classical and general.

The paper should be drastically shortened and the structure must be improved. In the current version new results start on page 16. Before that, general concepts are explained. Here are my concerns:
- The presented concepts are not new. The \rho-parametrization has been already considered e.g. by Avrachenkov et al. The T-parametrization is a standard approach in continous-time Markov chains. The W-parametrization is just a special case of a weighted graph. An interesting parametrization is a replicator, but it has been already suggested in the authors' previous work.
- The explanation is sometimes confusing. For example, in (21), eigenvectors of which caligraphic L do you mean? Also the formula suggests that z_1 depends on \theta(0), which is obviously not the case. A more clear exposure can be found in the on-line book on Markov Chains by Aldous and Fill, see chapter 3, section 4.
- What is the goal of explaining known results in such length?

Figures are well presented and are interesting.

The submission is mostly self-contained. In a couple of places more explanation is needed:
-  Explain what the term `sweep procedure' means.
- Explain what  the term `whisker' means.

There are typo/errors in the proof of Theorem 1, but I think those can be fixed.
- Why to use subindex caligraphic L in the definition of the volume? It seems to me that the volume as you define it is the same for any operator.
- Does the proof go thourgh only for the L as defined in the formulation of Theorem 1 or for other special cases as well? Same question about Algorithm 1.
- In the formula under line 396 on page 17 you sum over all pairs (u,v), so each pair appears twice, as (u,v) and as (v,u). Then, with W symmetric, the numerator in the second line is equal to zero.

In Algorithm 1 it is not clear whether only one case of L is used or it is applied to all different cases. Therefore, it is hard to understand where the computations for the four special cases differ in the community detection part. Only conductances are different? Or also the sets S_i? Or also the volume? This should be made clear in Algorithm 1 and in the experimental section.

I had difficulty following the text in Section 6. It is too long, and did not make it clear to me why we observe the differences between different procedures.

I found the figures interesting, and the experiments comprehensive and reproducible.

Notations are often confusing. For example:
- In the proof of Theorem 1, d_i is used instead of d_{W_i} (the authors say `with slight obuse of notations'). This is very confusing because d_i has a different meaning. Besides, in the formulation of Theorem 3, the notation d_{W_i} is used again.
- Caligraphic L is used interchangeably to different operators, it is often not clear which one the authors mean.
- Notation \nu is used for eigenvectors, but also for vertices e.g. in the proof of Theorem 1.
- There are many other instances.
Experimental design: From the math point of view, the original part is the proof Theorem 1.

The level of the results is weaker than the problem statement, which is very general. The experimental results are interesting.
Validity of the findings: If the errors/typos in the proof are fixed and the notations are clarified, I expect the results to be valid.
Additional comments: No comments


Instruction: You are one of the reviewers, your task is to write a review for the article. You will be given the title of the article, the number of the round in which the article is located, and your order among the reviewers.

Input:
Title: A TECHNOLOGY PROTOTYPE SYSTEM FOR RATING THERAPIST EMPATHY FROM AUDIO RECORDINGS IN ADDICTION COUNSELING
Review round: 1
Reviewer: 1

Output:
Basic reporting: The abstract and introduction of the article are quite well written. From the abstract it is clear what problem the authors addressed and what results they obtained. From the introduction how their work relates to broader research, and what approach they took. Many manuscripts that I have reviewed fail to meet this standard.

I do have one request though. The authors mention that they propose three quantification models, and then propose to fuse the results of these models. It would be helpful to briefly summarize why the authors chose this approach. Do these models have characteristics that lead one to suggest a priori that this is the best approach to take? Have others employed them for similar tasks? The authors explain this a bit later, but only in the context of a detailed description of the methods.. Also the correspondence between the narrative and the diagram in Figure 1 could be made clearer, since the diagram says “N-gram language model” and the narrative says “Maximum likelihood model training with human-generated transcripts”. I gather these refer to the same thing.

Overall I felt that the body of the manuscript clearly describes the methods used. I found the technique of training role-specific language models to be clever and novel.

I would like to see some clarification of the following points:
- How does the diarization algorithm perform in the case where speakers overlap or interrupt each other? The authors note that this is an infrequent occurrence.

Was it necessary to train an acoustic model specially for this purpose? How much performance improvement does this offer over off-the-shelf acoustic models?
Experimental design: The experimental design is clear, and evaluates the contributions of each component of the method.
Validity of the findings: The results appear to be valid, however I did not examine the modeling algorithms in detail.
Additional comments: In the final manuscript, make sure that all acronyms (e.g., MFCCs, OOV, SRILM) are explained or are obvious in context.

Typos:

p. 3: “clinical trails”

p. 5: “experiment results” -> “experimental results”

p. 9: “highly biased result” -> “highly biased results”

p. 11: “while drops” -> “while it drops”

p. 13: “pathes” -> “paths”

p. 14: “a SP” -> “an SP”


Instruction: You are one of the reviewers, your task is to write a review for the article. You will be given the title of the article, the number of the round in which the article is located, and your order among the reviewers.

Input:
Title: A TECHNOLOGY PROTOTYPE SYSTEM FOR RATING THERAPIST EMPATHY FROM AUDIO RECORDINGS IN ADDICTION COUNSELING
Review round: 1
Reviewer: 2

Output:
Basic reporting: The paper is readable but the English has to be improved. In particular using the first person style “We......” is should be removed. There are also frequent grammatical errors, to many to list; the whole style in which the paper is written should be revised.
From the engineering point of view, the paper does not introduce any novel technologies. Existing methods were put together for a specific application. However, this particular application (automatic rating) could be of interest to many readers.
Experimental design: Good.
Validity of the findings: Good
Additional comments: Please improve English.


Instruction: You are one of the reviewers, your task is to write a review for the article. You will be given the title of the article, the number of the round in which the article is located, and your order among the reviewers.

Input:
Title: A TECHNOLOGY PROTOTYPE SYSTEM FOR RATING THERAPIST EMPATHY FROM AUDIO RECORDINGS IN ADDICTION COUNSELING
Review round: 1
Reviewer: 3

Output:
Basic reporting: Generally speaking, this article is well written. Most parts of the article reach professional English writing standards.

To further improve clarity of the article, I feel that the authors need enrich most of captions in both figures and tables. For example, Figure 2 is quite complicate and its existing caption is too brief to properly guide the readers to fully understand this figure.
Experimental design: Mostly, the experimental design presented in this article meets research community’s standard. However, I feel that the following issues need the author's attentions.

around the line 60, the authors mentioned that they already “quantified prosodic features of the therapist and patient, and … “. I am wondering why the authors only limited their experiments on analyzing ASR outputs rather than considering prosodic cues (could be in a very simple format) given their previous research findings.

regarding 2.4 speaker role matching, I feel that many possible useful cues could be used beyond the current LM only approach. For example, it is possible that a therapist always initializes the dialog and he or she tends to use shorter time compared to a patient during the entire dialog.

regarding section 3, the motivations of using both n-gram LM and MaxEnt model were not clearly introduced. Reader need know why these methods were considered to be useful.

regarding Table 4, on which levels, the VAD and diarization, were evaluated. This seems not very clear from reading the paper.

around the line 415, session level ASR WER has a large variation range (from about 20% to 90%). Since the article focused on predicting empathy based on lexical cues, the accuracy of ASR impacts the prediction accuracy very much. In this sense, I am wondering whether the authors should focus their study on the sessions with good enough ASR WER. It is hard to convince the readers that ASR result with a WER about 0.4 or 0.5 still could be processed by the proposed empathy prediction method.
Validity of the findings: Overall, based on the research questions made and the experiments being conducted, the findings presented in this article are valid.

One issue I spotted is from Table 9. For RP, the Acc on ORA-T case was worse than ORA-D case (79.0% < 86.8%). This seems be against the intuition. Why the result from transcriptions (WER in such case is close to 0.0) could be worse than the result from an ASR output with an averaged WER about 0.4.
Additional comments: No additional comments


Instruction: You are one of the reviewers, your task is to write a review for the article. You will be given the title of the article, the number of the round in which the article is located, and your order among the reviewers.

Input:
Title: A TECHNOLOGY PROTOTYPE SYSTEM FOR RATING THERAPIST EMPATHY FROM AUDIO RECORDINGS IN ADDICTION COUNSELING
Review round: 2
Reviewer: 1

Output:
Basic reporting: The revised article conforms with PeerJ standards. The authors' revisions clarify the presentation, tie the figures better to the narrative, and improve the style overall.
Experimental design: The revisions clarify the experimental design and technical approach. The revised version is fully satisfactory.
Validity of the findings: The revised manuscript presents the findings and conclusions clearly.
Additional comments: It is clear from the revised manuscript that the authors have made a concerted effort to respond to the reviewers' comments on the earlier version of the manuscript.


Instruction: You are one of the reviewers, your task is to write a review for the article. You will be given the title of the article, the number of the round in which the article is located, and your order among the reviewers.

Input:
Title: A TECHNOLOGY PROTOTYPE SYSTEM FOR RATING THERAPIST EMPATHY FROM AUDIO RECORDINGS IN ADDICTION COUNSELING
Review round: 2
Reviewer: 2

Output:
Basic reporting: N/A
Experimental design: N/A
Validity of the findings: N/A
Additional comments: Thanks for the authors' efforts on addressing the comments from the three reviewers. Now, my major concerns to this article have been resolved. I feel that the paper could be published in PeerJ.


Instruction: You are one of the reviewers, your task is to write a review for the article. You will be given the title of the article, the number of the round in which the article is located, and your order among the reviewers.

Input:
Title: DETECTING PERIODICITIES WITH GAUSSIAN PROCESSES
Review round: 1
Reviewer: 1

Output:
Basic reporting: This is a well-written manuscript, with very little to criticise.  It is nicely self-contained, and focused.  The use of iPython/jupyter notebooks is to be commended, as well as the availability of code and data.

Current references to relevant literature seem appropriate.  I did wonder if it might be worth making reference to the "automatic statistician" project (of Ghahramani and others). However, I am certain the authors are aware of this project, and am happy to leave this to their discretion. I was also a little surprised that the authors did not cite more of their own relevant work.

I spotted just a few typos/grammatical errors, listed below:

1. Page 5: "which samples are periodic functions" and "This can obtain by".

2. Page 13: "Although the results cannot be compare to any ground truth"

3. Page 15: "We illustrated three fundamental feature of"

4. Consistency with "arabidopsis" vs. "Arabidopsis" (and italics).  Perhaps not so important for a computer science audience, but it is nevertheless nice to be consistent.  [Since arabidopsis has become the common name for A. thaliana, I personally think neither capitalisation nor italics are required if the name is just given as arabidopsis  -- which looks to be the case throughout this manuscript].
Experimental design: The experimental design appears sound, with the work conducted rigorously and to a high technical standard.  Reproducibility is ensured through open-source implementation, and the use of iPython/jupyter notebooks.

I am certainly not an expert on reproducing kernel Hilbert spaces, but I did not spot any obvious blunders.
Validity of the findings: The application of the proposed method to gene expression data is concise and to the point. The comparison to COSOPT is welcomed, and the level of agreement between these methods is reassuring.

I have just one query, which is minor but seems potentially important for future applications of the method.  As far as I understood, the classification threshold for the A. thaliana dataset was determined to ensure that the number of periodic genes was the same as the number returned by COSOPT (i.e. 3504).  But, in practice, how would I determine an "optimal" classification threshold for the proposed method, without running COSOPT?  For this particular example, could it be the case that the "optimal" classification threshold is quite a lot lower than 0.77, and actually all of the periodic genes reported by COSOPT would be reported by the proposed method (if this lower threshold had been chosen)?  That is, could it be the case that the effective difference between the methods (in practice) is one of having different sensitivity/specificity properties?
Additional comments: No additional comments


Instruction: You are one of the reviewers, your task is to write a review for the article. You will be given the title of the article, the number of the round in which the article is located, and your order among the reviewers.

Input:
Title: DETECTING PERIODICITIES WITH GAUSSIAN PROCESSES
Review round: 1
Reviewer: 2

Output:
Basic reporting: This is a self contained paper describing how additive covariance functions for Gaussian Process priors can be used to detect systematic periodicity in observed data. The motivating example of circadian enzymatic control is highly relevant to contemporary cellular biology and a good example of where this sort of statistical machinery proves to be useful.
Experimental design: This is all appropriate for the study undertaken.
Validity of the findings: In terms of the statistical methodology there is sufficient validation of the outcomes.
Additional comments: A reasonably comprehensive and self-contained article that will introduce readers to the Gaussian Process machinery of inference and how covariance functions may be constructed to address a specific study, in this case detecting periodicities. Well written and clear to read.


Instruction: You are one of the reviewers, your task is to write a review for the article. You will be given the title of the article, the number of the round in which the article is located, and your order among the reviewers.

Input:
Title: MATRIX DEPOT: AN EXTENSIBLE TEST MATRIX COLLECTION FOR JULIA
Review round: 1
Reviewer: 1

Output:
Basic reporting: Overall, it's an excellent paper and package.  Please consider
the following issues in the final published paper and code.

Detailed comments:

Abstract:
The three goals are excellent: (1) the user can add their own
test problems and use the same framework,  (2) one framework
with (2a) parameterized matrices (which can be generated),
(2b) regularization problems, and (2c) matrices from a mix
of applications and (3) uses Julia and its support for multiple
numeric types.

I don't understand, at least at the abstract, why (2a) and
(2b) are different.  It seems to me that the regularization
problems are simply a special case of the generatable, parameterized
matrices.  That is, (2a) and (2b) seem the same to me.

(2b) just seems to be a particular set of generatable matrices,
which also have the right-hand side b and given solution x.
But other matrices in (2a) could presumably give A,b, and x.
Likewise, many matrices in the UF collection also have both A and b.

Why treat (2b) as anything special?  It seems to me that your
matrix depot will break when someone wants to include matrix
problems from another domain that include more than just
r.A, r.b, and r.x (why not r.eigs, ...?   r.coord for
matrices arising from a 2D/3D discretization, for which you want
to keep the 2D or 3D coordinates of each node/row/col of the matrix?).

For issue (3), I'm interested to know what you will do with a matrix
from (2c) the mix of applications.  Suppose you have a matrix given to
you with a fixed precision.  What would it mean to (say) create
a quad precision version of that?  When the matrix only has
64 bit floating-point values at best?

Similarly, what if you ask for a matrix from the UF
collection in rational form?  Do nearest rational representations
get created?  That sounds like it could be numerically hazardous,
unless you use a lossless translation from double precision to
a ratio of integers.  The rational format would be really ugly
for 0.3333333333 which might be one epsilon away from 1/3.

As an aside, I will be renaming the "University of Florida
Sparse Matrix Collection" to "The SuiteSparse Matrix Collection
(formerly known as the University of Florida Sparse Matrix
Collection)."  The web site will move from its current ufl.edu
address to a new one at tamu.edu.  The content of the collection
will remain the same.  Can Julia be easily modified, perhaps
by the user, to reflect a change in URL to the SuiteSparse aka
UF collection?  You might want to make a note of the upcoming
change, in citation [8] perhaps.  I will likely be able to
continue to mirror the collection at both sites, however.

Perhaps a comment is useful: "In case the URL of the
external collection changes, you can give MatrixDepot the new
URL by ...".

Does the Julia download of matrices from the UF collection
preserve the meta data in each UF Problem struct?
Such as "notes"? "title"?  "author"? etc?  Or does that get
lost?  Some Problems have problem-specific meta data.
For example, the IMDB movie database has names for each
row and column of the matrix.  Another Problem (Moqri/MISKnowledgemap) is a set of documents, and the abstract of each document is included in the Problem. (matrix ID 2663).

Having read through the paper, I see that my questions
are unanswered.

It would be important to preserve the metadata for matrices
in the SuiteSparse aka UF Collection.  Some information
("notes", "author", "title", "id", and others) are in
all matrices. Other data is problem-specific.

I see you have a way of defining r.x for a regularization
problem.  Why not extend this to return all the metadata

    r = maxtrixdepot ("HB/arc130") ;

which gives:

    r.A     the sparse matrix
    r.title the title in the SuiteSparse aka UF collection
    r.b     the right hand side (if it exists)
    r.notes the notes about the matrix
    r.kind  the kind of matrix

and so on.  If a Problem has extra, non-standard information,
it goes in the 'aux' field, which could be either:

    r.aux.c
    r.aux.lo
    r.aux.hi
    r.aux.z0
    r.aux.coord
    r.aux.actorname
    r.aux.nodename
    r.aux.whateveryoufindgoeshere

or perhaps a flattened structure is fine too:

    r.aux_c
    r.aux_lo
    r.aux_hi
    r.aux_z0
    r.aux_coord
    r.aux_actorname
    r.aux_nodename
    r.aux_whateveryoufindgoeshere

The r.aux.c, lo, hi, and z0 are used for linear programming
problems.  r.aux.coord is for 2D or 3D coordinates, if the
problem comes from a 2D/3D discretization.  I don't have
many matrices with 2D/3D coordinates, but that info is very
important to some methods.

All of my linear programming problems have c, lo, hi,
and z0.  All of my model reduction problems have yet another
set of common 'aux' fields.  And so on, just like your
regularization problems, but extended to more than just one
class.

Since you can already have r.A, r.x, and r.b, it seems like
it would not be hard to extend this to r.anything.

Can that extension be done dynamically, without the need
to modify Julia?

I'd really like to see a format that can accommodate more
than just regularization problems.

In my collection, I keep each matrix and all its meta
data in 3 formats (MATLAB, Matrix Market, and Rutherford Boeing).
When I generate the 3 formats, I ensure that each format
contains exactly the same information, down to the very
last bit.  There is no O(eps) variation between the matrix
values, for example.  I preserve all the metadata too.
The Julia Matrix Depot would drop the meta data.

What do you do with the explicit zeros that are present in
some sparse matrices?  Those are in the *.mtx file, for
the Matrix Market format.  I assume you download that copy.
Are they preserved in Julia?  This is a minor nuance, but
an important one.  You don't have to address this in the
paper; just in the code.  I preserve them for MATLAB by
including another binary sparse matrix, Problem.Zeros,
which is 1 in the (i,j) position if the given matrix has an
expliticly provided entry whose value is zero, in that position.
Does that information get preserved?  It's important to keep it;
that structure is important to the matrix problem.

The overall gist of my review is this.  The matrix depot for Julia is user-extendible, which is great.  Can the *content* of each matrix problem also be extendible, to include extra meta data that is specific to each problem?  Can this be done without needing to rewrite the matrix depot interface?  If so, the 'regularization problem' becomes just one in a host of possible special problems.  Is that possible in Julia?  I think such a framework would greatly strengthen the power of this matrix depot.

Overall, it's an excellent paper and package, and I look forward to seeing the final paper.  Please consider these issues in the final published paper and code.
Experimental design: no applicable.
Validity of the findings: no comments.
Additional comments: I wrote a short MATLAB script that queries each problem in the UF Collection.  It lists all the various top-level fields, and the kinds of aux fields that are currently in use.  I can't seem to attach it to this review, however.  Here it is below:

clear
index = UFget ;
nmat = length (index.nrows)

fields = { } ;
counts = [ ] ;

for id = 1:nmat

    % get all the field names from the Problem
    Problem = UFget (id, index)
    f = fieldnames (Problem) ;
    if (isfield (Problem, 'aux'))
        auxfields = fieldnames (Problem.aux)
        for i = 1:length (auxfields)
            auxfields {i} = [ 'aux.' auxfields{i} ] ;
        end
    else
        auxfields = { } ;
    end
    f = [f ; auxfields]

    % set union, but keep the counts
    for i = 1:length (f)
        k = find (ismember (fields, f {i})) ;
        if (isempty (k))
            fields = [fields ; f{i} ] ;
            counts = [counts  0] ;
            k = length (fields) ;
        end
        counts (k) = counts (k) + 1 ;
    end

end

save getallfields fields counts

fprintf ('\nTotal number of matrices: %d as of %s\n', nmat, date) ;
for i = 1:length (fields)
    fprintf ('%5d : %s\n', counts (i), fields {i}) ;
end

Here is the output, where I sorted the results as well:

Total number of matrices: 2757 as of 22-Jan-2016

Every problem has these fields:

 2757 : title
 2757 : name
 2757 : kind
 2757 : id
 2757 : ed
 2757 : date
 2757 : author
 2757 : A

Then each problem may have these fields.  For instance,
1717 of them have Problem.notes.  These fields are sometimes
vectors or matrices (sparse or dense), scalars, or text.

 1717 : notes
  889 : b
  789 : aux
  417 : Zeros
  346 : aux.c
  342 : aux.z0
  342 : aux.lo
  342 : aux.hi
  187 : aux.coord
  101 : aux.nodename
   95 : aux.rowname
   91 : aux.mapping
   63 : aux.B
   62 : aux.C
   60 : aux.E
   50 : x
   50 : aux.population
   50 : aux.area
   31 : aux.A
   27 : aux.b
   16 : aux.cname
   14 : aux.M
   12 : aux.G
   11 : aux.iv
   10 : aux.Gname
    6 : aux.solution
    6 : aux.smooth_number
    6 : aux.pubyear
    6 : aux.gcs
    6 : aux.factor_base
    5 : aux.colname
    4 : aux.year
    4 : aux.nodevalue
    4 : aux.K
    4 : aux.cluster
    3 : aux.TSSOAR
    3 : aux.partition
    3 : aux.Mzeros
    3 : aux.guess
    3 : aux.Gcoord
    3 : aux.date
    2 : aux.Zeros
    2 : aux.w0
    2 : aux.subcat
    2 : aux.shift
    2 : aux.S
    2 : aux.country
    2 : aux.class
    2 : aux.category
    2 : aux.cat
    2 : aux.b2
    2 : aux.b1
    2 : aux.appyear
    1 : aux.Year
    1 : aux.Varnoldi
    1 : aux.Vansys
    1 : aux.Topics
    1 : aux.T
    1 : aux.t
    1 : aux.Source
    1 : aux.servicecode
    1 : aux.service
    1 : aux.roots
    1 : aux.rootname
    1 : aux.Q
    1 : aux.ps
    1 : aux.PIN_class
    1 : aux.phdyear
    1 : aux.nodesource
    1 : aux.nodes
    1 : aux.nodecode
    1 : aux.MovieBacon
    1 : aux.Label
    1 : aux.Keywords
    1 : aux.key
    1 : aux.KevinBacon
    1 : aux.inbook
    1 : aux.id
    1 : aux.elements
    1 : aux.edgecode
    1 : aux.Day
    1 : aux.D
    1 : aux.concreteness
    1 : aux.code
    1 : aux.closedform
    1 : aux.CiteCnt
    1 : aux.Authors
    1 : aux.Atop
    1 : aux.Aside
    1 : aux.ActorBacon
    1 : aux.Abstract
    1 : aux.Abottom

I'd like to take a 2nd pass at the revised paper, so I can see how you propose to take into account the presence of these extra components to the matrix problems.


Instruction: You are one of the reviewers, your task is to write a review for the article. You will be given the title of the article, the number of the round in which the article is located, and your order among the reviewers.

Input:
Title: MATRIX DEPOT: AN EXTENSIBLE TEST MATRIX COLLECTION FOR JULIA
Review round: 1
Reviewer: 2

Output:
Basic reporting: The manuscript passes all essential criteria, excellent exposition, sufficient introduction and background. The subject itself is useful to a broader range of researchers for the years to come.
Experimental design: Pass
Validity of the findings: Pass
Additional comments: No additional comments


Instruction: You are one of the reviewers, your task is to write a review for the article. You will be given the title of the article, the number of the round in which the article is located, and your order among the reviewers.

Input:
Title: PROBABILISTIC PROGRAMMING IN PYTHON USING PYMC3
Review round: 1
Reviewer: 1

Output:
Basic reporting: This article is well-written.  It is clear, understandable, and
self-contained.  The prose is tight; the examples useful, and the
figures tidy.

I only found one error, which is on page 14: r_t should equal "e" if
t>=s, not "r".
Experimental design: This article should be considered documentation for a software
package.  It is not "scientific", in the sense that there is no
clearly defined research question, at least not which is made explicit
in the text.  Rather than closing a "knowledge gap", as described in
the editorial review standards, this software closes a "practice gap".

The implicit aim of probabilistic programming is to cleanly separate
model-building and inference, and make probabilistic modeling more
accessible to a broader audience by reducing implementation
complexity.  This software package does indeed make strides towards
that goal, and does so with clarity and high standards.

It is not clear if software documentation is within the scope of PeerJ
or not.
Validity of the findings: There are no data, conclusions or experiments.  Since this is software
documentation, the paper mostly consists of details of how to install
it, and examples of how to use it.

Very few implementation details are given, although the high level
bits are included (object-oriented interface to constructing models;
Theano for automatic differentiation; NUTS+HMC for inference on
continuous variables; basic MCMC for sampling discrete variables).

The paper does not really contain enough information for an individual
to replicate it, but it's not clear that "reproducibility" is the
correct rubric for this sort of work.
Additional comments: In my opinion, the PyMC3 package should not be considered a "probabilistic programming" language; rather, it should be considered an API for constructing graphical models.

I draw a clear distinction between the two; I reserve the phrase "probabilistic programming" for systems which are "turing-complete", in the sense that they can model nonparametric distributions, recursive distributions, programs that can write programs, higher-order functions, and the inclusion of arbitrary (stateless) deterministic functions in the middle of probabilistic models.  As far as I can tell, PyMC3 does not support any of these.

PyMC3 seems much more comparable to, say, BUGS or BNT than to, say, Church or IBAL.  I would therefore not call it "probabilistic programming" at all.

I would strongly encourage the authors to change the title and introduction to reflect this, to help keep the terminology consistent throughout the community.


Instruction: You are one of the reviewers, your task is to write a review for the article. You will be given the title of the article, the number of the round in which the article is located, and your order among the reviewers.

Input:
Title: PROBABILISTIC PROGRAMMING IN PYTHON USING PYMC3
Review round: 1
Reviewer: 2

Output:
Basic reporting: Key prior work is not mentioned or contextualized. For example, Figaro is a Scala-embedded probabilistic programming system that exposes customizable inference strategies; WebPPL is another embedded language (embedded into JavaScript); Picture is another embedded language (into Julia); and Quicksand is another (into Lua).

Figure 6 is of very low resolution and needs to be redone.
Experimental design: As I understand the scope of the journal, the article may be suitable for publication as a PeerJ Preprint (after significant revision), but it does not describe original primary research. There is no research question or knowledge gap that is identified; instead, the paper is (clearly stated to be) a tutorial about a software system.
Validity of the findings: There are no conclusions, only capability demonstrations. The capability demonstrations would be stronger if some data about runtime performance and accuracy were included (including variability across ~10+ replications).
Additional comments: I want to encourage the authors to continue their work developing PyMC3 as a software tool, and to explore the question of what research questions in probabilistic programming they might be able to best study from the vantage point of their platform.


Instruction: You are one of the reviewers, your task is to write a review for the article. You will be given the title of the article, the number of the round in which the article is located, and your order among the reviewers.

Input:
Title: DAE TOOLS: EQUATION-BASED OBJECT-ORIENTED MODELLING, SIMULATION AND OPTIMISATION SOFTWARE
Review round: 1
Reviewer: 1

Output:
Basic reporting: In this work, the author presents DAE Tools, a software for building, simulating, and optimizing models comprised of DAEs. This software takes an approach in which the author combines the versatility of general-purpose programming with an interface resembling that of a modeling language. The software appears mature, and includes significant online documentation and a tutorial. These are all welcome, and contribute to my overall impression that this is a well-developed project suitable for publication in PeerJ Computer Science. However, minor revisions must first be made.

The Introduction provides a detailed description of the state of the field, and shows the relevance of DAE Tools. However, it is also relatively lengthy and verbose, and could be condensed without significant loss of descriptiveness. Lines 192-230 in Features read similarly to lines 20-47 and 59-71 of the Introduction. To address this, I propose condensing lines 31-71 to outline the limitations of existing modeling frameworks and the specific goals of DAE Tools: combining the qualities of domain-specific and general-purpose languages to handle model specification, simulation setup, execution of a simulation and/or optimization, and processing the results. The Features section should (and does) expound on how DAE Tools addresses these shortcomings.

The Architecture and Solution of a DAE System sections are well-constructed. While useful for inclusion in the Documentation (or a supplement), I do not think the class diagrams Figs. 1-7 are necessary for the main text of the manuscript. I did find Figs. 8-11 (UML sequence diagrams) to be useful.

The tone is generally refined, however there are several cases of informal phrases (e.g. "sort of").

The future aims of the project are briefly mentioned in the Main Features section, or alluded to ("only Order-1 DAEs, at the moment"). Are there significant plans for further expansion of this software? If so, it may be worth describing these plans.

Typographical changes:
- "log" is listed twice, line 337 and 379
- The definite article "an" should be used before "API." ("an API")
- C++ and Python should be capitalized throughout the manuscript
Experimental design: In this portion of the review, I have described minor bugs in the software or documentation. None of these changed drastically affect the conclusions of the work.

- SourceForge currently suggests 1.3.0 as the most recent release, yet 1.4.0 is described on the website. Since installation is significantly different for the versions, it would be useful to update SourceForge on the most recent version.

- Documentation describing installation of DAE Tools lists dependencies, but should also list versions (e.g. NumPy >= 1.6.1).

- The Docs (5.2.1) do not include import statements at the top. While it can be inferred, please include ``from daetools.pyDAE import *''

- The Docs (5.2.1 & 5.3.1) do not work in Python as-is. Specifically, self.m = myModel("myModel", "Description") in 5.3.1 is input to the myModel object as myModel(name="myModel", parent="Description"), thus the Parent is rejected.

- Please rename the Listings files (if allowed by the Journal) to replaces spaces with underscores to allow simple execution in Python.

- When running Listings 1 & 2 from the command line, they work well. However, executing these files line by line in a Jupyter console throws an error due to the long printout--is there a way to silence it?

- The listings of code in the manuscript should match the listings of code in the supplementary material, for consistency.
Validity of the findings: As the underlying solvers used in this toolkit are well-established, there is no question of their validity.

The examples used in the manuscript have also been published independently of the software.
Additional comments: Although not essential for publication, I recommend the author to consider using PEP8 (https://www.python.org/dev/peps/pep-0008/) conventions (especially naming styles) in the future for consistency with other Python packages.

Similarly, I would find it useful for this software to be available on GitHub so that users can easily compare versions, submit bugs, etc.


Instruction: You are one of the reviewers, your task is to write a review for the article. You will be given the title of the article, the number of the round in which the article is located, and your order among the reviewers.

Input:
Title: DAE TOOLS: EQUATION-BASED OBJECT-ORIENTED MODELLING, SIMULATION AND OPTIMISATION SOFTWARE
Review round: 1
Reviewer: 2

Output:
Basic reporting: The manuscript describes DAE Tools, a software tool for equation-based object-oriented modelling, simulation, and optimization of systems on differential-algebraic form. DAE Tools combines the strength of DAE modelling languages such as, e.g., Modelica, with the expressiveness and generality of ordinary programming languages such as Python and C++ by providing a modelling language API that can be used from Python or C++. After the introduction the paper focuses on the DAE Tools architecture and the operation of that are performed and the objects involved when a simulation is performed.Then a small example (CopperBlock) is used to illustrate the steps involved in modelling and simulation of DAE Tools. Finally some other DAE Tools applications are briefly described.

DAE modeling, simulation and optimization tools is an area that is currently attracting a lot of interest. Strong points of DAE Tools is 1) the tight integration with Python/C++ and 2) the fact that it is open source. However, there are other such tolls available. JModelica.org is a open source Modelica tool that is very well integrated with Python. In spite of this I cannot find a single reference to, or comparison with, this. Another example is "AP Monitor" which also supports modeling, simulation, and optimization of DAE systems. Still no references or comparisons.

DAE Tools also appears to lack vital functionality for DAEs. For example it currently only support index-1 DAE systems. Competing systems such as JModelica.org supports high-index DAEs through index reduction using Panetlides algorithm and dummy derivatives. Also, it does not seem as if DAE Tools currently supports discrete-time sampled systems. This is problematic since a major use case for DAE tools is simulation of closed control loops where the controller are implemented in discrete-time.

The manuscript is dominated by large figures containing UML class diagrams and UML sequence diagrams. I doubt that this is really useful for the readers. The same holds for the code examples. They are nice, but why duplicate them in both Python and C++. For a scientific paper like this would be enough to show the examples in one language, e.g., Python and then simply say that one alternatively could use C++ to express the same thing. It would have better if this page space had been spent on better describing the functionality provided by the modeling language API.

The final "Applications" section is too short to really explain the applications
Experimental design: No Comments
Validity of the findings: No Comments
Additional comments: Please focus your paper on 1) what is unique with DAE Tools from a DAE modeling point of view and 2) compare with alternative tools such as JModelica.org and AP Monitor. Focus less on the architecture.


Instruction: You are one of the reviewers, your task is to write a review for the article. You will be given the title of the article, the number of the round in which the article is located, and your order among the reviewers.

Input:
Title: DAE TOOLS: EQUATION-BASED OBJECT-ORIENTED MODELLING, SIMULATION AND OPTIMISATION SOFTWARE
Review round: 1
Reviewer: 3

Output:
Basic reporting: This manuscript describes the architecture, design philosophy, and structure of DAE Tools, a package for modeling and solving simulation and optimization problems. DAE Tools combines strengths from modeling languages and general-purpose programming languages to allow runtime model generation and simulation set-up, complex runtime operating procedures (for hybrid discontinuous systems), model exchange and co-simulation, etc.

The project itself seems quite interesting and the manuscript is mostly well written and organized.  As such I recommend the submission to be accepted, pending some minor revisions, as listed below.

See "General Comments for the Author"
Experimental design: See "General Comments for the Author"
Validity of the findings: See "General Comments for the Author"
Additional comments: Main comments/questions

1. My main concerns relate to the possible loss of flexibility in modeling and simulation when compared, for example, with user-defined models simulated with libraries such as PETSc or Sundials.  The tools provided by such libraries provide extensive user control which does not seem to have been completely exposed in DAE Tools.  Using the IDAS solver in Sundials as an example (since this appear to be the workhorse in DAE Tools), I hope that the author can provide clarifications on the following comments:

1.1 From the UML diagram in Fig. 10, it seems that the only user-defined functions relate to the definition of problem parameters, variables, and equations. However, IDAS itself provides more flexibility, e.g. specifying Jacobian-related information. Does DAE Tools expose this capability? Or does it always rely on AD-generated Jacobians? There are situations when an exact Jacobian may be overkill and a user-supplied Jacobian approximation (especially for large, multi-scale problems) leads to much more efficient solutions.

1.2 The statement on line 186 seems to suggest that DAE Tools also exposes the sensitivity analysis capabilities of IDAS.  However, no further mention is made of this. If calculating sensitivities is supported, how are the sensitivity equations generated? IS it also AD-based? Also, what are the limitations in terms of problems that can be addressed (given that IDAS cannot currently perform sensitivity analysis of hybrid discontinuous systems)?

2. There is no mention of support for parallel computation (either distributed or shared-memory), a main area of applications for both PETSc and Sundials, which were designed with large-scale problems in mind.  Are there any plans on developing DAE Tools along those lines?

3. The accuracy and efficiency comparisons between DAE Tools and Matlab's ode15s (lines 728 - 736) are questionable. First, it is mentioned that the two integrators were using "default solver tolerances for both" and that the "simulation outputs were indistinguishable".  Since both ode15s and IDAS allow user specified tolerances, why not set them equal to each other and then perform a more quantitative comparison of the achieved accuracy? Furthermore, there are many other possible reasons for efficiency differences between the two beyond those listed in the manuscript (algorithmic differences and language differences being two of them)

Minor comments

1. The very first classification of mathematical modeling (lines 10-19) differentiates between modeling languages (Modelica, gPROMS, etc.) on one hand and what I view more as solver libraries (PETSc and Sundials) on the other hand.  Indeed, neither PETSc nor Sundials really offer proper modeling support (a view that seems to also be acknowledged by the author later on, see lines 70-71).  In this case, is this classification really appropriate?

2. Is the author aware of the Assimulo project?  If so, what (if anything) is common between these two efforts of providing simpler interfaces to solvers such as those in Sundials?

3. The term "degrees of freedom" (lines 242-244) is used here in a rather unconventional sense. While one can better infer what their meaning is herein (from the description of the CopperBlock simulation), it may be useful to clarify that early on, when the term is first introduced (or else maybe come up with a better name).

4. Since DAE Tools relies on a "heavily modified ADOL-C", what (if any) are the plans on staying up-to-date and incorporate possible new releases of ADOL-C?

Language, grammar, manuscript structure and organization

1. While the manuscript reads pretty well, there are places that sound as if they were written by someone else! The Abstract and Conclusions sections, as well as the last paragraph in the Introduction, stand out in this respect. In particular: using past tense in the Abstract and at the end of the Introduction section (before anything has really been presented); very repetitive statements made in the Conclusions section.

2. There are numerous missing or incorrectly used definite and indefinite articles throughout the manuscript.

3. The description of the various interfaces in the six DAE Tools packages (in the Architecture section) is very difficult to parse (the one for the "core" package is effectively a 20-line long sentence).  These may be better presented in tables or else bulleted lists.

4. "linear" is misspelled on line 304


Instruction: You are one of the reviewers, your task is to write a review for the article. You will be given the title of the article, the number of the round in which the article is located, and your order among the reviewers.

Input:
Title: SONIFICATION OF REFERENCE MARKERS FOR AUDITORY GRAPHS: EFFECTS ON NON-VISUAL POINT ESTIMATION TASKS
Review round: 1
Reviewer: 1

Output:
Basic reporting: The manuscript was generally well-researched and well-written.  The topic is interesting and relevant to the auditory display research community, and the execution of the research was competent.  Overall, this manuscript was clear, concise, and enjoyable to read, and I think it offers a worthwhile contribution to the literature.
Experimental design: I have one major concern that, if addressed, could potentially strengthen the precision/clarity of the manuscript. I was not totally clear on the task structure (section 2.2.1).  I think more detail should be provided.  Was the target tone presented visually, with sound, or both? Or did the presentation of the target tone match the presentation format of the test condition (speech for speech, etc.)?  Was a target tone for the reference conditions presented with references or as a single tone without references? As participants responded, did they always see the visual display?  If yes, could they have visually anchored the target and simply used the visual slider to match its remembered position in space? Further, did the method of interaction with the keyboard possibly influence results in addition to the auditory display, since the interaction method was (necessarily) slightly different across conditions? The task is described as a point estimation task, but, if I understand correctly, the task required: 1) remembering the target tone; and 2) using the keyboard to move the test tone until it matched the remembered target tone.  As such, the current description in 2.2.1 makes the task sound more like a perceptual matching task—sort of like a method of adjustment approach--than a point estimation task. Point estimation usually requires the participant to generate and assign a value to the magnitude/quantity represented by the sonification.  I don’t see this as a critical flaw, but I think a clarification of the task (and if necessary a reconsideration of whether it actually involved point estimation or instead was more of a memory matching task) could strengthen the clarity and precision of the manuscript.  As a related point, I think the Discussion would be strengthened if the manuscript linked the study task to specific potential applications of this type of task.

Minor comments:

The design was described as mixed, but it was more of a series of within subjects designs, with one level of each of those separate designs compared between groups. This was a somewhat unusual approach, because it required all of the work of a factorial design without the benefit of having the extra data for all 9 conditions. The separate speech and visual conditions across groups shown in Figure 1 have no substantive difference between them (if I understand correctly), thus differences in these conditions across groups probably represent unreliability in measurement to the extent that they are different (though they look fairly similar, so that’s good). Because each sonification condition included two control conditions, the analysis also was forced to proceed with multiple individual ANOVAs for each group. A more economical approach, both experimentally and statistically, would have been a 1X5 design (each of the 3 sonification types plus the 2 control conditions), probably in a between subjects design. This comment is more of a suggestion for future research.  I don’t think it substantively affected the interpretation of any of the findings.

Effect sizes would be nice to report for significant results.

In 2.2.4, the DV was the absolute value of the difference, correct? This should be clarified.  Correspondingly, the Y axis in Figure 2 shows negative values that should be removed from the figure if no negative values were possible.

Yalla and Walker used a design that was similar to the manuscript’s one-reference mapping.  They called it a “double-tone” auditory scroll bar. It might be worth consulting:
Yalla, P., & Walker, B. N. (2008). Advanced auditory menus: Design and evaluation of auditory scroll bars. Proceedings of the Tenth International ACM SIGACCESS Conference on Computers and Accessibility (ASSETS2008), Halifax, Canada (13-15 October, 2008). pp. 105-112.

I wanted more explanation/extrapolation after the single sentence that was at the end of 4.1.4.
Validity of the findings: I thought the corroboration and discussion of the results in the context of the debriefing interviews was helpful.  I appreciate that the authors got the most explanatory value from all data they collected.  I also thought the exploratory detailed analyses of response patterns across conditions and positions on the response scale was novel and insightful.
Additional comments: The manuscript was quite well-written and clear overall.  There were a couple of typos that I noticed, however (e.g., “focusing” misspelled in 2.1, sentence fragments beginning with “for example” in paragraph 1 and “for instance” in 1.1, paragraph 2), so the authors will want to give a final proof before the version of record is published.


Instruction: You are one of the reviewers, your task is to write a review for the article. You will be given the title of the article, the number of the round in which the article is located, and your order among the reviewers.

Input:
Title: SONIFICATION OF REFERENCE MARKERS FOR AUDITORY GRAPHS: EFFECTS ON NON-VISUAL POINT ESTIMATION TASKS
Review round: 1
Reviewer: 2

Output:
Basic reporting: There is an additional "Further Discussion" section which appears to deviate from the standard form for Peerj.

There are small typos throughout (at one point we have a '?' in an example).

There seems to be segments of the literature that are missing, in particular work from the augmented reality sphere (which is relevant given the device being used in the experiment).  Examples include Air Pointing by Cockburn, Gutwin et al.,  haptic target acquisition by Fiannca.  These possibly are relevant.
Experimental design: The introduction of speech as a control condition seems to come out of no where.  It is not clear why it was introduced.
Validity of the findings: The data is robust, however the analysis is problematic in a couple of different ways:

1) By separating your variables out into one way ANOVAs you miss the potential for (interesting) interaction effects between the different variables.
2) The introduction of the ranges as a variable is interesting; however, again you are separating each out where there may be interactions that are happening with the other variables.  As a result you are also over testing - each test has the potential of finding an effect and you are compounding that with each of the ranges.  You should be able to integrate these ranges as emergent variables into your analysis.

There is a lack of discussion about limitations in the design, the possible confounds that may impact the results (such as physical muscle memory over the trials).
Additional comments: No additional comments


Instruction: You are one of the reviewers, your task is to write a review for the article. You will be given the title of the article, the number of the round in which the article is located, and your order among the reviewers.

Input:
Title: SONIFICATION OF REFERENCE MARKERS FOR AUDITORY GRAPHS: EFFECTS ON NON-VISUAL POINT ESTIMATION TASKS
Review round: 2
Reviewer: 1

Output:
Basic reporting: I noticed one typographical error that should be corrected in the final version: on page 12, line 5,  "However, previous research have only..." should be "research has."
Experimental design: To the extent possible, the authors have addressed all of my comments in the revision.
Validity of the findings: To the extent possible, the authors have addressed all of my comments in the revision.
Additional comments: I appreciate the authors' thoughtful responses to the comments in my original review.


Instruction: You are one of the reviewers, your task is to write a review for the article. You will be given the title of the article, the number of the round in which the article is located, and your order among the reviewers.

Input:
Title: PHILDB: THE TIME SERIES DATABASE WITH BUILT-IN CHANGE LOGGING
Review round: 1
Reviewer: 1

Output:
Basic reporting: The document is written in a clear and concise way, perhaps a little shorter than usual. It seems to be written as a report instead of an article. Motivation should be extended and its proposal better justified. I consider that the complexity of the installation of a database system is not reason enough to discard it. It will be one of the many other features to take into consideration.  In fact, this list should be added and used to compare PhilDB with other systems in the experimentation section. The organization of the paper is clear and correct but the experimentation should be extended and improved.
Experimental design: Regarding the experimentation, Phildb is only compared to InfluxDB because it is the closest in terms of minimal installation requirements and feature completeness but the paper also mentions SciDB as the only open source solution that addresses the scientific need to track changes to stored time data. Why is SciDB not evaluated in the experiment? Besides, this provides a SQL-like query language which has been always welcome and necessary in order to make the use of these systems easier. Datasets used should be better described and be available; alternatively datasets used in other benchmarks could be also used. The different metrics to be evaluated and the method to do it should be mentioned previously and then discuss the results. The paper does not show clearly how the advantage of including a built-in change logging could affect to other performance parameters of the system.  In short, the benchmark should be carefully planned and run in several time series databases and be reproducible in order to community considers its results valid
Validity of the findings: This issue must be improved as mentioned in the previous section.
Additional comments: No additional comments