Instruction: You are one of the reviewers, your task is to write a review for the article. You will be given the title of the article, the number of the round in which the article is located, and your order among the reviewers.

Input:
Title: CODONGENIE: OPTIMISED AMBIGUOUS CODON DESIGN TOOLS
Review round: 1
Reviewer: 1

Output:
Basic reporting: No comment.
Experimental design: This article describes a simple tool for the design of ambiguous codons for encoding multiple amino acids while minimizing undesirable designs. Additionally, using a specific scoring scheme, the tool can score the ambiguous codons for utilizing codons used preferentially in protein coding genes of a target organism.

The PeerJ Computer Science journal only considers research articles, and this manuscript details the implementation of a software tool that uses a brute force method to rank and suggest ambiguous codons to its user. I personally do not see a research objective, either in the form of a hypothesis that is tested, or an algorithm that offers some non-trivial time and/or space complexity. As such, I do not believe there exists a meaningful research question that is pursued.
Validity of the findings: No research question identified.
Additional comments: The manuscript describes an interesting and well implemented tool that can prove useful to the protein design community. It would benefit from pursuing a research question, such as the success of the methodology and scoring scheme described in experiments involving design and evaluation of protein variant libraries.


Instruction: You are one of the reviewers, your task is to write a review for the article. You will be given the title of the article, the number of the round in which the article is located, and your order among the reviewers.

Input:
Title: CODONGENIE: OPTIMISED AMBIGUOUS CODON DESIGN TOOLS
Review round: 1
Reviewer: 2

Output:
Basic reporting: The paper by Swainston et al. describes CodonGenie, a freely available online tool for designing ambiguous codons to code for defined sets of amino acids. Such a tool is very valuable when constructing protein variant libraries for directed evolution experiments. The ambiguous codons are then ranked according to their efficiency encoding the desired amino acids.
The paper is well written and the language is clear. However, the authors may consider to explain in a bit more detail how they derive the numbers in the sentence in lines 45 – 48.
Experimental design: no comment
Validity of the findings: The authors' approach is to derive suitable ambiguous codons and determine their "quality" is sound.
Additional comments: The web application is very easy and intuitive to use and I would like to commend the authors on their effort to provide a very slimmed down and clean interface that manages to provide the relevant information in a clear and concise manner.


Instruction: You are one of the reviewers, your task is to write a review for the article. You will be given the title of the article, the number of the round in which the article is located, and your order among the reviewers.

Input:
Title: CODONGENIE: OPTIMISED AMBIGUOUS CODON DESIGN TOOLS
Review round: 1
Reviewer: 3

Output:
Basic reporting: In this short paper, Swainston and colleagues describe a new online tool named CodonGenie, for helping practitioners of directed evolution to design their degenerate codons. Such tools are useful, although as noted by the authors (and in my revisions, below), they are not the first to provide one.

Overall, the basic reporting (writing, figure presentation etc.) is suitable for publication in PeerJ, with the following revisions:

1. The first two paragraphs should be rewritten. Currently, the primary point of them seems to be to advertise/cite a lot of the authors’ previous papers. The first sentence (lines 26-28) makes it sound as though directed evolution is the only way to do protein engineering – better to clarify that directed evolution is one approach of many (e.g. site-directed mutagenesis, de novo design, ancestral sequence reconstruction). Similarly, how do the authors imagine that site-directed mutagenesis (defined by Wikipedia, no less, as “…used to make specific and intentional changes to the DNA sequence of a gene”) is a method for directed evolution (line 28)? Do they mean site-saturation mutagenesis here? What is the point of the second paragraph, in the context of CodonGenie? My (re-)interpretation of the first two paragraphs is, “There are a lot of directed evolution techniques that make use of degenerate codons, including site-saturation mutagenesis and a number of newer methods.” The newer methods aren’t “in contrast” (line 35) to site saturation. And rather than citing four of their own papers, the authors might also consider other large-scale mutagenesis methods such as PFunkel (Firnberg and Ostermeier, PLoS One, 2012) and EMPIRIC (Hietpas et al., PNAS, 2011).

2. The calculations in lines 46-48 should either be explained, or deleted. While I understand where 1,048,575 comes from (line 46), I suspect many experimentalists would not (but may wish to know, so they can perform similar calculations on their own experimental systems). On the other hand, I can’t fathom where 15^3 and 4^3 (line 48) come from.
Experimental design: 1. CodonGenie is a really nice little algorithm, but the authors should better explain the gap that it fills. On line 56, they pertinently cite the DYNAMCC algorithm (Halweg-Edwards et al.). In my experience, the other algorithm closest to CodonGenie is AA-Calculator (Firth and Patrick, Nucleic Acids Res., 2008). CodonGenie has clear points of difference to DYNAMCC (which gives you a set of codons with no redundancy, instead of one optimal codon) and AA-Calculator (which gives all degenerate codon possibilities, but no convenient score to choose between them) – point out these differences to the reader!

2. The authors emphasize organism-specific codon design as a key advantage of CodonGenie (e.g. lines 53, 85, 103, 122-124). Can they provide examples where organism-specific codon usage might change the ‘answer’ provided by CodonGenie? Further discussion, and a table or figure, would be justified. There are perhaps half a dozen genetically-tractable species in routine use as hosts for protein engineering and synthetic biology. Could the authors come up with, for example, a case in which one degenerate codon might be best for E. coli, and another for S. cerevisiae? How commonly (or rarely) does the choice of host affect the identity of the top-ranked codon – once in a blue moon, or more often than we might think?
Validity of the findings: There are no data or experiments, however the algorithm and the scoring function are clearly described. The web application appears to be bug free and easy to use.
Additional comments: No additional comments


Instruction: You are one of the reviewers, your task is to write a review for the article. You will be given the title of the article, the number of the round in which the article is located, and your order among the reviewers.

Input:
Title: CODONGENIE: OPTIMISED AMBIGUOUS CODON DESIGN TOOLS
Review round: 1
Reviewer: 4

Output:
Basic reporting: The article is well written and clear.

Two minor comments:

1) The introduction equates protein engineering with directed evolution. Directed evolution is usually used in protein engineering projects, but it is certainly possible to also just do rational protein design without any diversity generation and screening.

2) It would be good to comment briefly on how the approach described by Halweg-Edwards et al. cited in the article differs from the approach described in this article.  This information would aid the reader in deciding which tool to use for a given task.
Experimental design: no comment
Validity of the findings: The results here are a tool that can be used to design variant libraries for protein engineering applications. The method underlying the tool is solid, the web-based tool works and would be expected to provide value for applications. The only data used for the method are publicly available codon usage tables for different organisms.
Additional comments: No additional comments


Instruction: You are one of the reviewers, your task is to write a review for the article. You will be given the title of the article, the number of the round in which the article is located, and your order among the reviewers.

Input:
Title: POWER1D: A PYTHON TOOLBOX FOR NUMERICAL POWER ESTIMATES IN EXPERIMENTS INVOLVING ONE-DIMENSIONAL CONTINUA
Review round: 1
Reviewer: 1

Output:
Basic reporting: The manuscript submitted introduces a 1D power estimation toolbox. It is clearly written and will be relevant for future studies within many fields.

Figures are clear and support the understanding of the paper, while not all y axis are labeled (4, 5, 6, 7, 8, 9,10, 12, 13) and different line styles would help the understanding of graphs when printed black-white.

As I am not as experienced with the mathematics that underpin the ideas behind code, I would have appreciated a bit more detail regarding what findings means - e.g. what values are we looking for?
Experimental design: The research question is well defined and the conclusion answers/addresses the purpose of the paper.

Methods applied in the paper are well described and proven using simulations. Code is made available allowing the paper to be replicated.
Validity of the findings: The processes introduced within the paper are sound and provide the user a ability to easily perform a 1d power estimation of a waveform.

Conclusion are well stated and differences to existing platforms are explained and justify benefit of the introduced methods and hence the need for publication.
Additional comments: Congratulations Todd to a well written and interesting paper.

Below you'll find minor suggestion that will eliminate a few typos and might make the paper a bit easier to read for users from a sport science background.

Could you provide a code with would provide an answer to the number of samples needed for an X% increase of a peak value?

Line 49: consider "q represents the day of the ..."
Line 66: "analytical COMMA one can ...."
Line 69-71: I am not sure if I got this right but there is a comma missing: "behavior COMMA ....."
Line 84: remove "also"
Line 84 and following: you use absolute terms use incorrect rather then clearly incorrect, while incorrect should be replace with inappropriate. This should be adapted in the following lines
Line 107: do you mean "model a signal as ...." ? The same in line 109.
Line 208 and following please review the style of the term DataSample and make it consistent.
Line 301: You might consider to mention / explain Monte Carlo simulation and the Omnibus test before mentioning it - as your techniques will be used in a sport science world.

Figure 12: p = 0.992 is not correct. Please explain null and alternative in the text before using it in the figure. Why did you choose COI = 3? What are the dotted line in the figure?

Line 527: "(see §)" should refer to something
Line 530: consider to omit "understandable"


Instruction: You are one of the reviewers, your task is to write a review for the article. You will be given the title of the article, the number of the round in which the article is located, and your order among the reviewers.

Input:
Title: POWER1D: A PYTHON TOOLBOX FOR NUMERICAL POWER ESTIMATES IN EXPERIMENTS INVOLVING ONE-DIMENSIONAL CONTINUA
Review round: 1
Reviewer: 2

Output:
Basic reporting: In the submitted manuscript, "Power1D: A Python toolbox for numerical power 1 estimates in experiments involving one-dimensional continua", the author presents a computer package called "power1d", which implements analytical 1D power solutions and develops a computational approach to continuum-level power analysis, which allows arbitrary signal and noise modeling.

Use of English is correct, clear and unambiguous. The introduction gives a good account of the statistical and computational problems that arise in continuum measurements. The need for the computational methods implemented in the package is well motivated with real data clearly explained. Figures are well designed, relevant and informative. Literature is relevant and well referenced.

Raw data as well as source code are supplied and well documented into the site of the package.

MINOR POINT:

End of line 45, continuing in line 46, substitute "to not vary completely independently" by "do not vary in a completely independent way".
Experimental design: The author proposes, implement, test and demonstrates a computational approach to continuum-level power analysis that permits arbitrary signal and noise modeling, which is clearly within the scope of "PeerJ Computer Science".

Implementation, dependencies and capabilities of the package are well described in the manuscript and more details are given in the site of the package (http://www.spm1d.org/power1d). The research question is well defined and its relevance is clear, while the solutions implemented in the package are well supported. The research presented fills a gap in the methods for continuum-level power analysis, as the author shows both, in the introduction as well as in the discussion of the results.

The author comprehensively demonstrates the use of the package for distinct scenarios, clearly showing the frameworks where the methods are of application. The use of real data makes the presentation easier to understand for statisticians and other interested in this topic. The use of a random seed makes the results replicable; in my opinion the research was carried out in a rigorous way with very good technical standards. The methods are described in detail, and the availability of the source code makes it possible to evaluate all details of the methods and their implementation.
Validity of the findings: Validations of the methods implemented in "Power1D" are clearly and extensively shown for 0D power, the 1D power with inflated variance and non-central RFT. Comparisons with other software packages are complete and fair; to my knowledge the author surveyed the main alternative packages, stating their limitations, some of which are filled by the methods implemented in "Power1D". In particular, the implementation of arbitrary continuum-level signal and noise modeling, which allows the power analysis though numerical simulation of random continua, is a remarkable characteristic of the described package.

The methods presented by the author and implemented in the package are statistically sound and well presented and exemplified by the use of both, simulations as well as real data. Conclusions are well stated, linked to the original research question and limited to supporting results.
Additional comments: No additional comments


Instruction: You are one of the reviewers, your task is to write a review for the article. You will be given the title of the article, the number of the round in which the article is located, and your order among the reviewers.

Input:
Title: VISUALISING HIGHER-DIMENSIONAL SPACE-TIME AND SPACE-SCALE OBJECTS AS PROJECTIONS TO ℝ3
Review round: 1
Reviewer: 1

Output:
Basic reporting: In this paper, the authors propose the following three methods to visualize n (>3) dimensional, or n-D, objects as 2-D images:

(1) n-D 'long axis' projection
(2) n-D parallel/perspective projection
(3) (Generalized) spherical projection followed by (generalized) stereographic projection

Among them, the last method (3) is a new one, while (1) and (2) are, in some sense, straightforward generalizations of well-known techniques in 3-D (and 4-D) computer graphics.

I agree with authors on the importance of visualization of 4-dimensional data in the knowledge discovery in general. This paper---with the source codes implementing the proposed methods---would be an important contribution to science and technology.
Experimental design: More detailed description of the prototype program would be desirable. For example: Is it possible to interactively specify the view position and angle in 4-D? Is it possible to interactively rotate the object in 4-D? If the answers are yes, how? Some readers would like to know since those 4-D operations are notoriously difficult with the standard user-interface. And how do the authors apply those operations (rotations, walk-throughs, and others) in higher n-D for n>4?
Validity of the findings: no comment.
Additional comments: (I do not insist that the following improvement is necessary. This is just a suggestion for a possible improvement of the manuscript, since the "dimension-independence" is one of the emphasized points in the paper.)

The methods proposed in this paper are formulated in general n-D space for n>=4. On the other hand, the presented examples (Figs. 3, 5, and 9) are all for n=4. It would be nice for readers if authors could include an additional example of higher dimensional visualization, say a simple regular polytope in 5-D that is visualized by recursively applied the proposed method (3) for two times. I know it would lead to a highly complex image, but the complexity itself would convey the challenge of the n-D visualization and the power of the proposed methods to the readers.

--- Much more minor points.

p.3: The matrix T is (n+1)-D, i.e., n-D homogenous coordinates, while the matrix S is written in n-D. This would be confusing for readers who are unfamiliar with computer graphics. A comment would be desirable.

p.12, line 309: I could not understand the sentence "This is necessary because..." and the footnote 8.


Instruction: You are one of the reviewers, your task is to write a review for the article. You will be given the title of the article, the number of the round in which the article is located, and your order among the reviewers.

Input:
Title: VISUALISING HIGHER-DIMENSIONAL SPACE-TIME AND SPACE-SCALE OBJECTS AS PROJECTIONS TO ℝ3
Review round: 1
Reviewer: 2

Output:
Basic reporting: The used English is perfect, and clearly readable.
The number of cited references proves that the authors made a comprehensive professional literature research.
The structure of the paper is suitable, figures and tables support the understanding.
Experimental design: no comment
Validity of the findings: no comment
Additional comments: No additional comments


Instruction: You are one of the reviewers, your task is to write a review for the article. You will be given the title of the article, the number of the round in which the article is located, and your order among the reviewers.

Input:
Title: VISUALISING HIGHER-DIMENSIONAL SPACE-TIME AND SPACE-SCALE OBJECTS AS PROJECTIONS TO ℝ3
Review round: 1
Reviewer: 3

Output:
Basic reporting: In general, the article is well written and clear to understand. The literature, as far as I know, is correct. The article structure is professional, together with figures and tables. The article is self-contained, including relevant results to hypotheses. The formal results include clear definitions of all terms and theorems, and detailed proofs.
Experimental design: This paper presents some original research that fits the Aims and Scope of the journal.
The research question posed in this paper is well defined, relevant & meaningful. Throughout the paper, it is stated how the proposed technique fills the lack of methods to properly visualize nD objects. I think the investigation performed meets all standards about rigorous research to a high technical & ethical standard. The methods are described with sufficient detail and enough information to replicate.
Validity of the findings: The paper has some important positive aspects, but also some clear dark points. See below.
Additional comments: First of all, let me say that I am not an expert on mathematical visualization, so I cannot assess the importance such approach could have the mathematical community, or to society in general. However, I have a large deal of experience with LoD models, urban and building modeling, GIS, BIM and related technologies, so I will do my review from these points of view.

Now, about the paper itself, I would like to commend the authors for their search for new visualization mechanisms for complex models or situations, like the ones proposed. I think this kind of research should always be welcome.

On the other side, although the paper presents some relevant results, their applicability at its current state is quite limited. First of all, the paper promises a system to visualize data with high dimensionality in 3D (Actually, 2D, as this is displayed on a sheet of paper or on a computer screen), but the higher dimensionality the paper demonstrates is 4D, with examples that are already easy to visualize with common tools. Throughout the paper, there is not a single example with initial dimensionality equal or higher than 5D. For this paper to be truly relevant, more complex dimensions should be explored and demonstrated.

The second serious drawback I can see about this paper at this state, is that all interesting examples are limited to LoD visualization, with the only exception of figure 7, which is a trivial example, and Figure 8, which is taken from somewhere else and that it is not much relevant here. Please, provide a larger variety of examples and applications, besides LoD.

The third drawback, in my point of view, is that LoD, the main application demonstrated in the paper, is mainly used in practice for two purposes: to generate different versions of the model for simulations (like the LoDs in CityGML), or for visualization, where adaptive LoD techniques are often used to select a different model (LoD level) according to some criterion like viewer’s distance. Now, this is basically a matter of selecting the appropriate model, and displaying it, so showing all LoD models can be done, either side by side, or with an animation in the case of a continuous LoD. I really do not see the need of a visualization like that, which can become really confusing for a complex object. Perhaps I might be wrong about this last part, so please, provide an example of LoD for a really complex building, also with several LoD levels. Actually, as the paper should be generic, other LoD applications, like for a tree or a character model, should also be shown and compared (observe here I am referring to mode types of LoD examples, not just simple houses, while in the previous paragraph I referred to non-LoD examples). By the way, how would this technique do to visualize a continuous LoD model, like the ones developed in the past decade (see missing reference below).

Other comments:
* please, add the following seminal reference for LoD:
Level of Detail for 3D Graphics, David Luebke, Martin Reddy, Jonathan D. Cohen, Amitabh Varshney, Benjamin Watson and Robert Huebner, ISBN: 978-1-55860-838-2
* Figures 8 and 11 seem out of place, please, remove.
* Figure 10 is trivial. Can you provide a more complex example?
* The word “cumbersome” in the last sentence before the references looks awkward. Probably, saying “not efficient” would have a better impact on the readers.
* Please, remove the trivial explanation about translations and scaling matrices, these are well known. Also, rotations in 4D are also well known, though homogeneous coordinates, so, in any case, provide an n dimensional formulation. However, I think this part is also trivial and can be safely removed.


Instruction: You are one of the reviewers, your task is to write a review for the article. You will be given the title of the article, the number of the round in which the article is located, and your order among the reviewers.

Input:
Title: VISUALISING HIGHER-DIMENSIONAL SPACE-TIME AND SPACE-SCALE OBJECTS AS PROJECTIONS TO ℝ3
Review round: 2
Reviewer: 1

Output:
Basic reporting: No comment.
Experimental design: No comment.
Validity of the findings: No comment.
Additional comments: No comment.


Instruction: You are one of the reviewers, your task is to write a review for the article. You will be given the title of the article, the number of the round in which the article is located, and your order among the reviewers.

Input:
Title: FINDING THE OPTIMAL BAYESIAN NETWORK GIVEN A CONSTRAINT GRAPH
Review round: 1
Reviewer: 1

Output:
Basic reporting: The authors propose the notion of constraint graphs for the task of handling prior knowledge in a Bayesian network, and give indications on the efficiency gains over existing methods. They cite Perrier et al 2014 in which the same task is accomplished by a different method. It must be noted that Perrier et al give explicitly the complexity of the method. Contrastingly, this manuscript begins by calling the new method 'intuitive', and does not mention the complexity, except for the tables and some statements which implicitly say that it is polynomial (46 to 52).

In my view, for the work in this manuscript to be complete, the complexity issue is better to be addressed bit more analytically. It would not be very difficult as the method in this manuscript basically relies on Tarjan's algorithm (139 to 143).
Experimental design: No comments
Validity of the findings: The tables in the results section give considerable evidence for the authors' claim. It would be better to include some stronger evidence. Or the authors may verify it analytically.
Additional comments: No additional comments


Instruction: You are one of the reviewers, your task is to write a review for the article. You will be given the title of the article, the number of the round in which the article is located, and your order among the reviewers.

Input:
Title: FINDING THE OPTIMAL BAYESIAN NETWORK GIVEN A CONSTRAINT GRAPH
Review round: 1
Reviewer: 2

Output:
Basic reporting: * The article fulfills the criteria of PeerJ, except maybe self-containedness: I'd propose that the authors shortly review the usually applied scores for Bayesian network learning, i.e. MDL, BIC, BD, and mention their property that they decompose over the network's nodes.

* Also, I think the dynamic programming method by Malone et al., 2011 should be described in a more detailed way. I see that any algorithm (even a greedy one) could be used, as long as it obeys the constrained graph. However, since the authors assume this particular algorithm, I think explaining it in a more detailed way would make the paper more self-contained and understandable for the non-expert.

* Minor comment: Figure 3 doesn't have a sub-figure C, but the caption refers to it (and it actually should refer to B)
Experimental design: * Data-sets should be introduced.
In Section 4.1, stock market data is used, but the source of this data is not given.
In Section 4.2, the data set is not described at all (synthetic?).
In Section 4.3, the data seems to be random, except that certain variables are clamped to the same values; please describe this process in a more detailed way.
Validity of the findings: Findings are valid. Some more experiments on real data would be appropriate.
Additional comments: The paper is surely somewhat incremental, in light of Fan et al. and Malone et al. However, since novelty is not a primary issue for PeerJ, this should not be a concern here. I find the constraint graph elegant, easy to understand and use for non-experts (in probabilistic modeling), and that the approach interplays nicely with the "spirit" of Bayesian networks (combining expert knowledge and data).

Generally, it would be desirable to see some more experiments with real-world data.

It would also be interesting to see how large datasets can get (in terms of number of variables), when placing stringent but reasonable constraints on the structure.


Instruction: You are one of the reviewers, your task is to write a review for the article. You will be given the title of the article, the number of the round in which the article is located, and your order among the reviewers.

Input:
Title: FINDING THE OPTIMAL BAYESIAN NETWORK GIVEN A CONSTRAINT GRAPH
Review round: 2
Reviewer: 1

Output:
Basic reporting: The authors have properly addressed my concern about self-containedness.
Fig. 3 still contains the minor mistake, but it can easily be fixed without further review.
Experimental design: The data-sets have been described in more detail, so the authors have addressed my concern.
Validity of the findings: The authors have included a small real-world example (feature selection using naive Bayes) which reinforces the validity of their findings.
Additional comments: The authors did a good job in revising the manuscript and I suggest acceptance.

In the final version, please fix the following really minor issues:
* Figure 3 still has a caption mistake
* The Markov Blanket of a BN (defined on page) also includes the co-parents of a node's children.
* On page 8, line 307, grammar: probably "that" needs to be deleted.


Instruction: You are one of the reviewers, your task is to write a review for the article. You will be given the title of the article, the number of the round in which the article is located, and your order among the reviewers.

Input:
Title: SCHOLARLENS: EXTRACTING COMPETENCES FROM RESEARCH PUBLICATIONS FOR THE AUTOMATIC GENERATION OF SEMANTIC USER PROFILES
Review round: 1
Reviewer: 1

Output:
Basic reporting: This paper explains how to build semantic scholar profiles using ontologies and did user level evaluations.
Experimental design: The whole semantic scholar profile system is clearly described, but I did not see any innovation in advancing the state of the art in the domain of semantic web. It is a mere of applying existing technologies. It is also more like a proof-of-concept exercise, as the test data is relatively small.

For user evaluation, the sample group is too small, which also include authors themselves, which is highly bias. The online survey only has 20 researchers evaluate, which is problematic.

There is no comparison among different systems, such as compare your system with VIVO profiling,
Validity of the findings: The validity of the findings is questionable. To me, this is just a proof-of-concept exercise. It lacks of thorough large scale implementation and evaluation.The user evaluation at this small sample is fine, but definitely not ideal. To show the validity, it has to be large scale comparison with similar systems, and large scale user evaluation for at least hundreds or thousands.
Additional comments: In general, I think this paper is a proof-of-concept exercise.

It has the following issues:
- missing reference and evaluation with related systems: such as Aminer, Semantic Scholar, VIVO, Academia Search, Stanford profiling system, harvard also has its own. What are the special contribution/uniqueness from your research.
- how to model expertise, it is not clear in this article on how to address this issue.
- you always use GATE system, but there are many NLP tools, GATE is not the main stream, no justification is provided why GATE has been chosen for NLP tasks
- if you store your triples, Apache Jena is slow, did you try others, how many triples in your system. Is your system just a mock-up system?


Instruction: You are one of the reviewers, your task is to write a review for the article. You will be given the title of the article, the number of the round in which the article is located, and your order among the reviewers.

Input:
Title: SCHOLARLENS: EXTRACTING COMPETENCES FROM RESEARCH PUBLICATIONS FOR THE AUTOMATIC GENERATION OF SEMANTIC USER PROFILES
Review round: 1
Reviewer: 2

Output:
Basic reporting: The article is well written and covers and interesting and important topic that has potentially important and significant impact.  The introduction provides a good background to the area and the previous literature and work by the authors.  The template used is effective and the figures are useful, though I think they could be made more 'stand alone' to encourage more readership.  Relevant data is provided, subject to anonymity concerns,  The work represents and improvement of work previously published by the authors as a conference report, but I think the additional work is significant.
Experimental design: The overall design and executions of the work is sensible and addresses the issues of automated completion of semantic profiles, is described in detail,  clearly and effectively and ethical standards look to have been maintained.
Validity of the findings: The data reported is interesting and useful and reasonable conclusions reported and has relatively limited numbers of people involved.  My concern arrises only from the fact that the group surveyed is purely from one discipline.  I understand why this has been done but I am concerned that the real use of the approaches taken by the authors apply when the the system would be used at scale across a wide range of disciplines as the semantic profiles will be most effective when used to find people in different disciplines to the searcher but having complimentary skills needed for an example for an interdisciplinary project.  I think it would be useful of the authors could add thoughts and any information they may have in this respect.
Additional comments: No comments


Instruction: You are one of the reviewers, your task is to write a review for the article. You will be given the title of the article, the number of the round in which the article is located, and your order among the reviewers.

Input:
Title: SCHOLARLENS: EXTRACTING COMPETENCES FROM RESEARCH PUBLICATIONS FOR THE AUTOMATIC GENERATION OF SEMANTIC USER PROFILES
Review round: 2
Reviewer: 1

Output:
Basic reporting: I think the authors made significant improvement this time, especially on literature review and evaluation.
Experimental design: The authors have made progress to improve the experimental design for evaluation.
Validity of the findings: Findings are still based on a small size of user evaluation
Additional comments: I think the authors made significant improvement this time, especially on literature review and evaluation.

Still some issues remained:
- how to model competence, this is a hard problem, there are many different ways of doing or attempting to model it formally. Domain taxonomies were used, such as MeSH, but still it is hard to model competence across difference domains, there is no integrated taxonomy across different domains. It is not clear how authors think about this issue. Other approaches are using keywords extracted from papers or their publications, which seems doable, again these applications are struggling with the accurate descriptions of competence.
- linked to LOD, it is not clear in this revised version, how they ontology are linked with other ontologies. I see that you are reusing many existing LOD ontologies, which are very good, but just link to FOAF, there is no much you can gain by connecting with LOD. Also most of the LOD datasets are not well maintained, except DBpedia. What are the benefits of linking to them?
- I understand your semantic approach of using RDF and SParql to query, but in this special case, what are the benefits of using RDF and Sparql (which sparql is very different to compose anyway). What is the ultimate goal in your application to use RDF and Sparql, it is not clear in your study.
- again, evaluation is good, but superficial. Using very small users, it is really hard to see whether your system will be useful or add value.

User profiling is a difficult problem. Google scholar, Microsoft academic search, and Aminer are doing such in different ways, but still struggle to provide better services. Your research is a good try, but it is more a proof-of-concept, it is far from a system which can work and add value.


Instruction: You are one of the reviewers, your task is to write a review for the article. You will be given the title of the article, the number of the round in which the article is located, and your order among the reviewers.

Input:
Title: SCHOLARLENS: EXTRACTING COMPETENCES FROM RESEARCH PUBLICATIONS FOR THE AUTOMATIC GENERATION OF SEMANTIC USER PROFILES
Review round: 2
Reviewer: 2

Output:
Basic reporting: The paper is fine in the basic requirements.
Experimental design: This is within the scope of the journal.  The evaluation that has been undertaken is methodologically sound but the concern is with the scope of the evaluation.
Validity of the findings: Within the relatively small scale and limited evaluation the findings are valid.
Additional comments: I was generally favourably disposed towards this paper but concerned with the limits evaluation.  Despite these limits I would have agreed to publication even though proper evaluation would have resulted in a much stronger and more interesting paper.  I have indicated my general suggestion for acceptance below) .  However, in reply to the referees comments the authors place extensive emphasis on the open source nature of the solution they provide and seem to imply that this is the main reason for publication., i.e. to make the code available even if the evidence that it will be really useful is not provided.

Given this then I feel that it may be more appropriate simply to make the software available via GitHub rather than as a publication.


Instruction: You are one of the reviewers, your task is to write a review for the article. You will be given the title of the article, the number of the round in which the article is located, and your order among the reviewers.

Input:
Title: A PROBABILISTIC MODEL TO RECOVER INDIVIDUAL GENOMES FROM METAGENOMES
Review round: 1
Reviewer: 1

Output:
Basic reporting: Overall, the manuscript reads well. The introduction gave a good literature review of the problem of recovering genomes from metagenomic data, which led to the probabilistic model proposed. The structure of the manuscript is clear.

Yet I suggest the authors highlight the advantage of the probabilistic model over other methods in both introduction and discussion section. In fact, some of them were shown amid the main text (Line 316-318, 334-338), but a summary of them would be useful to highlight the contribution of this work.
Experimental design: 1. Methods section needs polishing

1.a    In Line 100, N stands for the number of contigs of a shotgun metagenomic experiment, while in Equation 3,6,8 and 10, N seems to stand for the number of contigs in the genome/bin used to train the parameter θ. A different notation should be used.

1.b    The relation between P(contig_i|genome) (line 112) and the likelihood defined by Equation 1 is not clear, especially given that there is not a variable referring to a "genome" in Equation 1. I suggest putting a subscript “g” to the parameter term θ. This could help readers keep in mind that a θ is trained from a genome (or a bin) g, and L(θ_g|F_i) could unambiguously refer to the likelihood of that contig i originated from genome g.

1.c    Line 200 and 229 said Naïve Bayes models were chosen. However, the models are not Bayesian (also noted by the authors in line 201). They are simply linear models (after taking logarithm to Equation 7 and 9), and should not be described as “Naïve Bayes models”.

2. Questions for the experimental design

2.a  Nielsen et.al (nbt.2939, 2014) found that using the co-abundance across different samples, “species can be segregated accurately using as few as 18 samples”, but not fewer. However, it was noted from line 270 that only four communities of different abundance profiles were simulated in the paper. It is questionable whether four samples are insufficient. BTW, the authors should cite Nielsen et.al’s paper.

2.b    The taxonomy annotation was generated from BLAST alignment, according to the supplement. Yet the reference database was not shown. A proper reference database is very crucial in this experiment. To be realistic, the database used for should discard a significant proportion of genomes used to simulate the contigs, otherwise, the BLAST alignments alone would assign most of the contigs to the correct genomes.
Validity of the findings: 1.    Though it shows that combining the four models resulted in the best MSE and MPC scores, readers are not able to judge how good the aggregate model is. A comparison with other methods (e.g. use Kraken to assign) is welcomed.

2.    In the “Genome enrichment” section, there should be another figure showing the % of the target contigs being filtered v.s. p-value cut-off.

3.  The statement in line 406 “taxonomic annotations, as used in the model, need not be entirely correct, as long as the same annotation method is applied to all sequences” lacks experimental supports. Already noted in my comment 2.b of "Experimental design", the authors need to select a proper database to support the statement.
Additional comments: The process of creating the simulated dataset was detailed in supplement, but it is still useful to put the datasets accessible online.

The implementation MGLEX is open-sourced on Github and very easy to install via python-pip, which is most appreciated by the community. But it lacks basic document or guidance for users to start with.


Instruction: You are one of the reviewers, your task is to write a review for the article. You will be given the title of the article, the number of the round in which the article is located, and your order among the reviewers.

Input:
Title: A PROBABILISTIC MODEL TO RECOVER INDIVIDUAL GENOMES FROM METAGENOMES
Review round: 1
Reviewer: 2

Output:
Basic reporting: my comments are under 'general comments for the author'
Experimental design: my comments are under 'general comments for the author'
Validity of the findings: my comments are under 'general comments for the author'
Additional comments: In their manuscript entitled "a probabilistic model to recover individual genomes from metagenomes" Droege et al describe a model usable to assign contigs to genome bins, as well as evaluating the likelihood that bins are correct. The work is of interest, as genome recovery from metagenomes is only recently becoming a generally used strategy for analysis of metagenomic data.

To provide context for the review: I am a biologist by training, with some experience in analysis of metagenomic data in both a gene and genome centered manner. Most of my comments will thus concern these aspects of the work. I have some remarks on the manuscript, and some on the python implementation of the model.

On the manuscript:

The manuscript describes the model well and is readable for someone who is not intimately familiar with modeling. I was wondering if the implementation of relative abundance as it is described in the manuscript would be sensitive to datasets of uneven sequencing depth. It seems that without standardization between datasets, datasets with more reads wouyld be valued higher, because the variations between genomes in more deeply sequenced datasets is bigger?

I was a little surprised to see that the authors choose to simulate a realistic metagenome, but subsequently choose to avoid a realistic treatment of this dataset. While I see this is valuable in order to assess the performance of the model in a controlled manner, I would like to ask the authors to also show a usecase for their model in which some of the challenges of a real dataset are present. Especially a condition where the training data is not a priori known to be correct (as in a binning of a metagenome assembly) is of interest and I urge the authors to apply their model to either a real dataset, or a de novo assembled simulated dataset in which the available prior knowledge is not used to train the model.

specific comments:
line 28: I think the term binning might have been coined by Banfield and colleagues in their 2004 paper. It would be appropriate to cite that here.
Tyson, Gene W., et al. "Community structure and metabolism through reconstruction of microbial genomes from the environment." Nature 428.6978 (2004): 37-43.

line 80: The authors state that taxonomic affiliation has rarely been used as input in binning. Albertsen et al (2013) do use taxonomic affiliation in their (mostly manual) binning procedure and should be mentioned here. The banfield lab's ggKbase can also use taxonomic info, but I think is not completely open to use and I don't know if there is a paper to cite.

The explanation of using both absolute and relative abundance independently in line 185-189 is not entirely clear to me.

line 228-229 the implementation of taxonomic assignments for contigs is clear, but it is unclear to me what 'level-specific relative frequencies' means for the entries for genomes.

line 253: what is ment by 'amount of contig data'? Total number of bases in contigs or total nr of contigs, or something else?

line 305: missing r in strains

line 305: One of the problems with metagenome analysis is the partial coassembly of (the core genome of) strains, in which case genome abundance is not necessarily capable of distinguishing between strains.

On the python implementation:

Except implicitly on the https://pypi.python.org/pypi/MGLEX website, it is not mentioned that this is a python 3 package, which thwarted my intial attempt at installing it on my macbook

installing the module on our lab server with an anaconda3 environment was straightforward, with the only exception of a permission issue adding lines to easy-install.pth
This might seem oddly specific for a comment, but i encourage the authors to provide a very brief overview of dependencies and installation instructions.

Once installed, the module seems to run fine and the access to the command help is straightforward. However, a general workflow description is lacking. The various commands are listed, but it isn't immediately obvious in which order the commands should be run.

Going through the commands, it seems that 'buildmatrix' is run to generate the responsibility file that most other commands use. It is however unclear to me what the required input 'identifiers' and 'seeds' for this command are, and what file format they should be.
Then 'train' can be used with this responsibility file, to train the model required for 'classify'. As in 'buildmatrix' the input format for the various data required for the model features is unclear.
I suggest that the authors provide a less minimal description of the package use, such that it is accessible to a wider audience in the bioinformatics community


Instruction: You are one of the reviewers, your task is to write a review for the article. You will be given the title of the article, the number of the round in which the article is located, and your order among the reviewers.

Input:
Title: A PROBABILISTIC MODEL TO RECOVER INDIVIDUAL GENOMES FROM METAGENOMES
Review round: 1
Reviewer: 3

Output:
Basic reporting: 1.  In general, the manuscript was well written, with appropriate structure and comprehensive description of the methods and results.

2. In line 355, the sentence of “one would need could quantify bin similarity by direct comparison of
features such as the k-mer vectors or abundances.” is a little bit confusing.
Experimental design: 1. In this manuscript, the authors only use simulated data sets to evaluate the method. It will be useful to show how this method can be applied to some real datasets and how this method performs in a more realistic setting. It may be enough to only show the performance on one application, especially the classification/binning. How does this method further improve the quality of binning results generated by other existing binning tools?

2. With the models built using simulated training set, the best performance of  MPC is 0.68 using all the features. How does this performance compare to other existing binning tools? It will be interesting to see how existing binning tools work on the “testing data”.

3. More descriptions about the implementation will be appreciated. For example, does the implementation include functions to generate the k-mer frequency or coverage information? This can give the audience valuable information about how easily this tool can be integrated into their pipeline.
Validity of the findings: 1. The method described in this manuscript does need  a “training dataset” to build the models. Does this training dataset of sequences with known bins have to be generated by other metagenome binning tools? Is it possible to build the models with simulated data as shown in this manuscript? If this method requires some kind of existing sequence bins to work, this method is different from many existing binning tools, which can do the binning from scratch. And what if the performance of existing binning tools on the data set is not good enough? How will the quality of the bins used as training affect the performance of this MGLEX method?  It will be appreciated if the authors can make this more clear to the audience and discuss more about how to integrate this method with other existing binning methods to achieve better results.

2. Line 290, “The smaller fraction was used for  training because identifying the training data often represents a limiting factor in metagenome analysis”. What does the “limiting factor” mean here? It will be nice to explain the smaller fraction used for training  in more details.

3. In line 416, the authors advise the contigs should be splitted to have a certain length. It seems that this is not discussed in other sections of this manuscript. It will be nice to give more explanations about this, like how to determine the proper length.
Additional comments: No additional comments


Instruction: You are one of the reviewers, your task is to write a review for the article. You will be given the title of the article, the number of the round in which the article is located, and your order among the reviewers.

Input:
Title: A PROBABILISTIC MODEL TO RECOVER INDIVIDUAL GENOMES FROM METAGENOMES
Review round: 2
Reviewer: 1

Output:
Basic reporting: No comments.
Experimental design: No comments.
Validity of the findings: No comments.
Additional comments: The authors have addressed all my comments.


Instruction: You are one of the reviewers, your task is to write a review for the article. You will be given the title of the article, the number of the round in which the article is located, and your order among the reviewers.

Input:
Title: A PROBABILISTIC MODEL TO RECOVER INDIVIDUAL GENOMES FROM METAGENOMES
Review round: 2
Reviewer: 2

Output:
Basic reporting: -
Experimental design: -
Validity of the findings: -
Additional comments: In my opinion the authors have adequately addressed the my comments and those of the other reviewers, and thus recommend the manuscript for publication. I do urge the authors to indeed further expand the documentation.

two nitty gritty remarks:
line 374: euclidian rather than euklidian

line 436-438 has been changed to state that four different submodels are used, but then list coverage, composition and taxonomic asssignment. Maybe explicitly name relative & absolute coverage


Instruction: You are one of the reviewers, your task is to write a review for the article. You will be given the title of the article, the number of the round in which the article is located, and your order among the reviewers.

Input:
Title: A PROBABILISTIC MODEL TO RECOVER INDIVIDUAL GENOMES FROM METAGENOMES
Review round: 2
Reviewer: 3

Output:
Basic reporting: The authors made good efforts to address the issues brought up by the reviewers. The authors tested the performance of two other methods, NBC and Centrifuge for comparison. The authors also tested the method on the dataset from caimi-challenge.org, which is helpful.  Now I think that all of my comments on the manuscript have been answered.

I just have a question about the discussion about the new Table 3.
Experimental design: no comment
Validity of the findings: On Line 422-423, the authors mention “Table 3 shows that MGLEX bin refinement improved the genome bins in terms of the ARI for both sets of genome bins and  increased the recall for Metawatt but not MaxBin. “  However from Table3, the recalls for Metawatt and Maxbin are both increased to 1.00 for “all contig” , and do not change for “swapped contigs”. Also the ARI for MaxBin “all contigs” decreased from 0.90 to 0.76. This may require further clarification.
Additional comments: No additional comments


Instruction: You are one of the reviewers, your task is to write a review for the article. You will be given the title of the article, the number of the round in which the article is located, and your order among the reviewers.

Input:
Title: A SURVEY OF SECURE MIDDLEWARE FOR THE INTERNET OF THINGS
Review round: 1
Reviewer: 1

Output:
Basic reporting: The introduction is not strong. It only has some basic information about IoT, but nothing related to the security of the IoT and how the work contributes to the field.

In addition, the section 4 (summary and conclusion) doesn't have actual conclusion of the survey.
Experimental design: There is no need to discuss a list of the middle-ware with insufficient description of the security. It would be enough to just summarize how many middle-wares reviewed did not discuss the security.

In addition, this paper has two parts that are kind of disconnected. It first defines the security matrix, and then discusses the security design/implementation of reviewed middle-wares. However, the security design/implementation discussion has little emphasis on the challenges in the security matrix.
Validity of the findings: The findings are not well summarized.
Additional comments: I would suggest removing those middle-wares without any security information. On the other hand, it will be useful to have more insightful discussions for those with the security design and implementation. For example, there are several middle-ware using XML that is argued to have substantial performance overhead. However, the XML is already used by those middle-wares. Is there any performance study in original middle-ware design to support the use of XML? Do those middle-wares have poor performance because of XML? Do they have more than enough hardware power?


Instruction: You are one of the reviewers, your task is to write a review for the article. You will be given the title of the article, the number of the round in which the article is located, and your order among the reviewers.

Input:
Title: A SURVEY OF SECURE MIDDLEWARE FOR THE INTERNET OF THINGS
Review round: 1
Reviewer: 2

Output:
Basic reporting: This paper reviews the literature on the challenges and approaches to security and privacy in the internet of things, with an especial focus on how these aspects are handled in IoT middleware. The topic is important and  it is an interesting paper.
Experimental design: This paper mainly reviews the status of IoT middleware. It does not have design with high novelty and the related experiments.
Validity of the findings: This paper presents the states and the challenges of IoT middleware from different perspectives. It does not have a lot of data or result.  The conclusion is solid and appropriately stated.
Additional comments: This paper reviews the the current literature on the challenges and approaches to security and privacy in the Internet of Things. The content is solid and the conclusion is stated appropriately .


Instruction: You are one of the reviewers, your task is to write a review for the article. You will be given the title of the article, the number of the round in which the article is located, and your order among the reviewers.

Input:
Title: FORMATTING OPEN SCIENCE: AGILELY CREATING MULTIPLE DOCUMENT FORMATS FOR ACADEMIC MANUSCRIPTS WITH PANDOC SCHOLAR
Review round: 1
Reviewer: 1

Output:
Basic reporting: The manuscript does not follow the traditional structure of a research article but is still easy to follow. However, I consider that it would benefit a more traditional IMRAD approach since that would highlight the actual individual parts (and contribution) of the manuscript better than the current custom format where parts are more blended together.

I think the literature review concerning Open Access and article processing fees is only tangentially relevant to the article, would consider de-emphasizing that and rather expanding the review of literature that concerns the different markup languages (which I see as the main contribution of this article).

Language is of good quality but should further be improved, counted a handful of simple mistakes during my time with the manuscript.

The discussion section is non-existent, there needs to be ties back into both existing research as well as describing avenues for how this work can be used as an outgoing step for future research.
Experimental design: There is not really a research question in the traditional sense and that I think is the main problem - the manuscript needs to be written more in the style of an academic article rather than a more casual guide for how-to guide for using markdown.

I am not fully convinced that this article fulfils the criteria of being a 'research article' since f markdown and pandoc are presented largely in a vacuum from the section "CONCEPTS OF MARKDOWN AND PANDOC" onwards.

Could some kind of evaluation criteria be introduced to compare the different markup languages to each other to really emphasise the benefits of markdown? I think something like this would be needed.
Validity of the findings: The application of markdown in the context of academic writing warrants focus and I think it is a worthy pursuit to conduct research around it. However, the current manuscript does not fulfil some of the basic criteria for a research-based article but includes to much anecdotal,unquestioned, or unevaluated solutions. I would like to read more about the benefits of markdown vs. LATEX other than being more simple in its syntax.
Additional comments: I am interested in seeing how you may improve on the manuscript since I see potential in the topic, however, the current packaging has a lot that can and should be improved before publication.


Instruction: You are one of the reviewers, your task is to write a review for the article. You will be given the title of the article, the number of the round in which the article is located, and your order among the reviewers.

Input:
Title: FORMATTING OPEN SCIENCE: AGILELY CREATING MULTIPLE DOCUMENT FORMATS FOR ACADEMIC MANUSCRIPTS WITH PANDOC SCHOLAR
Review round: 1
Reviewer: 2

Output:
Basic reporting: This paper describes a strategy for manuscript writing that facilitates the creation of various output documents. The work is well-written and technically sound. Related work is enough detailed, introducing Open Access and justifying the motivation of their work.

In my opinion, some improvements could be introduced.

- Some figures could be resized: Figure 1, 3 and 4 could be smaller. Figure 5, 6 and 7 could be bigger.
- Table 2 may have some error. Tags h1 and h2 may have an additional "<" and ">". Example of bold typeface in HTML could have unnecessary elements (**).
- Some elements of Figure 3 could be better explained. Why MD is linked with bib? In this very figure, template is included in pandoc but no explication is given about it.
- Figure 6 includes an example of owncloud, however, a reader who does not know this platform could not understand what it is. Maybe, a very little introduction could be helpful.
Experimental design: The proposed methodology is compared with others well-known methods and some examples are depicted.

However, some changes could be included:

- Example of how to define additional parameters in figures should be included (Page 10, Line 244).
- In subsection Symbols (Page 10) an example of latex is included. In my opinion, it is not clear if the authors are comparing with latex or this example is the instruction in case of you want to transform MD to Latex.
- Section of "Manuscript with output of DOCX/ODT format and LATEX/PDF for submission to different journals" should be better explained. The listed command could be described. In my opinion, the aim of the section and the commands should be better defined.
The use of templates is interesting but it is not well-defined.
Validity of the findings: Conclusions are clear and well stated.
Additional comments: The use of templates are one of the most interesting parts of Pandoc, but almost no information is provided. This is the weak link of the article.


Instruction: You are one of the reviewers, your task is to write a review for the article. You will be given the title of the article, the number of the round in which the article is located, and your order among the reviewers.

Input:
Title: FORMATTING OPEN SCIENCE: AGILELY CREATING MULTIPLE DOCUMENT FORMATS FOR ACADEMIC MANUSCRIPTS WITH PANDOC SCHOLAR
Review round: 2
Reviewer: 1

Output:
Basic reporting: The authors have adequately responded to my earlier comments and improved the manuscript accordingly or explained why certain things are as they are in manuscript, I now see no major problems with the basic reporting other than that the discussion section and connections back to existing research could still be improved. However, I do not see this as a critical omission that would pose an obstacle for publishing the article.
Experimental design: The authors have adequately responded to my earlier comments and improved the manuscript accordingly.
Validity of the findings: The authors have adequately responded to my earlier comments and improved the manuscript accordingly.
Additional comments: Good work on improving the manuscript While I do recommend accepting the article I still think there is fairly easy work that could be done to make it more contextualized to existing research particularly in the last few pages of the manuscript, now the results don´t really feed back into any existing streams of research despite being based on a plethora of references at the initial outset.


Instruction: You are one of the reviewers, your task is to write a review for the article. You will be given the title of the article, the number of the round in which the article is located, and your order among the reviewers.

Input:
Title: FORMATTING OPEN SCIENCE: AGILELY CREATING MULTIPLE DOCUMENT FORMATS FOR ACADEMIC MANUSCRIPTS WITH PANDOC SCHOLAR
Review round: 2
Reviewer: 2

Output:
Basic reporting: This paper describes a strategy for manuscript writing that facilitates the creation of various output documents. The work is well-written and technically sound. Related work is enough detailed, introducing Open Access and justifying the motivation of their work.

However, one previous minor modification asked in the last review remains unchanged. According to my understanding,  Table 2 may continue having one error. Tags h1 and h2 may have an additional "<" and ">".

Apart from that,  I recommend to accept the manuscript. Congratulations for the authors.
Experimental design: Experimental design has been improved with regard to the previous version, and new  interesting features have been included.
Validity of the findings: Conclusions are clear and well stated.
Additional comments: No additional comments


Instruction: You are one of the reviewers, your task is to write a review for the article. You will be given the title of the article, the number of the round in which the article is located, and your order among the reviewers.

Input:
Title: GENDER DIFFERENCES AND BIAS IN OPEN SOURCE: PULL REQUEST ACCEPTANCE OF WOMEN VERSUS MEN
Review round: 1
Reviewer: 1

Output:
Basic reporting: The paper is clearly written. The literature pertaining to the main finding of the paper, that is the existence of gender bias, and the larger context of women in computing, is coherent and relevant. Figures are relevant and easy to interpret.
Experimental design: This study is indeed one of the largest field data source on gender bias. Few studies exist that pertain to “real world” contexts and go beyond testing the existence of bias in an experimental settings. As such, this paper provides a significant contribution to the literature, and is beneficial to the literature.
Validity of the findings: The data are unique, and at an unprecedented scale, offering significant validity. The research question is relevant, meaningful, and well designed. The investigation is rigorous and the methods well described. One issue to correct is to more clearly link the main research question to the analysis. Most of the write up of the findings do not address the main research question – that is, the question stated on page 5 “to what extent does gender bias exist among people who judge GH requests” is not answered until page 23. Most of the analysis is spent answering the question “why are women’s contributions accepted at higher rates than men”. I would recommend re-organizing the write up of the findings – start with the evidence on bias, then ask the question about women’s higher acceptance rate when bias is removed (e.g. when the gender is unknown to the reviewer), controlling for the appropriate factors and exploring why that might be. We also do not know much in the paper about the people who judge GH requests, although they are central to the research question. This limitation should be noted.
Additional comments: Note: my comments come from the perspective of social science – therefore, my recommendations on methods and specific literature to look to may or may not perfectly match the computing literature and approaches.

This paper makes a significant contribution to the literature by providing large-scale real world evidence of gender bias in Open Source. These findings have significant implications for the design of open source communities and technology workplaces and should be published.

There is a flaw in the way the paper is constructed that warrants a rewrite/re-organization but I think is absolutely achievable:

- The paper starts with a research question on the prevalence of bias in OS, and then spends almost all of its analysis on trying to explain a “surprising” finding that women’s contributions are rated higher when bias is removed. This leaves the reader wondering why they are spending so much time answering what seems to be a different question. The finding that clearly answers the research question, is found in Figure 11: that is, the paper demonstrates that women's contributions are significantly more likely to be accepted when their gender is masked (consistent with the literature), and that the effect is worst for outsiders (which is an important contribution to the literature on how "insider status" may mitigate bias).
- I would recommend starting the analysis with answering the main research question - to what extent is bias present in women's OS contributions? Show the findings from Figure 11th. Then, pose the follow up question "Why are women's acceptance rates so much higher than men's in the gender neutral condition – what could explain this?" - Then, posit one series of hypotheses that could explain the difference and present a single statistical model with all the control variables as opposed to a series of single variable cotrols (if possible). The one by one analysis does not offer one model to control for all variables and understanding the relative impact of one variable compared to another. Then, go into the further delving of the differences by over time data and language.
- The discussion can also strengthen articulating the theoretical contribution of the paper by elaborating on 1) the mitigating effect of being insiders on bias, and 2) the finding that women may be more competent in this context and link back to the social science literature that demonstrates that women have to work harder to be perceived as competent (see the work of Elizabeth Gorman and Julie Kmec for example), especially in masculine domains, and therefore explaining those women who participate in Open Source are indeed likely to be more prepared.

A final comment for the editors and authors: it is commendable and critically important to have multidisciplinary perspectives and approaches to address critical social issues. I feel like it is absolutely within reach for the paper to solidify its ties and contributions to the social science literature while preserving and highlighting the power of computational approaches to analysis.

Happy to speak live if authors have questions or want to discuss ideas more at length (csimard@stanford.edu)


Instruction: You are one of the reviewers, your task is to write a review for the article. You will be given the title of the article, the number of the round in which the article is located, and your order among the reviewers.

Input:
Title: GENDER DIFFERENCES AND BIAS IN OPEN SOURCE: PULL REQUEST ACCEPTANCE OF WOMEN VERSUS MEN
Review round: 1
Reviewer: 2

Output:
Basic reporting: I have found three minor wording problems that could lead to misunderstandings or misinterpretations, and have suggested minor revisions to avoid such difficulties.

1. Regarding this sentence in the abstract (p.1):
“This study presents the largest study to date on gender bias, where we compare acceptance rates of contributions from men versus women in an open source software community”

Taken as a whole, the overall sentence can be interpreted as being limited to a study of bias in an open-source software community. However, the initial phrase “the largest study to date” undermines such interpretation, and is also refuted by prior large scale demographic studies of gender bias in women’s vs men’s earnings, etc. These difficulties/ambiguities can be avoided (see related issue in 2.) by the following minor revision:

Change:  “This study presents the largest study to date on gender bias, . . . ”
To:           “This study presents a large scale study on gender bias, . . . ”

2. Regarding this sentence at the bottom of p2:
“To our knowledge, this is the largest scale study of gender bias to date.”

This sentence, if/when taken out of context (by media, for example), appears to make an unsupportable claim (due to prior large-scale studies of men’s vs women’s earnings, etc). However, within the context of open-source studies the statement is apparently valid. The difficulty here can be avoided via the following minor revision:

Change:  “. . . largest scale study of gender bias to date.”
To:           “. . . largest scale study to date of gender bias in open source communities.”

3. Regarding the lead sentence in the concluding paragraph on p6:
“As an aside, we believe that our gender linking approach raises privacy concerns, which we have taken several steps to address.”

This wording weakens an important awareness and response, which can be avoided via the following minor revision:

Change:  “As an aside, we believe that our gender linking approach . . . ”
To:           “We recognize that our gender linking approach . . . ”
Experimental design: No Comments
Validity of the findings: No Comments
Additional comments: No additional comments


Instruction: You are one of the reviewers, your task is to write a review for the article. You will be given the title of the article, the number of the round in which the article is located, and your order among the reviewers.

Input:
Title: GENDER DIFFERENCES AND BIAS IN OPEN SOURCE: PULL REQUEST ACCEPTANCE OF WOMEN VERSUS MEN
Review round: 1
Reviewer: 3

Output:
Basic reporting: Summary:  This article is absolutely interesting! The study investigates the
rate of pull request acceptance for women vs. men. This is the largest study
to date on gender bias and it also has much significance on the impact of
gender bias in the computer science workforce.  One of the key results is that
women's pull requests are accepted at a higher rate, which the authors explain
based on several theories on gender bias.
Experimental design: + The study uses almost 5 years worth of pull requests from GHTorrent data set.  This is a very large set of data to draw insights from. Therefore the findings from this study carry much weight and make a strong impact in the area of gender bias studies.

- One of the significant result is that, "Women outsiders' acceptance rates are higher but, only when they are not identifiable as women." This part of the methodology is very confusing, which can impact the validity of this finding. In particular, the reviewer was not able to understand how the authors identified a group of female github users who are not apparently identifiable as women.  I understand that the researchers used the method of linking a user's email address to the Google+ social network and extracting the gender information from the Google+ site.  However, isn't this something that other users of github could have done for users, where they only see identicons and they need to handle their pull requests? Therefore, I am confused how the study was able to identify a group of "true female contributors" who are not easily identifiable as women.

Along this line, page 17 mentions that "a mixed culture panel of judges could not guess the gender for."-- please explain who are "mixed culture panel of judges" and how they came to consensus. I think it would have been very useful to know whether there was a manual investigation of "gender" by contacting the github users directly through emails to confirm their gender.

- On page 18, there is an analysis of "insiders" vs. "outsiders" -- this part is confusing. How do the authors classify "insiders" vs. "outsiders"? The text mentions "who are owners or contributors" vs. "everyone else" but is this distinction clear cut for every open source project out there on the github?  Some company projects hosted on the github have many "insiders" a.k.a contributors in the same company who may not be given the role/ permission of "contributors." Please clarify the classification process in detail.

- I hope you can add some more detailed process of "controlling co-variates" with a concrete example. There's a citation to "propensity score matching" but the reader is not able to understand how the authors identify "similar" data and what is the notion of "similarity" in the subsequent analysis for each dimension?  I also have a related question. Does this analysis require the size of data from one group to be the same as the size of another similar group? What if the size of similar groups are very different?  The section on page 19 talks about "matched data" but it is not clear how "matched data" are identified.
Validity of the findings: + The results are definitely interesting and the authors do very good job leading the audience from one question to the next investigation question and interpreting the results in the context of rich literature on gender bias.  For example, it starts with the high level question of "are women's pull requests less likely to be accepted" to the question of "do women's pull request acceptance rates start low and increase over time?" I also appreciate the author's effort of mitigating the internal validity of the study by contrasting group of users with similar characteristics.

+ The related work was an excellent introduction to those who are not familiar gender bias studies.

+ The reader very much appreciated the authors' effort of interpreting the results in the context of many theories on gender bias. For example, the theory of the theory of "survivorship bias" and "self selection bias" could explain the observation very well. I also agree that "women are often held to higher performance standards than men" and therefore expected to submit higher quality of contributions.

- The section on "are the differences meaningful?" seems very important as any study with large participants can easily find a statistically significant difference. The part that is very confusing about this section is that the methodology is written at a very high level. I suggest the authors to add some more details on the "effect size" analysis method. For example, "Davison and Burke's meta-analysis of sex discrimination found an average Pearson correlation of r = .07, and the study's overall 4.1% acceptance rate difference is equivalent to r = .02. How does "4.1%" map to r=.02. How should the readers interpret r value? What is the overall conclusion of this number
on the significance of the entire study?
Additional comments: See above. Very interesting work!


Instruction: You are one of the reviewers, your task is to write a review for the article. You will be given the title of the article, the number of the round in which the article is located, and your order among the reviewers.

Input:
Title: GENDER DIFFERENCES AND BIAS IN OPEN SOURCE: PULL REQUEST ACCEPTANCE OF WOMEN VERSUS MEN
Review round: 1
Reviewer: 4

Output:
Basic reporting: The paper presents a study of pull request acceptance differences between genders. The authors ask the question of whether differences exist between (self proclaimed) male and female among GitHub users and find a statistically significant difference of around 4% between the acceptance rates. They then identify several confounding factors and attempt to explain this difference; they find that women's PRs are larger and are less likely to serve a project need and that women continue to have larger acceptance even as their tenure in the project increases. However, when they are outsiders in a project (typical case in GitHub OSS), and their gender is identifiable through their profile pics or name, they tend to be less successful than men. The authors attribute this last difference to the existence of bias against women in OSS.

The paper is written in a straightforward manner that is very refreshing; without too much fanfare, the authors convey large amounts of information. I really enjoyed reading this paper. I also think the plots and the tables in the paper convey all information necessary. To make it short, I believe that in terms of presentation this paper is very good.

The authors chose not to redistribute the data due to privacy concerns. I understand this decision. However, this limits my ability as a reviewer to check the code that generated the data and analysis scripts. In this case, I think it is very important to ensure data quality through transparency. I would strongly recommend to the authors to release the tools and analysis scripts they used.
Experimental design: I have no general comments about the experimental design; it looks like solid science done on an almost full population study. The comments below reflect specific issues I had with parts of the analysis.

- l17: Why is under-representation an illustration of bias? If there are population differences on sites with gender-oriented content would that constitute bias (active discrimination) or different content choices between genders? Please find a reference to support this claim.

- l115: I do not see the connection between interviewee evaluation and PR evaluation. At the very least, the interviewee gender is immediately apparent, which is not true for PRs (as you also show).

- l120: The Clopper-Pearson method assumes a Bernoulli process where the chance of success is constant. I am not sure this is the case here, as the differentiated variable (PR acceptance) varies significantly across projects. Can you justify your choice of test here, or select a non-parametric test such as the Matt-Witney-Wilcoxon rank sum (also to test for significance as in l123)?

- l123 and elsewhere: You test for significance, but you do not report effect sizes. Can you use an appropriate test (such as Cliffs delta) and report its results?

- l294-onwards: I appreciate the fact that you used propensity analysis here; this is novel in software engineering. However, when you use matched data, the differences the main effect you are observing (e.g. differences in the acceptance rate for women when the gender is obvious) vanishes (comparing Fig 7 and 11). Moreover, it looks like (Fig 11) that the drop in acceptance rate for women and increase for men is within the confidence interval of the gendered-neutral case for both genders.

What does this indicate? In my view, that controlling for covariates, there is a trivial difference (if any) in the acceptance rate of PRs between genders.

Materials and methods section: Given that the differences between acceptance rates are very small, it is important to report ALL statistics when describing data analysis methods.

- l493-500: What is the PR acceptance rate after applying all treatments? -

- l501-510: Gousios et al. in ref [13] and Yu et al. in their MSR 2015 paper calculate the core team as the number of people that have either committed directly or merged a PR the last 3 months before the specific processed PR. If you keep track of project core team members for each PR, you can detect members that were added or removed to the core team by looking ahead in time. So if user A was in the core team for PR 12 but not for PRs 13-last PR, you can definitely say that PR is an outsider for PRs 13 onwards.

- l511: What is the technique? This is crucial to report as gives an indication of the coverage of your study over the general population.

- l518: quantify "relatively few"

- l527: Gousios et al. found that 95% of PRs are closed in 25 days; you could have removed open pull requests that are older than 25 days from your denominator.

- l543: How did you determine whether somebody has a profile image rather than an identicon by parsing the source page? How about profile images that use other types of computer graphics (e.g. cartoons, logos etc)?

- l546: Did you check the accuracy of genderComputer? If yes, what is the recall and precision? How did your changes improve those? In general, how can you be sure that what genderComputer calculates is valid?

- l547-578: This is an interesting approach. How did you scale the panel's findings to the whole population? Did you implement a machine learning based tool?

- l580: I am not sure what the matching procedure attempts to do. My best guess is that you are trying to make causal inferences about the effect of gender and other covariates on whether a PR will be merged or not. Can you please add a few lines about what you where trying to achieve with it? Also, what are the types of features used (factors, integers etc) esp for the "file type" feature? I understand that much of what I am discussing here is mostly presented in l306 onwards, but it helps to have the discussion about propensity score matching in one place, either here (which I recommend) or before.
Validity of the findings: I have one major issue with the paper in its current form: the fact that the observed differences are attributed to bias.

Bias is an overloaded word: in science, it usually means deviation from the truth, while in every day life it is a synonym for discrimination. There are various documented forms of bias that are related to research, both in a statistical sense (e.g. detection bias) and also relating to assumptions and pre-occupations on the researcher's part (various forms of cognitive bias). The authors acknowledge most (all?) of those in their threats to validity section. For bias (as in discrimination) to occur one group or individual needs to consciously or unconsciously ACT against another. Otherwise put, there needs to be a causal link between the fact that an individual is a woman whose profile details are open and the fact that their PRs will be rejected.

The authors do identify an *association* between women with open profile details and drop in PRs acceptance rates, but, in my opinion, they do not identify a causal link. They attempt twice to do so:

 * With their propensity analysis, they attempt to control for covariates in the PR process, but in the end the gender profiles from both men and women have exactly the same acceptance rate (Fig 11). The authors may consider as bias the drop in acceptance rates between gender-neutral and gendered profiles for women (and corresponding increase in men); however, the drop is very low (3%) and possibly within the confidence intervals (Fig 11) of both groups.

* In the discussion, the authors attempt a step-wise elimination of possible reasons for the observed difference. Rejecting a set of theories that fail to explain a result does not automatically make a newly proposed one valid.

What I think the authors are measuring are differences in how the two genders perform on various activities wrt pull request handling, including their under/over-representation in the process. This is, in my opinion, an important finding on its own. But I do not believe this constitutes bias, at least given the evidence presented by the authors.

Given the use of GitHub, I would also avoid to extrapolate my findings to OSS in general.

Finally, one thing I am missing from the paper is an outlook: given the differences that the authors find, what would they propose projects to do?
Additional comments: No additional comments


Instruction: You are one of the reviewers, your task is to write a review for the article. You will be given the title of the article, the number of the round in which the article is located, and your order among the reviewers.

Input:
Title: GENDER DIFFERENCES AND BIAS IN OPEN SOURCE: PULL REQUEST ACCEPTANCE OF WOMEN VERSUS MEN
Review round: 2
Reviewer: 1

Output:
Basic reporting: The authors have strengthened the framing of the paper, the literature review, and relevance of the key research question. In doing so, they have clarified the contribution of the paper and highlight the value of a multidisciplinary research perspective. Combining social science with computer science research methods is a significant contribution to both fields.

One minor suggestion on page 9 - add a reference to the likeability penalty for women who exhibit the same leadership behaviors as men - https://www.ncbi.nlm.nih.gov/pubmed/17227153
Experimental design: no further comment.
Validity of the findings: Robust data methods, and controls and an appropriately nuanced interpretation of the results.
Additional comments: I hope there is widespread dissemination of these findings in computing, in social science, and in the OS community and that it will generate further discussion of the value of multidisciplinary approaches to addressing the under-representation of women in computing. Please let me know when it comes out and we at the Clayman institute for Gender research would love to spread the word about it.


Instruction: You are one of the reviewers, your task is to write a review for the article. You will be given the title of the article, the number of the round in which the article is located, and your order among the reviewers.

Input:
Title: GENDER DIFFERENCES AND BIAS IN OPEN SOURCE: PULL REQUEST ACCEPTANCE OF WOMEN VERSUS MEN
Review round: 2
Reviewer: 2

Output:
Basic reporting: I have reviewed the paper before and commented positively on how nicely it reads. The new version makes an already nice paper nicer to read.
Experimental design: No comment
Validity of the findings: I have gone through the author's extensive rebuttal to both my comments and the comments of other reviewers. I am mostly convinced by the authors responses/changes.

What I am still not convinced about is the fact that the authors attribute the differences in PR acceptance to bias. However, on absence of a better explanation or evidence to contradict the author's theory, I am willing to accept this interpretation. Focused replications, and perhaps a large scale experiment, are needed to strengthen the findings reported here.
Additional comments: No additional comments


Instruction: You are one of the reviewers, your task is to write a review for the article. You will be given the title of the article, the number of the round in which the article is located, and your order among the reviewers.

Input:
Title: SPECIES-SPECIFIC AUDIO DETECTION: A COMPARISON OF THREE TEMPLATE-BASED DETECTION ALGORITHMS USING RANDOM FORESTS
Review round: 1
Reviewer: 1

Output:
Basic reporting: The article is in general clearly written and well-structured. The algorithm statements are commendably clear.

The paragraph between lines 167 and 168 is unclear. I believe that the "5 selected neighbourhoods" are 5 strongly-matching temporal regions. I also believe that 266% is a freely-chosen parameter. Please rewrite the paragraph to be clearer.

The article does not make enough connection with related literature on the topic.
* A very obvious point of comparison is the template-matching species classification introduced by Gabor Fodor and used by Mario Lasseck to perform species classification and get extremely strong results in the annual BirdCLEF contests. Since that method is currently a leading method in the exact topic of this paper, it should be discussed.
* The two-stage method in this paper (cross-correlation followed by SSIM refinement) is somewhat related to Ross & Allen (2013, Ecological Informatics). This comparison could be made in the discussion.

Line 292 claims that the method is "non-species specific" - this is not a good description of the method, since the method relies on templates for the target species and so is species-specific. It would be appropriate to call the method "generic" or "general-purpose".
Experimental design: Broadly fine. However some issues:

The step size used for moving through each spectrogram (in step 5 of "Algorithm to create the similarity vector") is potentially problematic. Moving by a jump of 16 steps rather than 1 carries quite a strong risk (in fact, a 94% chance) of missing the strongest-matching alignment. The authors should have tested for the procedure's sensitivity to this parameter choice.

It would be better to use AUC (area under the ROC curve) rather than accuracy. This is widely-known and is especially important when the classes are unbalanced, as is the case here. The authors claim (line 196) that accuracy is a suitable proxy for AUC in the balanced case: this argument is misleading and must be removed, since the authors are not considering the balanced case.

It is a shame that the authors only used around 900 recordings, when they had access to many more. Some of the folds must have as few as 3 examples, for some species. However, since the main outcomes appear to be significant, this is not a fatal flaw.
Validity of the findings: The findings appear to be valid, and significance is appropriately determined. The results would be more reliable if AUC rather than accuracy was used.
Additional comments: The title mentions both "detection" and "classification". I understand why. However the introduction should clarify for the reader what task is being attempted here: it is to develop a binary classifier for the presence/absence of a single species, and then applying it to each of a set of species of interest. At present the article does not make this quite explicit, the reader must infer it. I would argue that the title should really refer to a comparison of three detection algorithms not three classification algorithms, since the three template-matching procedures used here are used for detection.

The introduction MUST tell the reader something about the three similarity measures that are being tested here, since that is the main comparison. The full algorithm statement comes later, but the intro needs to prepare the reader for what the paper is about. Why these three? What properties does one expect of them?

Typos:
* "Hanning" -> "Hann"
* "lets define" -> "define"
* "algorithms have order of" -> "algorithms have time complexity of"
* "analize" -> "analyze"


Instruction: You are one of the reviewers, your task is to write a review for the article. You will be given the title of the article, the number of the round in which the article is located, and your order among the reviewers.

Input:
Title: SPECIES-SPECIFIC AUDIO DETECTION: A COMPARISON OF THREE TEMPLATE-BASED DETECTION ALGORITHMS USING RANDOM FORESTS
Review round: 1
Reviewer: 2

Output:
Basic reporting: The manuscript compares three template-based classification algorithms using random forests. A web-based cloud-hosted system is quite useful to the bioacoustic community. This paper is well organized, but has some grammar errors, which needs to be improved.

Line 17, Line 24, Line 34, Line 63-64 1 to one, Line 69 id? Line 101 optimize....
Experimental design: In this paper, the template is created by all ROIs submitted by the user. The template is highly affected by the training data. Since animal calls are often different in different regions and different times, can you figure out some methods to address this problem? or can you update the template when doing the comparison to make it less sensitive to those factors? In environmental recordings, there are often multiple species in an individual recording, a single-instance single-label classifier might not suitable for the classification task. However, multi-instance multi-label learning or multi-label learning have shown better performance in previous studies, such as "Acoustic classification of multiple simultaneous bird species: A multi-instance multi-label approach"  "Using multi-label classification for acoustic pattern detection and assisting bird species surveys", "Detecting Frog Calling Activity Based on Acoustic Event Detection and Multi-label Learning".
Validity of the findings: The novelty of this paper is limited, but the impact is high. The data is robust and the conclusion is well stated. The conclusion and future work section is weak and should be expanded and added, respectively.
Additional comments: No additional comments


Instruction: You are one of the reviewers, your task is to write a review for the article. You will be given the title of the article, the number of the round in which the article is located, and your order among the reviewers.

Input:
Title: SPECIES-SPECIFIC AUDIO DETECTION: A COMPARISON OF THREE TEMPLATE-BASED DETECTION ALGORITHMS USING RANDOM FORESTS
Review round: 2
Reviewer: 1

Output:
Basic reporting: OK. Clarity and English is improved as requested.

The added mention of Fodor/Lasseck's work is appropriate, but the reader would benefit from understanding WHY it is relevant - i.e. because it is also a template-matching paradigm.

The mention of multi-label methods added to the conclusion is a bit odd. I recognise that reviewer 2 requested it, but the authors present it as "a way to filter out sound activity not related to fauna", which is not really a good description of what multi-label provides. Also, note that the method presented in this paper is equivalent to the "binary relevance" approach to multi-label, namely, simplifying the problem to an independent binary problem per species. The paragraph should be reconsidered to provide a more meaningful reaction to reviewer 2's request.
Experimental design: OK. The inclusion of AUC as well as the other statistics improves confidence in the results.
Validity of the findings: OK.
Additional comments: No additional comments


Instruction: You are one of the reviewers, your task is to write a review for the article. You will be given the title of the article, the number of the round in which the article is located, and your order among the reviewers.

Input:
Title: SPECIES-SPECIFIC AUDIO DETECTION: A COMPARISON OF THREE TEMPLATE-BASED DETECTION ALGORITHMS USING RANDOM FORESTS
Review round: 2
Reviewer: 2

Output:
Basic reporting: The language issue has been well fixed by the author.
Experimental design: The experiment is well designed.
Validity of the findings: The paper provides a method, which is implemented in the ARBIMON web-based system. It is especially useful for those ecologists who want to study bioacoustics.
Additional comments: This paper is well documented and useful for interested readers. The author has well addressed the proposed problem. I would recommend the paper to be accepted.