Instruction: You are one of the reviewers, your task is to write a review for the article. You will be given the title of the article, the number of the round in which the article is located, and your order among the reviewers.

Input:
Title: THE APPROPRIATION OF GITHUB FOR CURATION
Review round: 2
Reviewer: 1

Output:
Basic reporting: The authors have made substantial changes in this revised version, some of the most important review comments have been addressed. The readability and structure of the paper has improved from the initial version.

Thanks also for preparing a "diff" version which made it easier to track the changes from the previous version.

A few minor comments:
* line 101: "Software Developers’ Motivations in Participating Online Communities" -> "Software Developers’ Motivations for Participating in Online Communities"
* line 468: "to keep them up-to-date" -> "to keep themselves up-to-date"
* line 502: "a only starting point" -> "only a starting point"
* line 508: "was to used ”curated list”" -> "was to use ”curated list”"

Please give the paper another thorough pass and check for any orthographic issues, typos, etc.
Experimental design: Although the design and methodology have been improved (e.g., by adding the coding scheme in Table 1), I still believe that the paper could have been improved by following a more rigorous methodology for analyzing the data and by presenting the results in a more structured and "condensed" way (i.e., aggregating the results). For example, in addition to quoting exemplary statements by individual respondents, the authors could have put more emphasis on extracting some numbers on differences and commonalities in the respondents' answers. (See also the comments related to "Validity of the findings" below.)

There still seems to be a bit of overlap between the Results and Discussion sections. For example, both sections discuss motivations for curators. That said, the Discussions section has improved from the last version as it now discusses limitations and future directions.
Validity of the findings: I still find it a bit unfortunate that the authors were unable to extract a few more quantitative results in their analysis. Performing a basic numerical analysis and cross-correlation in terms of answers by different respondents could substantially add to the value of the contribution.

It has been mentioned in the rebuttal that the sample size is too small to draw such conclusions. Arguably, this should be mentioned in the paper (Limitations section) as well. Note that, even if the data does not allow for conclusions with a very high level of confidence (e.g., 95+%), the confidence interval could be adjusted (e.g., ~90%) to at least show a general trend which can then be further substantiated with more data in future work. I am hesitant to flag this as a strong requirement for acceptance, but at least I would urge the authors to consider it for the final revision, and for future work.
Additional comments: No additional comments


Instruction: You are one of the reviewers, your task is to write a review for the article. You will be given the title of the article, the number of the round in which the article is located, and your order among the reviewers.

Input:
Title: DGPATHINTER: A NOVEL MODEL FOR IDENTIFYING DRIVER GENES VIA KNOWLEDGE-DRIVEN MATRIX FACTORIZATION WITH PRIOR KNOWLEDGE FROM INTERACTOME AND PATHWAYS
Review round: 1
Reviewer: 1

Output:
Basic reporting: The article has a clear and logical structure, but I noticed several typos and grammatical errors in the manuscript, which should be corrected. Attached below, I provide a list with some of these errors and suggestions for corrections, but this is not comprehensive. I therefore strongly suggest to revise the language of the manuscript with the help of a native speaker.
Figure 1 should ideally also outline the meaning of the matrices F and V, and how they are integrated into the analysis workflow, since they contain key information for the proposed approach. I have no comments on the other figures and tables.
Finally, the authors should also discuss how their method relates to their previous approach published in Scientific Reports (Vol. 7, Article number: 2855, 2017, doi:10.1038/s41598-017-03141-w).

List of grammar/spelling errors and suggested corrections:
- Abstract, line 19: "prior knowledge from interactome" --> "prior knowledge from an interactome";
- Line 36: "cataloging driver genes from genes with passenger mutations" --> "distinguishing driver genes from genes with passenger mutations";
- Line 49: "another research on driver genes" --> "another research study on driver genes";
- Line 51: "Bayesian approach is also applied" --> "Bayesian approaches are also applied";
- Line 63: "their interacted neighbors" --> "their interacting neighbors";
- Line 75: "Instead of network propagation, MUFFINN prioritizes" --> "Instead of using network propagation, MUFFINN prioritizes";
- Line 76: "their directed neighbors" --> "their direct neighbors";
- Line 100: "use graph Laplacian technique" --> "use a graph Laplacian technique";
- Line 100: "from interaction network" --> "from an interaction network"; line 123: "in the related gene for which the related tumor sample," --> "in the respective gene and tumor sample.
Experimental design: Most aspects of the experimental design are clear and appropriate; however, I see a potential issue resulting from the authors' claim that previous methods have only used network information but no pathway information (line 21-22): Public molecular interaction databases like STRING, which have been used for previous cancer driver gene mutation prediction approaches (e.g. the approach cited in line 68), also contain interaction data derived from pathway databases (in fact, this is one of the main data sources for STRING). It is conceivable that the authors' approach may still provide an added value over previous network-based approaches through the use of information on pathway boundaries, which is not explicitly included in STRING, but to test this, the authors would need to compare their method against a network-based approach using STRING data (or another network data source exploiting pathway information) rather than an approach based on iRefIndex (an interaction database that only assembles data from other primary interaction databases, but does not provide the coverage of pathway interactions that is offered by STRING).
A question that is not discussed in the manuscript and not addressed in the experimental design is whether some genes may contain both driver and passenger mutations - while the authors' current approach focuses on gene-level predictions, can it also help to make predictions at the level of individual mutations? This should also be discussed in the manuscript.
Validity of the findings: The authors provide multiple sources of quantitative evidence to support the claim that their cancer driver gene prediction method is superior to previous approaches. However, while the presented precision-recall curves suggest a clear difference between the performance of the authors' method and previous approaches, the authors should additionally use a statistical test to assess whether and to which extent the difference is also statistically significant. For example, the authors could compute the areas under the curve and compare them with the non-parametric Friedman test.
The same applies to the comparison of average ranks and the evaluation of top-ranked genes, which should be accompanied by a statistical test (the authors should also briefly comment on whether changes in the number of top-ranked genes considered affect the results).
Importantly, the potential issue regarding the use of networks without interaction data from pathway databases for the comparative evaluation, mentioned in the experimental design section, may also affect the validity of the findings, and the validation results should therefore be recomputed for a network data source like STRING that contains pathway-derived interaction data.
Additional comments: As a minor point, in line 267 the authors probably wanted to refer to the "smallest average rank" instead of the "largest average rank" (in Table 2 lower average ranks represent better results, therefore please check this again).


Instruction: You are one of the reviewers, your task is to write a review for the article. You will be given the title of the article, the number of the round in which the article is located, and your order among the reviewers.

Input:
Title: DGPATHINTER: A NOVEL MODEL FOR IDENTIFYING DRIVER GENES VIA KNOWLEDGE-DRIVEN MATRIX FACTORIZATION WITH PRIOR KNOWLEDGE FROM INTERACTOME AND PATHWAYS
Review round: 1
Reviewer: 2

Output:
Basic reporting: 1) Prior literature on driver gene identification is well explained; however, more intuitive explanation is needed on why matrix factorization is an appropriate model for this task.

2) There's typo in Table 2 where the proposed method is incorrectly named as 'PathInterDG'

3) There's typo in Discussion: 'which are also play important'
Experimental design: 1) Based on the presented results, the proposed method seems to be better in identification of known cancer driver genes. However, it would be useful to see the individual contribution of network information or pathway information; as currently it is not clear whether this success is due to the model (i.e., matrix factorization) or due to the prior knowledge coming from networks and pathways.

2) The authors state that "λC, λV and λL are empirically set to 0.01, 0.01 and 0.1", how stable is the model with respect to these parameters. Namely, do the results vary a lot if these parameters are changed?
Validity of the findings: 1) In "Driver gene identification" section, the authors state that "In the identification result of HotNet2, a higher delta score of a gene indicates a larger potential of being driver genes".  Here, it is not clear what delta score represents. Also, to the best of my knowledge, it is not possible to rank the genes in any way in Hotnet2 results. In Hotnet2,  the edges with weights smaller than a threshold is used, and all the genes that are in strongly connected components in the remaining graph are identified as candidate driver genes. As such, the authors should better explain how they rank genes in Hotnet2 results.

2) How many of the DGPathInter-output genes (e.g. top 100) are neighbors of TP53? It would be useful to mention this. There seems to be a bias among network-based driver gene identification methods  where hot nodes and their neighbors are always identified as candidate drivers.
Additional comments: No additional comments


Instruction: You are one of the reviewers, your task is to write a review for the article. You will be given the title of the article, the number of the round in which the article is located, and your order among the reviewers.

Input:
Title: DGPATHINTER: A NOVEL MODEL FOR IDENTIFYING DRIVER GENES VIA KNOWLEDGE-DRIVEN MATRIX FACTORIZATION WITH PRIOR KNOWLEDGE FROM INTERACTOME AND PATHWAYS
Review round: 2
Reviewer: 1

Output:
Basic reporting: no comment
Experimental design: no comment
Validity of the findings: no comment
Additional comments: The authors have addressed my comments and their revisions have improved the manuscript.


Instruction: You are one of the reviewers, your task is to write a review for the article. You will be given the title of the article, the number of the round in which the article is located, and your order among the reviewers.

Input:
Title: DGPATHINTER: A NOVEL MODEL FOR IDENTIFYING DRIVER GENES VIA KNOWLEDGE-DRIVEN MATRIX FACTORIZATION WITH PRIOR KNOWLEDGE FROM INTERACTOME AND PATHWAYS
Review round: 2
Reviewer: 2

Output:
Basic reporting: Most of the problems with English are now fixed but the manuscript still has some typos and errors:
p3: the revised computational model -> a revised computational model
p4: heterogeneity tumor samples (??) Do the authors mean heterogeneous?
p5: to integrated -> to integrate
Table 2: PathInterDG has to be replaced by DGPathinter again.
p13: performances of our method -> performance of our method --same mistake is done throughout the page
Experimental design: The additional experiments done by the authors have strengthened the claims of the paper. In Figure S5, using network information doesn't change the performance for all three cancer types. The authors should add some discussion about this result.
Validity of the findings: no comment
Additional comments: No additional comments


Instruction: You are one of the reviewers, your task is to write a review for the article. You will be given the title of the article, the number of the round in which the article is located, and your order among the reviewers.

Input:
Title: RESEARCH ARTICLES IN SIMPLIFIED HTML: A WEB-FIRST FORMAT FOR HTML-BASED SCHOLARLY ARTICLES
Review round: 1
Reviewer: 1

Output:
Basic reporting: The overall command of English needs improvement:
Line 24 a set tools -> a set of tools
30 conferences and journals -> conferences, and journals
33 -> technical ->  technically
44 about it validation -> about validation
58  Nonetheless HTML -> Nonetheless, HTML
58  as output -> as an output
62 trasferred -> transferred
81  annotations), handling all ->  annotations), handing over all
96 paper is in fact to -> paper is, in fact, to
Lines 103 to 109 Section numbers are missing, probably due to RASH conversion error
107 introduction of the -> introduction to the
109 paper sketching -> paper via sketching
112 papers have resulted -> papers has resulted
113 following subsections we -> following subsections, we
114 them in two -> them into two
123 that has proposed -> that proposed
Between 138 and 139 Table 1. A comparison among existing HTML-oritented formats for scholarly papers according to seven distinct categories. -> Table 1. A comparison among existing HTML-oriented formats for scholarly papers according to seven distinct categories.
144 In services made available by the company would enable also the conversion from Microsoft Word document into such ScholarlyHTML format. -> The company also provides services that enable the conversion from Microsoft Word document into ScholarlyHTML format.
147 developing a HTML -> developing an HTML
153  proposals which is ->  proposals, which is
154  is Web applications (still under-development) ->  is a web application (still under development)
156 which annotations -> which are annotations
165  as primary -> as a primary
167 we limit to a -> we impose a limited
171 writing papers, and -> writing papers and
175 for her scholarly ->  for the scholarly
186 paper to others -> paper with others
197  Leaving the authors of using their own HTML format could be also counterproductive -> Allowing the authors to use their own HTML format could be counterproductive
200 camera ready -> camera-ready
203 camera ready -> camera-ready
208  such Web-ﬁrst  ->  such a Web-ﬁrst
218 theory, we developed ->theory we developed
219  section we -> section, we
224 guideline for fostering -> guidelines in fostering
226 that then has been already applied -> that has since been applied
236 plausible exemplar within -> plausible example within
256 allow to increase the reusability -> allow increased reusability
259 of document -> of the document
260 tools, can make hypothesis -> tools, can make  a hypothesis
261 study global properties of sets of -> study the global properties of a set of
264 is enough expressive to -> is expressive enough to
276 Any RASH documents begins as -> All RASH documents begin as
280 standards, and -> standards and
288 subsection we -> subsection, we
288 about pattern-usage in -> about usage  pattern in
292 be enough expressive for -> be expressive enough for
294 Missing Section number
299 Missing Section number
301 Missing Section number
303 prohibit to contain additional -> prohibit it from containing  additional
336 used as default -> used by default
421 guarantee a proper -> guarantee proper
442 consideration, since -> consideration since
442 possible so as to -> possible to
455 adds other two -> adds another two
546 such environment -> such an environment
550  section we ->  section, we
574  that these kinds of ->  that this kind of
580 the more suitable -> the most suitable
585  crucial step to ->  crucial steps to
590 perform conversions from -> perform conversion from
594 converted in the -> converted to the
604 a chanche to have fruitful feedbacks, comments and -> a chance to have fruitful feedback, comments, and
605 of the research papers, project proposals and deliverables, documentation, and two Ph.D. thesis we -> of research papers, project proposals and deliverables, documentation, and two Ph.D. theses we
611 workshops and -> workshops, and
617 a HTML -> an HTML
622 the reviews and -> the reviewers, and
626 whenever it will be possible -> whenever it is possible,
628 Framework has concerned -> Framework concerns
650 represents logical -> represents a logical
651 As sections -> Sections
659 detect sententes and -> detect sentences and
661 task SPAR -> task, SPAR
663 support english only -> support English only
665 with open -> with an open
671 that provides a sample -> that provide a sample
756 text-editors -> text editors
761 companies and -> companies, and
766  publishing it ->  publishing, it
769 markups, the -> markup, the
774  camera ready ->  camera-ready
777 and USA -> and the USA
783 Section number missing
785 participated to our -> participated in our
792 and United -> and the United
793 demos and 1 position papers -> demos, and 1 position paper
798 participated to the ->  participated in the
807 a HTML -> an HTML
815 with a different -> with different
818 assessed strengths -> assessed the strengths
831 details to the -> details from the
844  as limit in ->  as a limit in
851  a HTML  ->  an HTML
854 access to the -> access the
856 of them however added -> of them, however, added
858 writing research -> writing a research
876 ability of converting -> ability to convert
881 figures and captions. However, most of authors -> figures, and captions. However, most of the authors
903 person background -> person's background
904 and and -> and
912 ﬁnd unfeasible -> ﬁnd it unfeasible
928 appear in -> appears in
949 conferences and -> conferences, and
951 technical savvy -> technically savvy
960 ﬁnd hard -> ﬁnd it hard
964 As far it concerns the analisys of RDF annotations in RASH documents -> As far as the analysis of the RDF annotations in RASH documents is concerned
965 adopt few -> adopt a few
972  statement in ->  statements in
995 as additional -> as an additional
Experimental design: The experimental design is sound and appropriate for a journal article dealing with the subject.
Validity of the findings: The findings appear to be in line with the experimental results.
Additional comments: Please use a spelling checker on any document you are considering for submission to a journal or conference as a basic requirement. Then ask a friend or colleague that has a better command of the English Language (the best solution is a teacher of English) to check the document for grammar mistakes. Otherwise, you risk the acceptance of such a good scholarly work!

Also, if you are transforming the content from one type of document to another, check if the conversion introduced mistakes of its own. Missing section numbers in this manuscript fall into this category.


Instruction: You are one of the reviewers, your task is to write a review for the article. You will be given the title of the article, the number of the round in which the article is located, and your order among the reviewers.

Input:
Title: RESEARCH ARTICLES IN SIMPLIFIED HTML: A WEB-FIRST FORMAT FOR HTML-BASED SCHOLARLY ARTICLES
Review round: 1
Reviewer: 2

Output:
Basic reporting: This is a well-written paper tackling two of the most important issues facing scholarly communication today: first the enormous amounts of time, effort and money spent re-formatting articles to meet the requirements of different publishing pipelines; and second, the growing need to be able to unambiguously identify and cite objects other than publications in published works.

The article's structure is coherent, and the prose reads well. The balance of background, theoretical and experimental content is appropriate.

Something bad has happened to the section references within the article -- they are all missing and of course will need to be re-instated, and there are numerous cases (too many to list individually) where opening typographer's quotes are the wrong round (this will no-doubt just get fixed later in the process).

A few optional suggestions to improve the text:

63: "Scholarly HTML3 defines a set of descriptive rules to use a reduced amount of HTML"; perhaps a "defined subset" instead of a "reduced amount"?

67: "customized version of HTML"; I'm on the fence about this, but 'customized' slightly suggests a deviation from HTML (i.e. something that wouldn't parse), whereas RASH appears to be a defined subset.

82: "This is a well-established principle in scientific publishing: clear separation of concerns"; I'm not sure this is a well-established principle in scientific publishing even if it should be -- it strikes me that many of the problems we face today come from the fact that concerns haven't been adequately separated, and that authoring, archival and presentation of articles have been muddled up a bit.

218: The comma after 'patterns theory' shouldn't be there, and I think 'patterns theory' could do with a citation or explanation. It's a fairly well understood concept, but given that understanding it is core to the paper, it's probably worthwhile elaborating further.

742: suggest "rests on its adoption by authors and workshops" instead of 'good number'

754: "worth anticipating" -- does this mean 'worth noting' perhaps?

At the risk of sounding like a plea to cite my own work,

S. Pettifer, P.McDermott, J. Marsh, D. Thorne, A.Villéger, and T. K. Attwood. Ceci n'est pas un hamburger: modelling and representing the scholarly article. Learned Publishing, Jul 2011.

does consider some of the 'separation of concerns' issues discussed here (albeit from the point of view of storage and presentation, not so much from authoring); it may be worth including a reference to this paper.
Experimental design: The authors have made a respectable attempt to evaluate RASH, and while the numbers involved in the evaluation are relatively small, I think they are perfectly acceptable for this kind of work. This is primarily a description of a novel technology that has been developed to meet a real-world need, built using sound engineering principles and theory; at this stage it would be wrong to over analyse the 'experimental' component of the work.
Validity of the findings: No Comments
Additional comments: No additional comments


Instruction: You are one of the reviewers, your task is to write a review for the article. You will be given the title of the article, the number of the round in which the article is located, and your order among the reviewers.

Input:
Title: RESEARCH ARTICLES IN SIMPLIFIED HTML: A WEB-FIRST FORMAT FOR HTML-BASED SCHOLARLY ARTICLES
Review round: 1
Reviewer: 3

Output:
Basic reporting: Detailed comments (including grammar suggestions) are available as hypothes.is annotations on the RASH version of the article: https://via.hypothes.is/https://essepuntato.github.io/papers/rash-peerj2016.html
(Expand on the right, see inline comments)

The language of the article is OK, but I'm afraid it needs some work several places to improve clarity, e.g. by rephrasing or simplification.

Some of the data (the CSV files) has been shared on Figshare and cited as such, but I am missing the raw data of the extracted RDF annotations as well as the scripts used for extraction.

The HTML file of the article in RASH format has for some reason not been submitted as an additional file (only cited by URL) -- perhaps this was not supported by PeerJ's submission system?

The RASH framework and associated software is referenced by GitHub URLs, but without using versioning. For archival purposes and future availability I would appreciate a Zenodo or Figshare archive of a tagged/version of the software, cited using DOI.
Experimental design: No Comments.

The article describes well the rationale and methodology.
Validity of the findings: The survey data is robust, but perhaps of a bit small sample size to be statistically sound. This is however helped by the fact that the survey was run over two consequent years.

The paper makes several unfounded claims using words like "guarantees" - I believe this is more of an English language/grammar issue than actual claims. See detailed comments on https://via.hypothes.is/https://essepuntato.github.io/papers/rash-peerj2016.html
Additional comments: This article presents RASH, an HTML-based format for authoring, exchanging and publishing academic articles, arguing that this allows a "Web-first" approach to authoring with a focus on content; but with facilities for semantic annotations. The associated RASH software allows conversion to more traditional article styles (for PDF), as well as conversion from a traditional word processor to RASH HTML.

While the authors argue that we should aim for a "Web first" publishing model with no conversion to traditional PDFs, as championed by the perhaps more visionary linkedresearch.org movement, here it is proposed that RASH gives a pragmatic approach that requires smaller adaptation to existing co-authoring and publishing workflows.

The authors has performed a kind of usability survey for RASH users at two consequent workshop, which gives validity to the claims of its purpose, but also (as recognized by the authors) highlights the current gap in tooling and skills to produce the underlying HTML and RDF annotations.

I think RASH can be seen as an important element of modernizing academic publishing; and while it can be argued that a restricted HTML template like RASH can limit academic authors from publishing articles augmented with state of the art web technology (for instance for interactive data rendering), this model is also a stepping-stone with a stronger focus on portability and longevity that lowers the barriers to get existing publishers on board.


Instruction: You are one of the reviewers, your task is to write a review for the article. You will be given the title of the article, the number of the round in which the article is located, and your order among the reviewers.

Input:
Title: RESEARCH ARTICLES IN SIMPLIFIED HTML: A WEB-FIRST FORMAT FOR HTML-BASED SCHOLARLY ARTICLES
Review round: 2
Reviewer: 1

Output:
Basic reporting: I am very pleased that the authors took their time to correct the numerous spelling and grammar errors that I have mentioned in my previous review. However, some still exist and must be addressed:
In the abstract, findings, "be" is repeated twice.
 In the introduction, 1st paragraph, remove the 2nd could

There are sufficient and relevant literature references to provide a solid backround to the reader. Also the use of tables and figures is appropriate with all relevant raw data provided. However, I encourage the authors to label the code extracts provided throughout the paper as listings with a listing number and in-text references to these listings.

The results provided are of high quality, since the authors used their tools to produce the paper (eat their own dog food). Suffice to say, the first version of the manuscript suffered from missing titles due to this process, which have been corrected in this manuscript by fixing the error in the toolchain.
Experimental design: The article is definitely within the scope of the journal (computer science) and is a highly original research, aiming to streamline the publication of research. The authors clearly define their aims and proceed to fulfil them in a well-performed research experiment, especially in producing the manuscript using their own toolchain.
Validity of the findings: The validity of the experimental design is exposed though the source code they provide together with the raw data and the final product being the manuscript itself. The results stand on their own.
Additional comments: The reviewers comments on the first manuscript were addressed in a proper manner, you took your time and made your work a well-presented article that I expect to gain nottice from the academic publishing world.

A job well done.


Instruction: You are one of the reviewers, your task is to write a review for the article. You will be given the title of the article, the number of the round in which the article is located, and your order among the reviewers.

Input:
Title: RESEARCH ARTICLES IN SIMPLIFIED HTML: A WEB-FIRST FORMAT FOR HTML-BASED SCHOLARLY ARTICLES
Review round: 2
Reviewer: 2

Output:
Basic reporting: The authors have addressed all the comments from the reviewers in the spirit in which they were intended. The result is an excellent paper reporting an important idea.
Experimental design: No comment
Validity of the findings: No comment
Additional comments: No additional comments


Instruction: You are one of the reviewers, your task is to write a review for the article. You will be given the title of the article, the number of the round in which the article is located, and your order among the reviewers.

Input:
Title: RESEARCH ARTICLES IN SIMPLIFIED HTML: A WEB-FIRST FORMAT FOR HTML-BASED SCHOLARLY ARTICLES
Review round: 2
Reviewer: 3

Output:
Basic reporting: The paper have clear language, significantly improved since first review. The dataset is augmented with extra material, and referenced properly from Figshare with https://doi.org/10.6084/m9.figshare.3980463.v5

Argumentation is well-structured and founded, although a couple of citations or examples are missing, e.g. claim that HTML allow ambiguous structures, or the novel (and unnecessary) use of RDF/XML in a script tag.

See https://via.hypothes.is/https://essepuntato.github.io/papers/rash-peerj2016/2017-07-06.html#rash-eval for my detailed review per section of this version.
Experimental design: The paper describes well the motivation and design of the RASH framework, while also giving an extensive and up to date review of comparative technologies and approaches.  The paper explains also challenges and peculiarities encountered in its implementation.

RASH is a well-designed subset of HTML that emphasizes document structure and semantic annotations. I think it could also be argued that unlike "any HTML5" this design also improves longevity for articles **published** in RASH HTML.

My only slight concerns is the extension of WAI-ARIA roles (e.g. "doc-endnotes"), which I could not find any citations for being allowed (or not) within HTML5; as well as the novel use of RDF/XML in a HTML script tag.
Validity of the findings: The survey part provides valuable insight into the uptake potential of RASH-like technology - although this should be taken with a grain of salt as the relative low number of survey participants means the data is (as the authors point out) NOT statistically significant. The paper do however provide a good *qualitative* analysis of the findings, which warrants their inclusion.

The authors provide well-reasoned conclusions. While my previous review identified some speculative language, this have now been improved.
Additional comments: I am Stian Soiland-Reyes http://orcid.org/0000-0001-9842-9718 and believe in open reviews.

This review is licensed under a Creative Commons Attribution 4.0 International License http://creativecommons.org/licenses/by/4.0/

The authors present the RASH framework, a subset of XHTML for academic publishing, along with software tools for its validation and conversion. The paper review the state of art in academic HTML publishing, motivate and detail the design of the framework, and evaluate its uptake and future challenges.

While personally I would have welcomed a more visionary/revolutionary approach for changing academic publishing for the Web, the authors take a more conservative approach with emphasis on pragmatic tooling to support existing authoring workflows (e.g. support LaTex and MS Word). From this, RASH can provide a valuable stepping stone for more structured and accessible Web publication workflows for academic publishing.

I think this is a solid article that presents an important contribution to the further development of web-based scholarly communication.

I Recommend this article as "Accept" - although I have left some annotations in https://via.hypothes.is/https://essepuntato.github.io/papers/rash-peerj2016/2017-07-06.html#rash-eval (Click "<" and "Show Annotations") that I hope the authors will consider (along with this review) if a revision nevertheless take place.


Instruction: You are one of the reviewers, your task is to write a review for the article. You will be given the title of the article, the number of the round in which the article is located, and your order among the reviewers.

Input:
Title: GUIDELINES FOR USING EMPIRICAL STUDIES IN SOFTWARE ENGINEERING EDUCATION
Review round: 1
Reviewer: 1

Output:
Basic reporting: The English is generally good.  The structure is less so and the balance between the trivial and the interesting should be changed.

Overall it's ok but there's plenty of scope for improvement.

I give more detailed suggestions in the General Comments
Experimental design: The paper brings together some relatively novel ideas and is / could be a stimulating read.

The main approach is advocacy and author experience.  More objective evidence might be added but even if it wasn't the paper would generate discussion and so could be published.
Validity of the findings: The worth of the paper stands or falls by the ideas.  There are few statistics.
Additional comments: The title should be changed as the paper describes many things in addition to experiments e.g. simulation and case studies.

Overall the paper brings together some relatively novel ideas and is thought provoking so I believe it should be published but there are many opportunities for improvement.

(i) There’s a lack of structure, other than around different forms of empirical research.  I think the authors struggle to shed their perspective of empirical software engineering researchers. This paper is about pedagogy. And what about different forms of learning, different types and levels of student.  In any case is this always the answer, sometimes the answer?  Indeed answer to exactly what question?  (Potentially Tables 5 and 6 might serve that purpose but it needs moving to near the start of the paper and I’d think about some additional dimensions such as type of student/course etc.

(ii) I would prune the generic material and try to focus on teaching examples. It would be very interesting to know how some of these ideas were implemented.  What went well and what less so?  Presently the reader has to wait until Section 5 and that's extremely brief.

(iii) I think the claims for the benefits of this type of teaching need to be sharpened.  "In our experience, this approach prepared students 36 better for later working life” l35-6.  So better than what?  How do you know?  How might this be demonstrated (perhaps in the future)?  Likewise claims of causality "Training in empirical instruments, such as experimentation ... cause students to become more …”.  Such claims could (should?) be put forward but you need to be clear that you’re essentially using rhetoric or advocacy.  Again right at the end of the paper are a few references to improved results (ll596-600) but it’s all too late and too brief.

However I do agree that ”little attention has been paid to using empirical studies with an educational purpose in mind.”  This is an opportunity and the paper is useful in provoking thought and discussion around this topic.

DETAILED REMARKS:
The Related Work section could be improved.  It seems to wander around and then keep repeating that the literature thus far really only considers students as participants research experiments.  It might help to try to think about types of education and learning goal.

Also a good deal of material is a rather generic discussion of different empirical research methods; what would be more interesting is a detailed evaluation of how such methods can be used for teaching purposes and also some examples.  There are some tantalising references to examples such as continuous experimentation with student groups (ll264-9) but the reader learns almost nothing without going to the original article.  Even by going back to the original Kuhrmann et al. (2014), article I didn’t get a very clear idea of how the treatments were identified, students allocated, how the debriefing work and on what basis this was deemed a pedagogic success.  (I’m sure it was very engaging and quite likely valuable but it would be nice to have some reasons to support this expectation.)

If simulation is used for teaching how do you ensure the model is sound / valid?  What situations are best / least suited for simulation.  I note that later on, Table 3 (C08) also comments on this problem.  In what sense is this experimentation?

It’s unclear how individual studies are a form of experiment.  Please explain more clearly what is meant here as I can’t really see how they differ from a standard piece of coursework.

How does the discussion on replication impinge on teaching?

What are the Pn’s in Table 2?  Teaching objectives?  Also there seem to be a lot of overlaps between the purposes.  Part of P01 is capturing the problem.  Doesn’t this entail collecting relevant data?  What’s the difference between impact and changing the culture. Perhaps there are some missing higher level concepts?  There are also some time and other dependencies.

Why are Tables 3 and 4 distinct, i.e. what’s the difference between a challenge and validity consideration?

The coclusions should also consider when not to use these ideas.

TYPOS:
l63: "An example of a disciplines that already has a high level”?

Table 3: C06 " Bias is automatically introduced by the respective settings,” ???

l632 Incomplete reference


Instruction: You are one of the reviewers, your task is to write a review for the article. You will be given the title of the article, the number of the round in which the article is located, and your order among the reviewers.

Input:
Title: GUIDELINES FOR USING EMPIRICAL STUDIES IN SOFTWARE ENGINEERING EDUCATION
Review round: 1
Reviewer: 2

Output:
Basic reporting: The authors wrote a well structured paper on their approach introducing empirical research methods into their software engineering (SE) courses. They were able to identify purposes, challenges and validity considerations for the use of empirical studies in SE education. They suggest a mapping of  these identified matters to the four major research approaches : experiment, case study, continuous experimentation and simulation. They motivate their findings based on the course evaluation data from their students. The authors conclude their students are more motivated and get higher scores when using empirical studies in education.

Title: I think the title is misleading. I suggest the authors change it to one that fits better to the aim of the paper: ‘The present paper provides an initial systematisation of empirical instruments from the educational  perspective’. The paper does not only provides a experiment-driven approach and does not really go into detail about how to setup a course to use this experiment driven approach as the current title could suggest.

Introduction:  misplaced comma (line 16). Line 29-31: seems somewhat in contradiction to say it ‘beats empirical evidence’, but that same evidence is not available. Line 52-55 mention the sections, but leave out section 5. Section 6 is mentioned, but says ‘Conclusion’. The section in the paper is named ‘Summary’. I suggest the authors change it to ‘Conclusion’.

Related work: The related work is mostly linked to motivation of students. Did the authors search for any literature on the long term effects that they mention in the introduction(line 17/18)? Why did the authors not mention anything on educational approaches such as evidence based learning, problem based learning or project based learning? The last one for example is used a lot in Europe (e.g. The Netherlands, Sweden, Belgium, Germany, France …). During the projects often case studies are used that come from industry. In these projects different research activities are trained (for example: analysis).

Section 3 - Overview: the overview is clearly written. The examples that are given are nice and informative. How did the authors came to this set of types? This does not cover al empirical study types. Did they consider other types?  Line 260-261: ‘From the teaching perspective, continuous experimentation helps understand the connection between  software development techniques and business.’ Do the authors have any experiment-data / argument / reference to support this?

Section 4 - Guideline: Line 425 - 426 ‘were extracted .. ‘ which studies were used? Home many?
It is not clear how the tables 2,3,4 were constructed. Why are these categories / criteria used? Was there discussion between several experts and/ or industry? In tables 5 and 6 a mapping is presented. How was this mapping done? What where the criteria? For example, I can imagine that P01 and P03 also can be learned during a case study.  Or do I mis some information here?
4.3 ‘The factors’ : which factors?
The authors could have elaborated more on the process of setting up a course that uses empirical studies. The section focus mostly on which study type to choose. Can the authors provide a step by step approach for the educator?

Sections 5 - Experience : Line 559 - 560. The description of the experiment is not completely clear. The students used communication, but communication was avoided? (The experiment sound great by the way). Line 585 / 586: not surprising.  Line 587-595: This part is a little to vague: ‘certain learning curve’, ‘inappropriate modelling concept’, certain ‘certain design decisions’ . What were the key matters that lead to the students’ awareness?

Summary: ‘Using empirical studies in SE education is not yet common ..’ I disagree. As mentioned above there are educational approaches that already use components of empirical studies sinds years. I do agree it is not often written down and indeed it lacks of guidance  for educators in SE. In my opinion the conclusion that the authors’ experience ‘shows’ increased motivation and participation and better quality is too firm. The number of participants is to low to really state that and generalise it. On the other hand the initial framework / guideline that they present seems to be a very valuable contribution.

References: Line 632: Missing title
Experimental design: One one hand this paper does not really provides an experiment. On the other hand the authors do analyse participants of several courses where a certain method was used and use the results to argue the use of empirical studies improves different matters. In that case the authors could have elaborate more on the background of the participants. Because of the low amount of the participants (N= 6,8 or15). The authors could have chosen to focus more on the qualitative data.
Validity of the findings: As mentioned above, maybe the conclusion is a little too firm. Although I don't doubt using the guidelines will positively support students’ learning, the low number of participants per course can not be used to generalise.

I do agree with the authors this paper and their guideline can fill a need for educators.
Additional comments: Thanks for the contribution, love to read the revision.


Instruction: You are one of the reviewers, your task is to write a review for the article. You will be given the title of the article, the number of the round in which the article is located, and your order among the reviewers.

Input:
Title: GUIDELINES FOR USING EMPIRICAL STUDIES IN SOFTWARE ENGINEERING EDUCATION
Review round: 1
Reviewer: 3

Output:
Basic reporting: The article provides a survey of empirical studies in general and with special emphasis  for using in software engineering courses.
The article is written  in English using clear and unambiguous text and conforms to professional standards. The article includes sufficient introduction and background in terms of related work. Comprehensive relevant literature is referenced.
Instead of figures, several tables with relevant content of the article are included.
The layout of the tables (border) should still be adapted.
Please note that I am not a native speaker so that I can not finally evaluate the written text. But I found the following typos:
- row 16: "necessity of"...?
- Table 2, P01, row 5: "Formulate use[r?]- or customer related.."
- row 539: "Especially the [the] practical projects .."
- row 540: "the workshops [were] highlighted."
Further minor problems are the following:
- row 55: A sentence about the content of Section 5 is missing.
- row 127: I miss a reference to SWEBOK.
- Table 1: I would like to have a third column with a description from the teachers perspective as describe in the text of the Subsections 3.1 to 3.6
- Table 4: Is "Priority" really the right term?
- Table 5 and 6 should be reconsidered. For example
-- Challenge C05 is also addressed in both tables and not as textually described more in  Table 6 (row 447).
-- Why C04 is not relevant for case studies in both settings?
Experimental design: The experimental setting is described in Section 5.
The authors report about experiences with empirical studies in two software engineering courses. Their experiences are divided into a formal and an informal evaluation.
The number of the samples are rather small. The courses were attended at most by 19 students. It would be interesting if and how it is possible to perform empirical studies in courses with a large numer of students (100 and more).
In Example 2, the relationship of the terms "groups" and "teams" are not clearly explained. It is confusing to read.
References:
- row 632: I miss the url of the referenced portal.
Validity of the findings: The findings (Table 8/9/10/11 and the discussion) could be doubted because of the small sample. But from the point of view of an experienced teacher who also used empirical studies in SE education, the findings can be accepted/believed.
The selection criteria for the choice of a study (Subsection 4.5 and Table 7) is based on literature and "experience". But it needs more explanations. Furthermore I miss a legend: what do +/o/-/"o/+" and so on mean?
Additional comments: I like the article especially because of the classification of empirical studies and their hints for the use in SE education in terms of purposes, challenges and validity considerations.
All in all the first part of the article seems to be more mature than the second part.


Instruction: You are one of the reviewers, your task is to write a review for the article. You will be given the title of the article, the number of the round in which the article is located, and your order among the reviewers.

Input:
Title: GUIDELINES FOR USING EMPIRICAL STUDIES IN SOFTWARE ENGINEERING EDUCATION
Review round: 2
Reviewer: 1

Output:
Basic reporting: Well written
Experimental design: No comment
Validity of the findings: Fine
Additional comments: I’d like to thank the authors for their constructive responses and extensive changes. Table 8 is particularly valuable and the example in 6.3.

There’s not much to add.  I suppose we still have slight differences in what are the most valuable aspects; I think the examples and experiences are the most important whilst the authors also place emphasis upon setting out the underlying basics such as background on underlying empirical methods. However, I think the paper is considerably improved and can be published.

It would still be useful to label the first column P01 etc for Tables 2-4 i.e. student purpose etc as  it’s not until l537 I can begin to infer what they mean.


Instruction: You are one of the reviewers, your task is to write a review for the article. You will be given the title of the article, the number of the round in which the article is located, and your order among the reviewers.

Input:
Title: GUIDELINES FOR USING EMPIRICAL STUDIES IN SOFTWARE ENGINEERING EDUCATION
Review round: 2
Reviewer: 2

Output:
Basic reporting: The authors wrote a clear and well structured paper about their experiences in applying empirical research methods in their software engineering (SE) courses. From literature they selected five research types: experiment, case study, continuous experimentation, simulation and individual. Based on their own experience the authors suggest guidelines for using empirical studies in SE education. They describe 3 cases in which they applied empirical studies in education. Based on the course evaluation data from their students the authors conclude their students are more motivated and get higher scores when using empirical studies in education.

A strength of the paper is that the suggested study types are all supported with examples of own (published) research.
Another strength is that the examples use an educational perspective and an industrial perspective.

A weakness of the paper is that they don’t discuss any validity threats. You would expect that from a paper that advertises research as an instrument.

- Title
Maybe the authors should go for ‘studies’ in stead of ‘instruments’. Although ‘instruments’ is used, the paper discusses ‘studies’. For the rest the title reflects the content of the paper.

- Introduction
line 19,20,21: rephrase, no it says that the goal of the paper is to develop guidelines. The authors probably mean to show that the already developed guidelines work in the observed cases.

- Related work
The related work describes application in other disciplines, requirements and application in SE education.
line: 125 ‘little attention …’ The authors are somewhat negative about the current application in SE education. In educational approaches such as problem based and project based learning students apply empirical instruments. Also case study approaches in collaboration with industry are widely used. It could be more though.

- Research approach
line 169: which papers are analysed and why were they important and leading?

- An Overview Of …
line 186: why are these instruments listed, on what ground did they author choose them? Why are they common?
Table 1: how was this table constructed, on what ground?
line 293: ‘our experience show .. requirements’ Is there source that claims this is often the case?
line 420: ‘from a teaching perspective’ -> ‘from an educational perspective’ (maybe also other places in the text)
line 437: ‘Further Instruments’ -> ‘Supporting Instruments’ ?

- A Guideline …
line 518: ‘we identified … ‘ -> how?
line 531: ‘we derived …’ -> from what?
Table 3: are there possible resource challenges in terms of amount of lecturers / hours?
Table 5/7: How were the tables created (process, arguments, etc.)?
line 571: ‘For students … consideration’ a credit point reflects time, it is a fixed number. How could different times be ‘required’ for a credit?
line 603: This subsection is very unclear. On what ground was it included? Why and how were own criteria added?

- Experiences
suggestion: include the study type in all the subsection titles
It is not clear on what ground the authors claim the basic course content is the same. Is it really comparable?
It is very confusing that in the text of example 1 and in table 9 the authors write about grades. It seems that it should be about student judgements. No exam grades are provided.
How many students were involved?
line 690: Example 3, this example is obvious. What is new about using empirical studies in the master thesis period? The example does not describe some special approach in comparison
to more conventional approaches.
line 757: is this meant as overal discussion? Make it a section then. Otherwise: you are missing a discussion section.
line 770: first use of ‘exams’

- Conclusion
line 775: ‘there is a lack of guidance’ I think there still is. The authors did provide compact guideline in the form of tables. They really can be of help for lecturers to setup their courses using empirical studies. The guidelines in this paper don’t provide guidance yet. Guidance suggests that there is a guide (handbook), which would be great actually.
line 780: ‘Our experience .. exams’ Here the authors claim, based on 1 comparison of grades (that are not show in the paper) the use of empirical studies have a positive influence.
Strongly advise: rephrase this, or show the evidence. Still it is only 1 comparison.
line 790/791: sentence does not ‘flow’.
line 798 - 804: maybe rephrase this. Are the authors mention matters twice? It is not clear what they mean here. What is the difference between ‘student assessment’ on line 798 and ‘assess learning outcomes’ on line 802?
Experimental design: The authors did not really conducted an experiment. They claim to have a mixed method research design, but this is not reflected in the paper. The qualitative part is very clear and well structured. The quantitative part is not well described. The authors present tables without explanation of the data collection process. Was it done in previous research? The authors do not discuss the participants (students, lecturers, other staff members) that participated in the studies. The authors don’t write in much detail about the data collection and data analysis processes.
Validity of the findings: The authors do not discuss any validity threats. From the paper it is not clear how the quantitative data was collected and how large the sample was. The authors mention exam grades, but it is not clear where in the paper this is discussed in more detail. Still the authors base a conclusion on that data. The conclusions based on the qualitative data is more and better discussed, but the authors should have taken the responsibility to discuss the validity of their research (for example biases, construct validity, conclusion validity etc.).
Additional comments: I agree with the authors that this paper and the guideline its presents can be of great value for educators. I hope the authors will revise the paper. I like to see it published.


Instruction: You are one of the reviewers, your task is to write a review for the article. You will be given the title of the article, the number of the round in which the article is located, and your order among the reviewers.

Input:
Title: GUIDELINES FOR USING EMPIRICAL STUDIES IN SOFTWARE ENGINEERING EDUCATION
Review round: 2
Reviewer: 3

Output:
Basic reporting: The article is written in English using clear and unambiguous text and conforms to professional standards of courtesy and expression.
However, the article should be finally reviewed by a native speaker.

The article includes a sufficient introduction and background to demonstrate how the work fits into the broader field of knowledge.
Relevant and comprehensive literature is appropriately referenced.
In the keyword list, "Onboarding" should be removed because this term is not used in the article (only in the title of a referenced article).

The structure of the article was improved in the first revision and now conforms to an acceptable format.
The article includes 12 tables which are relevant to the content. However, the layout of the tables (bordering) must be unified.
Furthermore, there is an inconsistency in Table 1: "Other instruments" is related to Subsection 4.6 renamed as "Further Instruments".
Aside from this inconsistency, the title "Instruments" is confusing because what you mean are the subjects of empirical study types
(Replication studies and OSS as crosscutting aspects).
I would also prefer to use the term "empirical study types" instead of "empirical instruments" in the title of the article.

The submission is "self-contained" that means it includes all relevant results with respect to the intention of the article.

Results basically include clear definitions of all terms and explanatory statements for the listed guidelines.
Minor issues are the following:
- Table 8: Change "Bachelor's level (first 4 years of study)" accordingly Row 611 to "Bachelor's level (0-3 years of study)"
- In Subsections 6.3 and 6.4, the authors write about types of case studies which are not listed in Section 4:   Intrinsic case study, explorative deductive study, and qualitative survey design, as well as explorative (curiosity-driven) and confirmative experiments.   These terms "break" the used framework of terms.
Experimental design: The submission clearly defines the research approach, which is relevant and meaningful for teachers in software engineering education.
The knowledge gap being investigated is identified. The guidelines for integrating empirical studies with software engineering courses contribute to filling that gap.
The investigation is performed to a high standard. The used research method is described with sufficient details.
Validity of the findings: The authors emphasize that the article provides an "INITIAL systematization of empirical instruments from the educational perspective" and also an INITIAL guideline based on their teaching experience.
They describe three examples of courses (Section 6) and discuss the limitations of their work.
The authors also clarify that further studies are needed to assess learning outcomes of courses using their proposed guidelines.
Additional comments: I like your first revision of your article with a clearer structure and clarifications of open issues. I see your guidelines as a basis for further discussions and experiments in software engineering education.
I recommend the article to be accepted if you consider in some revisions my minor issues and comments.


Instruction: You are one of the reviewers, your task is to write a review for the article. You will be given the title of the article, the number of the round in which the article is located, and your order among the reviewers.

Input:
Title: NOWCASTING COMMODITY PRICES USING SOCIAL MEDIA
Review round: 1
Reviewer: 1

Output:
Basic reporting: The authors are commended for reporting in a largely clear and unambiguous manner, particularly across languages. However, the English language should be improved through a closer proof-reading. Some examples include:

- Lines 31-32: confusing/disjointed sentence (e.g., break into two sentences);
- Footnote 1: pluralisation - "region" should be "regions";
- Line 70: check phrasing - "Indonesia is one of most tweeting countries..."
- Line 94: pluralisation - "is" should be "are";
- Line 108: missing word - "...we suggest [a] Nowcast model...";
- Line 113: check phrasing - "...from June 2012 to Sep 2013 total 78,518 tweets from 28,800 accounts..." --> e.g., could be rephrased as: "... to Sep 2013 [a] total [of] 78,518 tweets...";
- Line 136: "to be" --> should be "was";
- Figure 1 caption: "...variations due [to] geographic diversity..." (missing word). Also "multi-model" should be "multi-modal"?;
- Line 202: "The fist model we review..." --> should be "The [first] model we review...".

Terms and theorems are reasonably clear. However, there is some unnecessary repetition. For example, the variable δ (delta), which assigns bounds on price rate change, is defined several times (Lines 174-176; Lines 188-189; Lines 194-195).

Although sufficient background/context is provided, I think it would greatly enhance the paper if the authors provided more information about significance of the domain of the study. Food shortages in developing countries constitutes a serious social and governmental issue, and this research provides a valuable contribution to solving such problems. For example, in the Summary section, the authors state that "governments ... can make better policy decisions to protect vulnerable populations" (Lines 378-379). A brief sentence or two about how this research can enable "better policy decisions" would add value to the paper, particularly for interdisciplinary readers in policy and administration.
Experimental design: By and large, the experimental design is sound and to be commended. However, there are two minor aspects that would need to be addressed (or at least clarified) prior to accepting the paper.

Firstly, the aims and research questions and/or hypotheses of the study are not clearly defined. With some effort it is possible for the reader to discern these (e.g., from the Introduction section). I recommend that the authors (1) devote a brief paragraph to clearly state the aim(s) of the study; and (2) devise a set of research questions and/or hypotheses that clearly define what the study seeks to answer. These could be presented in a brief table on Page 2, for example. For the latter, this is particularly useful in relation to comparing the performance of alternative models and also for the analysis of Granger causality.

Secondly, the authors should be commended for their novel approach to filtering and analysing socially-generated data that are, by nature, noisy and error-prone. However, I query the removal of bot accounts (see Lines 123-126) from the data. The rationale for removing bots is that these accounts "have a disproportionate impact on the price monitor". However, the literature tells us that 'socialbots' play an important role in, for example, shaping markets and financial events (Steiner, 2012) and politics (Graham and Ackland, 2016; Ratkiewicz et al., 2011). In this way, the bots may have a potentially important 'social' role to play in commodity prices in Indonesia, even if their presence complicates the modelling problem. If the authors could clarify the removal of bots, that would be helpful.

References cited in Section 2 (of this review):

Graham, T., & Ackland, R. (2016). Do Socialbots Dream of Popping the Filter Bubble? The role of socialbots in promoting participatory democracy in social media. In Bakardjieva, M. & Gehl, R. (Eds.) (2016). Socialbots: digital media and the automation of sociality. Routledge: New York.

Ratkiewicz, J. Conniver, M. Meiss, M. Gonçalves, B. Patil, S., Flamini, A., and Menczer, F. (2011). Truthy: mapping the spread of astroturf in microblog streams. In Proc. 20th ACM Intl. World Wide Web Conf. Companion (WWW), pp. 249–252.

Steiner, C. (2012). Automate this: How algorithms came to rule our world. New York: Portfolio/Penguin.
Validity of the findings: Overall the findings are interesting and valid. I have two relatively minor comments for further improvements to the paper.

Firstly, the application of Granger causality (Lines 268-289) augments the analysis nicely and adds depth to the results. However, I have a point of clarification. The authors assume that, in this study, market prices are a non-stationary time series. This is fine. However, given that Granger causality assumes the signals to be covariance stationary, could the authors please outline any appropriate transformations they applied to the non-stationary data, i.e., to transform the time series into a stationary process. This will help to make the research reproducible and clarify the statistical rigour.

Secondly, as per Section 1 of this review, it would be helpful to clearly define the research questions at the beginning of the paper, in order to link these to the findings and conclusion at the end of the paper. I believe the findings to be valuable, both empirically and theoretically, so linking these clearly to research questions will only enhance this research.
Additional comments: Overall, the authors are commended for contributing a novel technique for modelling market prices based on social media data, as well as eliciting interesting empirical results to inform public policy and future research. I also praise the applied aspect of this paper (improving public policy in developing countries). I recommended this paper for publication, subject to a set of minor revisions, as outlined in detail in this review. Best wishes to the authors.


Instruction: You are one of the reviewers, your task is to write a review for the article. You will be given the title of the article, the number of the round in which the article is located, and your order among the reviewers.

Input:
Title: NOWCASTING COMMODITY PRICES USING SOCIAL MEDIA
Review round: 1
Reviewer: 2

Output:
Basic reporting: This article describes the use of prices reported in social media data to predict commodity prices.  Focusing on four food items in Indonesia, the paper presents the modelling approach and evaluation results comparing the prediction to ground truth data.  The paper also describes additional exploratory analyses to assess: (1) effect of data volume; and (2) credibility and user modelling.

Overall this article was a very insightful read.  The use of social media analytics to commodity prices is an interesting application.  The paper suffers at times from grammatical errors (see below for minor edits), however the overall presentation of the information and argument are clear and easy to follow.
Experimental design: In terms of the evaluation of the proposed model, the article could be strengthened by clarifying (1) the nature of the gold standard data; and (2) the usage of the data.

1.	In this work, the authors describe having access to historical data for actual food prices.  The source should be described and referenced if possible.  Some metadata description about the historical data would also be useful for the reader.   What time granularity does this come in?  How is the historical data collected?  How easily is this data compiled (perhaps to motivate the use of social media data).

2.	More importantly, the authors describe the use of the historical data to determine values for the parameters to the model, such as delta, the maximum allowable price change rate (line 195).  The authors describe this value as being tuned based on 80% of the data (line 245).  Presumably, the results reported in Figure 5 and Table 2 are on held-out data (presumably 20%).  However, this isn’t stated clearly anywhere.  Needless to say, if this is not the case (the reported results is on the whole historical data set), the NOWCAST model has some unfair advantage over the other models.
Validity of the findings: If the authors have used held-out data for the evaluation, the comparisons and findings seem sound (see previous section on Experimental Design).
Additional comments: This article describes the use of prices reported in social media data to predict commodity prices.  Focusing on four food items in Indonesia, the paper presents the modelling approach and evaluation results comparing the prediction to ground truth data.  The paper also describes additional exploratory analyses to assess: (1) effect of data volume; and (2) credibility and user modelling.

Overall this article was a very insightful read.  The use of social media analytics to commodity prices is an interesting application.  The paper suffers at times from grammatical errors (see below for minor edits), however the overall presentation of the information and argument are clear and easy to follow.

***Improvements***

In terms of the evaluation of the proposed model, the article could be strengthened by clarifying (1) the nature of the gold standard data; and (2) the usage of the data.

1.	In this work, the authors describe having access to historical data for actual food prices.  The source should be described and referenced if possible.  Some metadata description about the historical data would also be useful for the reader.   What time granularity does this come in?  How is the historical data collected?  How easily is this data compiled (perhaps to motivate the use of social media data).

2.	More importantly, the authors describe the use of the historical data to determine values for the parameters to the model, such as delta, the maximum allowable price change rate (line 195).  The authors describe this value as being tuned based on 80% of the data (line 245).  Presumably, the results reported in Figure 5 and Table 2 are on held-out data (presumably 20%).  However, this isn’t stated clearly anywhere.  Needless to say, if this is not the case (the reported results is on the whole historical data set), the NOWCAST model has some unfair advantage over the other models.

***General comments and questions***

3.	Related work
At a high level, it might be interesting to comment on other well-known uses of Twitter for prediction.  For example, the work by Asur and Huberman (2010) in using Twitter to predict movie box office sales is rather well known (http://dl.acm.org/citation.cfm?id=1914092).   It would be interesting to know how the proposed model compares to prior work such as the one proposed by Asur and Huberman.  Even a discussion about the applicability of the prior work would be useful.

4.	With respect to Natural Language Processing
The decision to use minimal natural language processing techniques is understandable.  However the article presents the model as being robust to variations in the way one can express prices, leaving it to the model to make the appropriate inference.

Line 107: “Instead of using hundreds of regular expressions for normalizing 108 various types of units into an identical units, we suggest Nowcast model which can handle a commodity 109 unit difference issue via numerical approach.”

This is perhaps due only to the specific commodities analysed in the article.  (One would need more evidence and examples to show that text variation is systematically handled by the model, which seems unlikely.)  Perhaps, state that, for the purpose of simplicity of the model, by assuming that prices for specific foods fall within distinct price ranges.

Line 297, “Hence, our model has minimum language dependency”.  This should be clarified to say that aside from the taxonomy (Table 1) and the queries used to collect the data, the approach makes little use of linguistic information.   Imagine the case for Chinese, Korean or Japanese where word segmentation is a problem and may hinder the creation of similar regular expressions as found in Table 1.

5.	With respect to data/results

On Line 96, the authors write: “Four basic food commodities were chosen for monitoring: beef, chicken, onion, and chilli, based on the availability of data in terms of tweet mentions and the country-level priorities for food security”.  Where there any country-specific priorities that weren’t found on Twitter?  It would also be useful to know what limitations there are in using Twitter for this purpose.

On Line 118, the authors write “a tweet is an exact duplication of another (22.9%)”.  Exact duplicates are often retweets or people sharing news articles.  To what extent does the data come from news articles, and what effect does that have on the approach?  If these are indeed news, could one simply have analysed the news articles rather than Twitter?

Line 152, “ beef price jumped to 1.4Mrupiah – 17.5 times larger than official price – for certain days in July2012 from our data”  Comment: Is there an error analysis for the spike?

Line 288, “social price is reflected from real-world price and social price may also affect real-world price simultaneously” Comment: To what extent is this feedback loop dependent on the maturation times to grow both commodities?

6.	With respect to generality of methods

Line 302, “The nowcast model … predict trends in other essential commodities and across countries”
While the proposed method is attractive, it’s applicability to other countries remains to be seen and so this claim may be too general.

For example, Indonesia, as the authors point out, has very high usage levels of Twitter.  For this reason, projects like Petajakarta (using Twitter to gauge flooding levels in Indonesia) have been very successful (https://petajakarta.org/banjir/en/research/; From Social Media to GeoSocial Intelligence: Crowdsourcing Civic Co-management for Flood Response in Jakarta, Indonesia T Holderness, E Turpin. Social Media for Government Services, 115-133).

Similarly, the modelling approach in this paper relies on the heavy usage of Twitter by the community, in this case to report on food prices.  This may not be the case in other countries where the predominant online social network is not Twitter (for example, Facebook, which has no search facility).  Therefore, while the approach is an attractive one and possibly useful in scenarios outside of Indonesia, one may want to be cautious regarding claims about the generality of the approach.

Minor Corrections:

Line 28, Grammatical or Spelling error:
   is largely contributed -> is largely due

Line 30, Grammatical or Spelling error:
   Nonetheless, because commodity price and in particular food insecurity in developing regions is extremely dynamic, the ability to track market status quickly at times it is harder to predict is all the more a critical challenge (Gouel, 2013).
->
   Nonetheless, because commodity price and, in particular, food insecurity in developing regions is extremely dynamic, the ability to track market status quickly is, at times, hard, making prediction an all the more a critical challenge (Gouel, 2013).

Line 35, Suggested change
   Technology enables developing countries to reach a level of financial data access that is available in more developed economies
->
[Comment: Assuming you mean mobile technologies]
 Such technology (mobile) enables developing countries to reach a level of financial data access that is available in more developed economies.

Line 38, Grammatical or Spelling error:
    price indexes-> price indices

Line 51, Grammatical or Spelling error:
    people behavior in crisis situations -> people’s behaviour in crisis situations

Line 70, Grammatical or Spelling error:
   Twitter was chosen as data source -> Twitter was chosen as a data source

Line 71, Grammatical or Spelling error:
    Indonesia is one of most tweeting countries ->  Indonesia has one of the highest adoption rates for Twitter

Line 75, Grammatical or Spelling error:
    from the Indonesia region -> from the Indonesian region

Line 76, Suggested change
   Another filter on numerical test was also used … -> A (numerical) model parameter was also used …

Line 78, Grammatical or Spelling error:
   In the second step, a statistical model was then built to accurately estimate food prices for each commodity in the way OSN data could assist official statistics published by the Indonesian government.
->
   In the second step, a statistical model, using OSN data, was then built to accurately estimate food prices for each commodity in order to assist with the publication of official statistics by the Indonesian government.

Line 94, Grammatical or Spelling error:
    there is enough online signals -> there are enough online signals

Line 111, Suggested change
   “Keyword combination for tweet collection: ( Commodity Names ) AND ( Price Values ) AND ( Price Units | Commodity Units )”
[Comment: This information seems to occur unexpectedly and could do with better introduction.]

Line 120, Grammatical or Spelling error:
   Mostly major accounts -> Most major accounts

Line 121, Grammatical or Spelling error:
   This finding infers that -> This finding indicates that

Line 136, Grammatical or Spelling error:
    the average number of tweets per account to be 2.73 -> the average number of tweets per account is 2.73

Figure 1, Grammatical or Spelling error:
    while solid red line is government official price in identical period.
-> while the solid red line is the government official price for the identical period.

Line 142, Grammatical or Spelling error:
    price rising period than price decreasing period ->     a price-rising period compared to a price-decreasing period.

Line 161, Grammatical or Spelling error:
    that of the yesterday’s -> that of yesterday’s

Line 181, Grammatical or Spelling error:
   nowcast model assumes there is no price changing. ->    nowcast model assumes there is no change in price.

Line 194, Grammatical or Spelling error:
    within certain bound. -> within a certain bound.

Line 206, Grammatical or Spelling error:
   perform poor when the data is multi-modal, ->    performs poorly when the data is multi-modal,
[Comment: defining “multi-modal” may be useful for the reader.]

Line 262, Suggested change:
  [Comment: An added figure and an explanatory paragraph highlighting the two data sets being correlated (presumably, price predictions in time for the held-out data) would be useful for the reader.]

Line 271, Suggested change:
   According to the concept that suggested by Granger (Granger, 1969) … -> Granger (1969) argues that for two simultaneously measured time series …

Line 281, Suggested change:
   We find that nowcasted time series for every target commodity have greater significance level of causal influences.
[Comment: this is unclear, please rephrase.]

Line 292, Grammatical or Spelling error:
    help predict economic segments -> helps predict economic behaviour

Line 329, Grammatical or Spelling error:
   Another kind of data issues to be considered -> Another data issue to be considered

Line  458,
Vieweg, S., Olteanu, A., and Castillo, C. (2015). What to expect when the unexpected happens: Social 459 media communications across crises. Proceedings of CSCW 2015 (forthcoming).
[Comment: presumably a 2015 is no longer forthcoming]


Instruction: You are one of the reviewers, your task is to write a review for the article. You will be given the title of the article, the number of the round in which the article is located, and your order among the reviewers.

Input:
Title: NOWCASTING COMMODITY PRICES USING SOCIAL MEDIA
Review round: 2
Reviewer: 1

Output:
Basic reporting: I am satisfied that the authors have addressed all the issues relating to "Basic reporting" as detailed in my original review of this paper (the changes to the paper are summarised in the Rebuttal).
Experimental design: I am satisfied that the authors have addressed all the issues relating to "Experimental design" as detailed in my original review of this paper (the changes to the paper are summarised in the Rebuttal).
Validity of the findings: I am satisfied that the authors have addressed all the issues relating to "Validity of the findings" as detailed in my original review of this paper (the changes to the paper are summarised in the Rebuttal).
Additional comments: I believe that the revised paper has addressed all of my main concerns in the previous review, and therefore should be accepted as is. Congratulations to the authors for tackling an important issue using novel methods. I hope to see more of this kind of work in the future.


Instruction: You are one of the reviewers, your task is to write a review for the article. You will be given the title of the article, the number of the round in which the article is located, and your order among the reviewers.

Input:
Title: NOWCASTING COMMODITY PRICES USING SOCIAL MEDIA
Review round: 2
Reviewer: 2

Output:
Basic reporting: The authors have addressed the reviewer comments adequately.  The introduction and motivation for the work is clearer.
Experimental design: The authors response argues that clarification has been added about (1) how the data was sourced and filtered; (2) how the data was used.

The changes for (1) are satisfactory.

For (2) although this is a largely unsupervised approach (except for the delta parameter), it would still be useful to clarify if the subset of data used to tune parameters (which is 80% of the first 3 months) was separate to the data used to calculate the daily prediction error and the correlation coefficient results in Figure 5 and Table 3.

That is, for these reported results, are the metrics calculated over the 15 months data except for the developmental training data?

The authors do note that all compared models use the same data splits for evaluation (so the comparison between models could be fair).  The only concern is whether the report errors and correlation coefficients are higher that what they would be on completely held-out data, which will be the case in a real-world application.  (One would expect that this is unlikely given that only the delta parameter is determined using the development data, otherwise the approach is largely unsupervised.)

If this is specified anywhere in the article, it should be clarified again around line 300.
Validity of the findings: As mentioned in #2 above, the only remaining concern to be addressed is whether the reported correlation coefficients are indicative of what one would expect in a real-world application.
Additional comments: The authors have made great progress with the article and it is nearly ready for publication.  The only outstanding substantial issue is to clarify which subset of the 15 months worth of data the performance metrics were calculated on.  Ideally, the development set was not used in reporting these correlations, otherwise the results may be artificially elevated (at least for the first 3 months of the 15 months of data).  Again, one expects the difference not to be large since there would be an addition 12 months of data; still this is a point to be addressed in the article.

One suggestion to tackle this problem would be for the authors to describe the approach as one that requires some initial data set is needed to calculate delta, after which the approach can be used in an unsupervised manner on unseen data.

Minor changes:
Page 1, citation mistake:
	line 41
  Content: " Premise (Premise, 2016)."  -> remove name from parenthesis

Page 2, grammatical or spelling mistake:
	line 86
  Content: " therefore we could clarifies that the price"
  Comment:  therefore we can verify that the price

Page 2, Suggested change:
	line 97
  Content: "This paper also describes the effect of several important social network-wide variables, via testing the robustness of the model under data scarcity conditions and by modeling user-level credibility to suggest an enhanced sampling strategy."
  Comment: This paper also describes the effect of several important social network-wide variables, such as (1) testing the robustness of the model under data scarcity conditions, and (2) modeling user-level credibility to suggest an enhanced sampling strategy.

Page 3, grammatical or spelling mistake:
	line 116
  Content: "monitoring in consultantation with the Ministry of"
  Comment: monitoring, defined in consultation
with the Ministry of ...

Page 4, grammatical or spelling mistake:
	line 144
  Content: "Tweet data contain noisy information and need to be cleaned prior to analysis"
  Comment: Tweet data contains noisy information (for example, ...) and needs to be cleaned prior to analysis.

Page 4, Suggested change:
	line 144
  Content: "We employed the following measures in data cleaning. First involves removing ambiguity in meaning"
  Comment: For example, we use filters to remove data that is ambiguous in meaning.

Page 4, Highlight (Yellow):
	line 147
  Content: "5% of the price quotation data"
  Comment: Please clarify, at this point, that this 5% is removed (this is stated later but would be helpful to the reader to state here)).

Page 4, Suggested change:
	line 152
  Content: " removing redundant messages or spam bots."
  Comment:  removing redundant messages from spam bots.

Page 4, grammatical or spelling mistake:
	near line 163
  Content: "Meet Ad"
  Comment: Meat Ad

Page 4, Suggested change:
	line 165
  Content: " and it seems not reasonable to assign more credibility to bot-tweeted information based on its proportion of volume than human-tweeted information"
  Comment: (End of sentence) .  Correspondingly we do not want to assign more credibility to bot-tweeted information based on its proportion of volume than human-tweeted information.

Page 4, grammatical or spelling mistake:
	line 172
  Content: " a tweet is an exact duplication of another "
  Comment:  a tweet is an exact duplicate of another

Page 5, Suggested change:
  Content: " data sensitization"
  Comment: Define sensitization

Page 12, grammatical or spelling mistake:
	near line 399
  Content: "Figure 7B "Follwers"
  Content: Followers


Instruction: You are one of the reviewers, your task is to write a review for the article. You will be given the title of the article, the number of the round in which the article is located, and your order among the reviewers.

Input:
Title: A METHOD FOR CREATING INTERACTIVE, USER-RESEMBLING AVATARS
Review round: 1
Reviewer: 1

Output:
Basic reporting: Given article proposes the method for addressing the problem of creating realistic graphical representation of the user for VR environment with available commercial technologies. In order to keep the review coherent, the comments for each sections are given following the structure of the paper.

Introduction is well-organized, coherent. In the first paragraphs definition of VR is introduced and its development history is shortly described. The key issue of control over immersion and interactivity is introduced and the possible solution using multiple technologies of motion tracking and HMDs is proposed.
[Line 27, 37, 38, 40] The abbreviation of head mounted devices spelled wrong
[Line 47]: Instead of “ideal system” it is suggested to say “Ideally, this system ..”
[Line 49-50]  Stating the motivation incoherently, move the sentence to following paragraph to line 57 “ In accordance with growing evidence…  ,… this article presents…”

In the “Background and related work’ section authors described technology of implementation, immersion factors and importance of user resembling avatar embodiment. It is suggested to include comparison of available VR development technology and choice justification.
[Line 75-79] Reference the technical specifications
[Line 84] Maybe “improved the first iteration” instead of “improves on the first iteration”?
[Line 108-109] how do those levels influence or was exploited in the presented system?

“Materials and methods” section describes software implementation and hardware use of the proposed method. It would be easier to follow the implementation process if authors would use dataflow (fig. 2) diagram during the explanation of core functionality of the system.
[Line 132-134] Repetition the information from background
[Line 155] Typo: “of” instead “on”
[Line 160]The concept of point cloud should be referenced
[Line 230-231]Reference the publications

In the “Results” section two types of avatar visualization and attempts of its interactivity implementation are presented. First and third perspectives of avatar are shown in figures.

Discussion’s subsections of avatar resemblance as an immersion factor and possible applications of  VR could be moved to the introduction and background sections, leaving the limitation and further improvements for discussion. In this section it is also suggested to mention eye and gaze tracking technologies for improvement immersion quality as it influences FOV (line 100).
[Line 279-280] The features were not mentioned in the table of requirements introduced on line 151.
Experimental design: no comments
Validity of the findings: no comments
Additional comments: no comments


Instruction: You are one of the reviewers, your task is to write a review for the article. You will be given the title of the article, the number of the round in which the article is located, and your order among the reviewers.

Input:
Title: A METHOD FOR CREATING INTERACTIVE, USER-RESEMBLING AVATARS
Review round: 1
Reviewer: 2

Output:
Basic reporting: no comment
Experimental design: no comments
Validity of the findings: no comments
Additional comments: Well-known depth cameras and libraries to deal with complex physics interaction. The authors fail to properly describe what was the research outcome and impact. Novelty part is very weak.


Instruction: You are one of the reviewers, your task is to write a review for the article. You will be given the title of the article, the number of the round in which the article is located, and your order among the reviewers.

Input:
Title: A METHOD FOR CREATING INTERACTIVE, USER-RESEMBLING AVATARS
Review round: 1
Reviewer: 3

Output:
Basic reporting: The paper is written well, however, there are several grammatical issues throughout the text. The literature review is very relavant and follow well with the current state if the art.

The main issue that I see is that the contribution of the work is limited on a theoretical level. All the methods mentioned is the paper is already published. On an implementation side of things, the library provided is indeed useful for those working on developing VR tools.

Another main issue in the limitations imposed by point could data. The sensitivity of these can have a several practical implementation such a system on the quality and speed.
Experimental design: The experimental design is rather not very complete. While the approach adopted is to report the implementation details, it does not sufficiently address the performance issues and limitations.

The impact of reduced data points and the mismatchs needs be analyzed in detail. The simulations also needs to take into account different platforms such as that in mobile devices, which is growing popular for these applications.

The performance analysis also needs to include a large number of images and have a statistical analysis to be scientifically relavant. On the implementation side, some aspect of software testing and code optimization would also benefit to validate the usefulness of the method.
Validity of the findings: The novelty is not sufficiently justified. The new combination of methods seems to be the main scientific contribution, however, it is not enough contribution to justify publication. I would suggest the authors to justify and analyze the combination methods, and compare with other methods to justify the originality.
Additional comments: No additional comments


Instruction: You are one of the reviewers, your task is to write a review for the article. You will be given the title of the article, the number of the round in which the article is located, and your order among the reviewers.

Input:
Title: A METHOD FOR CREATING INTERACTIVE, USER-RESEMBLING AVATARS
Review round: 2
Reviewer: 1

Output:
Basic reporting: Authors addressed all comments given in the previous review. However, it is recommended to use language editing service to avoid stylistic issues throughout the paper.
Experimental design: no comment
Validity of the findings: no comment
Additional comments: no comment


Instruction: You are one of the reviewers, your task is to write a review for the article. You will be given the title of the article, the number of the round in which the article is located, and your order among the reviewers.

Input:
Title: PREDICTION OF PROTEIN FUNCTION USING A DEEP CONVOLUTIONAL NEURAL NETWORK ENSEMBLE
Review round: 1
Reviewer: 1

Output:
Basic reporting: There are a few small errors with English.
Experimental design: Zacharaki has developed a new method to predict the EC class of a protein structure, known to be an enzyme. The results are superficially excellent (~90% accuracy), but I think this is due to using a flawed dataset, to a large extent. The author uses all the enzyme PDB files. As many of these files are from the same enzyme (e.g. there are 1734 lysozymes), the method will usually work simply by recognising itself. For example, if the unknown protein is a lysozyme and there is a lysozyme in the training set, then all an algorithm needs to do is to spot a copy of itself in the training set (k-mean algorithms will do this). If a dataset like this is used, then there is no need to develop any kind of new algorithm, since PSI-BLAST will already work perfectly well. It may well, however, work poorly on any structure that does not have a homologue in the training set – hence it is overfit.

I therefore think that the work should be redone using a data set that has no pairs of similar proteins. A sequence cut-off of no more than 20% sequence identity could be used. The author says herself:  “Assessment of the relationship between function and structure (Todd et al., 2001) revealed 95% conservation of the fourth EC digit for proteins with up to 30% sequence identity.” I think that is why the method works – if proteins are present with high sequence identity, then they have the same EC number.

It is assumed that the protein is an enzyme in advance (with a single EC number). The work would be much more powerful if it was coupled with predicting whether it is an enzyme, since then it could be applied to any protein structure.

If these changes are made, and the model has a good performance, it could be publishable, as the method is original.
Validity of the findings: The numbers of proteins in each EC class look very odd. The pdb (http://www.rcsb.org/pdb/browse/jbrowse.do?t=3&useMenu=no) gives these frequencies for each class:
•	  1: Oxidoreductases - [ 12950 Entities ]
•	  2: Transferases - [ 20733 Entities ]
•	  3: Hydrolases - [ 28864 Entities ]
•	  4: Lyases - [ 5705 Entities ]
•	  5: Isomerases - [ 2852 Entities ]
•	  6: Ligases - [ 2747 Entities ]

Why are the numbers in Table 1 so different? Ligases should be the rarest, but here they are the most frequent.
Additional comments: It is informative to report a confusion matrix for each model, i.e. numbers of false positives, false negatives etc., rather than just accuracy.

“The assessment of structure/function relationship however is hampered by the lack of a unified functional classification scheme of the protein universe.” What about SCOP, CATH or DALI?

Figure 1 is hard to understand. More explanation in the legend would help. What is Bnorm, for example?

The axes in Figure 2 should be labelled.

Figure 2 is simply low resolution Ramachandran plots for each amino acid. I don’t think it is helpful to put every amino acid on the same scale of 0,4. Rare amino acids will then have few visible features, so very rare (e.g. Asx) are solid blue.

Clarify in Table 1 that the correlation columns are the k-mean results.

Do the colours in Figure 4 largely reflect the abundance of each amino acid in each class? For example, does EC2 have a lot of Ala, thus making the plot nearly all yellow? It would probably be clearer to rescale for each plot, as all yellow or all blue is unhelpful.

Say what the 23 rows in Figure 4 are.


Instruction: You are one of the reviewers, your task is to write a review for the article. You will be given the title of the article, the number of the round in which the article is located, and your order among the reviewers.

Input:
Title: PREDICTION OF PROTEIN FUNCTION USING A DEEP CONVOLUTIONAL NEURAL NETWORK ENSEMBLE
Review round: 1
Reviewer: 2

Output:
Basic reporting: No Comments
Experimental design: There are some issues with the experimental design:

1) the experimental data contains redundancy/ That is, some proteins are highly similar at sequence level or structure level. The authors shall remove highly redundant proteins  from both the training and validation set. For example, they can use 30% or 40% sequence identity as cutoff to exclude redundancy.

2) is there any redundancy between the training and validation set?
Validity of the findings: There is no detailed comparison with the other methods using the same set of test proteins. It is hard to judge if the proposed method indeed improves the state of the art or not.
Additional comments: No additional comments


Instruction: You are one of the reviewers, your task is to write a review for the article. You will be given the title of the article, the number of the round in which the article is located, and your order among the reviewers.

Input:
Title: PREDICTION OF PROTEIN FUNCTION USING A DEEP CONVOLUTIONAL NEURAL NETWORK ENSEMBLE
Review round: 2
Reviewer: 1

Output:
Basic reporting: I still think that changing the scales in Figure 3 would help. Solid blue plots are of no value.

Similarly, I still think that rescaling Figure 5 so that the plots are not nearly all blue would be helpful.
Experimental design: My main previous recommendation was to see whether the method works without redundancy. Unfortunately, I still think that the additional work to address this issue is unsatisfactory. The work is redone using a subset that is claimed to be non-redundant. Non-redundant is not defined, however. A reasonable definition would be no two pairs of proteins have more than 20% sequence identity when aligned. Using the PISCES site with the “highest resolution structure available and then the best R-values” would be a resolution of 1.6Å and an R-value of 0.25. Even with a sequence identity cut-off of 90%, there are still 7085 PDB files with these criteria on PISCES, as of 23/2/17. I don’t understand where the subset of 23242 proteins comes from. In any event, there must still be much redundancy if that many proteins are used. I therefore don’t think the issue of how well the method works in the absence of redundancy has been addressed.

I still think that section 2.2 would benefit from more explanation.
Validity of the findings: I think the method should be tested on a true non-redundant data set, with no protein pairs having a sequence identity of >20% when aligned.

All my other suggestions have been satisfactorily addressed.
Additional comments: No additional comments


Instruction: You are one of the reviewers, your task is to write a review for the article. You will be given the title of the article, the number of the round in which the article is located, and your order among the reviewers.

Input:
Title: PREDICTION OF PROTEIN FUNCTION USING A DEEP CONVOLUTIONAL NEURAL NETWORK ENSEMBLE
Review round: 3
Reviewer: 1

Output:
Basic reporting: No comment
Experimental design: No comment
Validity of the findings: No comment
Additional comments: I previously recommended using a truly non-redundant data set to assess the algorithm. This has still not been done. However, I accept that large data sets are required for the deep learning methods used here. The text has been rewritten to clarify this issue.

My other suggestions have been adequately addressed.

I am therefore pleased to support publication now.