Instruction: You are one of the reviewers, your task is to write a review for the article. You will be given the title of the article, the number of the round in which the article is located, and your order among the reviewers.

Input:
Title: JOURNAL OF OPEN SOURCE SOFTWARE (JOSS): DESIGN AND FIRST-YEAR REVIEW
Review round: 1
Reviewer: 2

Output:
Basic reporting: This manuscript describes the design of the Journal of Open Source Software and describes experiences after the first year since its inception. Let me start by mentioning that I fully support the ambition to publish open source software in articles that describe the software and which reviews some quality metrics and which makes it possible to cite software in scientific literature, with proper acknowledgements of the authors.  I also appreciate the openness of the process and I congratulate the authors of this manuscript with the success of JOSS so far.

A paper like this can hardly be rejected on scientific merit, since it describes a process, and my comments are mostly an advice, sometimes a personal note and rarely a correction.

Section 1
Firstly, the intended audience is not  entirely clear. There is quite a bit of technical jargon regarding software development in the paper. I think the jargon relates to the community of research software engineers and possibly bit wider than that (including this reviewer). The subject of publishing software in articles, making it citable and having the reward and recognition system in place for research software engineers in a wider academic context is indeed insufficiently appreciated. This paper is not going to respond to this discussion directly, so I suggest to make clear in the beginning that the targeted community is the already specialized research software community and readers who are likely to submit to this journal, rather than the wider academic community.

Related to the above, a bit wider discussion on the open science aspects would be appreciated (still within the scope of this particular article). I support the open software movement related to open science, but there are drawbacks and criticisms as well. Which aspects do you explicitly not resolve in this set up? The authors mention the ethics of open source in the beginning and describe at the end the issue of software development, but not the openness part. How does this journal contribute to the wider development of open science practices (open access clearly, but beyond open data to open software)? This is a subject which could be described in the introduction a bit more rather than taking it for granted

Comparable journals do exist, also journals that keep discussion papers open (the abstract suggests this is the first journal), e.g. those of the European Geophysical Union, including software and data journals such as GMD. It would be good to highlight more what the niche of JOSS is wrt these domain journals with similar goals.

It may also be relevant to refer and discuss instrumentation journals which have a similar goal. They don’t publish new scientific results, but rather describe instruments, calibrations etc.

A comment is that journal seems to be very US centered, with some exceptions, how does this reflect in article submissions? Is there a strategy in place to broaden it?

Section 1 ends without giving a summary what the journal entails. It is described further. It is about publishing software, but it doesn’t say how far it goes in that (does it include the code, documentation, tests or is it just a description?). A summary is needed for the reader.

Section 2.
While I agree with this section a number of comments:

The authors mention that actual scholarship is the full software environment, code that produced the data. This is a very limited view of science and scholarly research. It is a very technological view of scientific research only that suggests that the whole creative process of research is in software development. Also, it takes the pure academic view only, while 21st century research has tight connections to societal, economical and ecological challenges which is not reflected in a code. I highly disagree with the statement and I suggest to phrase it much more carefully. This is a personal note.

The motivation of this journal should not be the career of the research software engineers. It should be good scientific practice. As a side effect it could help the career development, but this problem relates more the reward and evaluation system in place in many academic institutions.
Having this journal may solve one aspect, but not the actual cause of lack of career perspectives of software engineers in the scientific community. This is to be addressed in the academic system, in particular the leadership of it.

That developers shun away from publishing a citable paper is not a strong argument. They should do so as part of good scientific practice since it contains an independent peer-review quality check, a time stamp and allows for referencing. The notion that a paper is also a time stamp recognizing the original contribution of the author to the scientific process could also be more clearly spelled out.

Note that there are quite a few domain specific journals that do not require new scientific results. From my own personal experience, as an editor of one of them, it is more a problem of culture. The reviewers focus a lot on the scientific results and find it hard to review papers purely based on methodologies and tools.

Section 3

'dressed up as ...' that sounds a bit weak.

It is unclear whether JOSS also checks whether the software is original, in the sense that it doesn’t re-invent a wheel. One of the goals of peer reviews of scientific papers is that the author is the first one to discover a result and puts a time stamp on it (see above). Does the journal allow any research software or does it help to avoid re-inventing the wheel? I miss this aspect about originality (see also section 4 on review criteria where it seems to be hidden in statement of need).

This section is a bit unclear on what is actually the target. On the one hand it is ‘ just’  advertising, on the other hand it mentions detailed code review.

If only high level functioning is reviewed, what is there actually to review in JOSS. How is quality and originality defined? So, how to determine impact and quality as peer review of scientific papers generally does? Does the code review have this role? Does this include coding standards, bench marking, checking for correctness of scientific result (in addition to code tests).

Section 3.4 compares only to generic research software journals. It would be good to compare to some domain journals as well.

Section 4/5

I find a minimum of one reviewer really low. What was the rationale to do so?

Given the review criteria, the scope of JOSS is much more than an advertisement. It is unclear though how much of the research application is required as well. It seems to focus primarily on the software aspects as such, not to the research application and the appropriateness of the software for that. This may be a deliberate choice, since more specialized domain journals pick this up, but it should be clearly stated.

I would be interested in two additional statistics: the country of affiliation of the first author (US centric?) and the distribution over the research application domains.

Section 6

The last paragraph is very long, a bit unclear and unsatisfactory in its ending. Is there a process foreseen to come to a choice to deal with this for JOSS?

Section 7

The first paragraph mentions the career perspective of RSEs, but this cannot be the reason to set up JOSS. It is set up because software is an essential part of the research process, and therefore it should be properly evaluated and published such that it is findable, with a measure of quality and originality.

I miss the open science perspective in this concluding section.

In conclusion, I have some remarks on the paper, but I strongly support JOSS as it is essential to publish research software and contribute to open science. I hope it will develop further in the near future!

Wilco Hazeleger
Experimental design: No specific comments
Validity of the findings: No specific comments
Additional comments: No specific comments


Instruction: You are one of the reviewers, your task is to write a review for the article. You will be given the title of the article, the number of the round in which the article is located, and your order among the reviewers.

Input:
Title: JOURNAL OF OPEN SOURCE SOFTWARE (JOSS): DESIGN AND FIRST-YEAR REVIEW
Review round: 1
Reviewer: 3

Output:
Basic reporting: The English is excellent, clear and unambiguous. The literature references provide the appropriate level of background and context.
The article is self-contained and uses a professional article structure. Data are available from a figshare article.

section 2 is a little repetitive (the final paragraph) as the case for garnering credit for software has been well made.
Experimental design: The purpose of the paper - which is to present the design of a new journal targetted at giving software "proxy" papers in a lightweught way and review its first year of operation - is clearly stated and presented.
The case for such a journal is  convincingly made.
Section 3.1 (goals and principles)  clearly presents the principles and justifications, and the mechanics of the Journal submission system and workflow are described in some detail in sections 3.2 and 4.

1. However, the design from the reader's experience is almost entirely missing. Journals have authors, editors/reviewers and readers, and it seems that the design for the reader experience is overlooked.
The Journal does not seem to have, for example, search functions and at over 246 articles (at time of access) a list without tags or labels is hard to navigate. Section 6 (the next year) focuses on automation of the submission process, including handling versions of previous submissions, but offers little in the way of features for readers.

It seems that the focus is on assigning a marker for software rather than a journal for seeking software articles. If this is an explicit design decision this should be stated.

The article focuses its design decisions on the editorial and reviewing ease of use. It would be instructive to have some feedback on the author and reader experience.

2. The review process primarily uses 1 reviewer - what is the confidence that this is sufficient? If the review process is primarily checklist based (as appears) with little judgement, then this could be enough.
Validity of the findings: The number of articles submitted and published is a clear indication of the justification and value of the Journal and its place in the ecosystem. The data presented is robust. Partnership with OSI and NumFOCUS is encouraging.

The review of the first year gives useful information. Some points that would improve the paper

1. the statistics given are from the point of view of the editorial team. What would be more interesting would be an analysis the submissions in the first year, by article topic, community served, geographic distributions (is this USA or European focused?) and so on. It is hard to tell from the user interface at http://joss.theoj.org/papers

2. As described in review section 2, there is little discussion of the reader experience. For example, I could not easily find if the Whedon RubyGem Library has it own article.

3. in addition to thoughts on versions, has the journal developed a strategy on software forking, in particular by different developer teams to the original developers?

4. although it is early days, is there any preliminary information on citation patterns for the articles? there is a hint only in "we may need to invest more effort into raising awareness of good practices for citing JOSS articles"

5. the reviewer counts and reviewer lists are different - are the reviewers rewarded? the implication is that the volunteers are not known.

All in all the article presents an innovative journal serving the software developer community, and I commend the the authors. The weaknesses are the sole focus on the editorial process and less on the reader and citation experience. I recognise that it is early days but some remarks on citation (the justifications given in section 2) would be welcome.
Additional comments: See sections above


Instruction: You are one of the reviewers, your task is to write a review for the article. You will be given the title of the article, the number of the round in which the article is located, and your order among the reviewers.

Input:
Title: JOURNAL OF OPEN SOURCE SOFTWARE (JOSS): DESIGN AND FIRST-YEAR REVIEW
Review round: 2
Reviewer: 1

Output:
Basic reporting: The article is a special type of paper as it reviews the set up and experience of JOSS after 1 year. It is clearly written and the authors responded well to my comments. Otherwise, no comment.
Experimental design: See above, no comment.
Validity of the findings: See above, no comment.
Additional comments: Although I am still a bit concerned about the number of reviewers per article, I accept the arguments made and the role of the reviewers, which is different from those for a traditional scientific article. I congratulate the authors with the setup of JOSS and hope it will be a success.


Instruction: You are one of the reviewers, your task is to write a review for the article. You will be given the title of the article, the number of the round in which the article is located, and your order among the reviewers.

Input:
Title: JOURNAL OF OPEN SOURCE SOFTWARE (JOSS): DESIGN AND FIRST-YEAR REVIEW
Review round: 2
Reviewer: 2

Output:
Basic reporting: This is a resubmission - I will not repeat my previous comments, only commenting on the rebuttal and updates

The reporting is clearer. The scope of the journal, its primary aim and its relationship to other software journals is much clearer.
Experimental design: This is a resubmission - I will not repeat my previous comments, only commenting on the rebuttal and updates

The key point is the purpose of the journal - which is now much more clearly stated as a software developer publishing vehicle rather than a "traditional" journal where originality are part of the criteria and peer review is about quality gatekeeping rather than preparing to a reporting standard.  This makes the various design and implementation decisions much more understandable.

The future features to embrace analysis, tagging, reader features, citation tracking and versioning are encouraging.
Validity of the findings: This is a resubmission - I will not repeat my previous comments, only commenting on the rebuttal and updates

Figure 8 and the new paragraphs on page 15 and page 17 have helped to address my comments on statistics and update strategies
Additional comments: This is a resubmission - I will not repeat my previous comments, only commenting on the rebuttal and updates

Clarifying the purpose and scope of the journal has considerably helped address some of the quirks of the design and reviewing methodology.

You are challenging not just the need for a forum for publishing new forms of research - like software - but also what it means to publish and to whose benefit it is. That will confuse traditionalists.


Instruction: You are one of the reviewers, your task is to write a review for the article. You will be given the title of the article, the number of the round in which the article is located, and your order among the reviewers.

Input:
Title: MCLEAN: MULTILEVEL CLUSTERING EXPLORATION AS NETWORK
Review round: 1
Reviewer: 1

Output:
Basic reporting: The paper is very well written and thus straightforward to read: The English is clear and unambiguous, literature references and background are provided and the given figures are helpful to support the information presented in the text.
Just one idea for a possible improvement: To the inexperienced user it not obvious how the network representation of the clustered dataset (figure 4) results from the node-link network transformation (figure 3). Maybe it would help to orient the figures such that the correspondences between aggregated nodes and the original clusters can be more easily identified.
Experimental design: The paper presents a novel method for interactively exploring complex datasets in order to discover patterns and clusters. The novelty lies in the combination of computational analysis and visual interpretation. In view of the plethora of ever more detailed and complex data becoming available in many fields, a tool to aid in analysis is very relevant.
The paper adequately explains the underlying computational methods and demonstrates the capabilities of the new tool using a suitable data set.
Validity of the findings: The authors validate their tool by studying the usage by members of their target audience. Thus, they can establish that the new tool has the desired advantages over state of the art methods.
Additional comments: No additional comments


Instruction: You are one of the reviewers, your task is to write a review for the article. You will be given the title of the article, the number of the round in which the article is located, and your order among the reviewers.

Input:
Title: MCLEAN: MULTILEVEL CLUSTERING EXPLORATION AS NETWORK
Review round: 1
Reviewer: 2

Output:
Basic reporting: The article is up-to-date and well structured with references form the last years and present clearly the advantages of the study performed.
Experimental design: In an efficient manner the methodology aims to highlight the multilevel representations of the clustered dataset with community finding algorithms.
Validity of the findings: The article exposes clearly the results of the study performed.
Additional comments: The article is up-to-date and well structured with references form the last years and present clearly the advantages of the study performed.
In an efficient manner the methodology aims to highlight the multilevel representations of the clustered dataset with community finding algorithms.
The article exposes clearly the results of the study performed.


Instruction: You are one of the reviewers, your task is to write a review for the article. You will be given the title of the article, the number of the round in which the article is located, and your order among the reviewers.

Input:
Title: THE MODERN RESEARCH DATA PORTAL: A DESIGN PATTERN FOR NETWORKED, DATA-INTENSIVE SCIENCE
Review round: 1
Reviewer: 1

Output:
Basic reporting: I note a few minor problems that, if fixed, would help with readability.

•	The term “science DMZ” is introduced at line 49 but not explained until line 145.  It would be helpful to insert a footnote to the first occurrence to replace the latter explanation.
•	Similarly, Fig. 1 references perfSONAR but perfSONAR is not explained until line 158, and there only refers to Fig. 2.
•	I believe the code snippet following line 381 is missing a closing parenthesis
•	The text gets a bit repetitive in the explanations of authorization
•	In Table 1, the units for Rate are given as mbps.  It should at least be Mbps, or perhaps MBps – it is not clear.
Experimental design: The paper is not reporting on an experiment, but rather on the design and deployment of a research data portal.
Validity of the findings: This is a very valuable description of a software infrastructure that supports data discovery, transfer, and dissemination.
Additional comments: There is little to complain about in this manuscript from Chard et al. on the Modern Research Data Portal.  The paper is well-written and timely, representing further fine work by the well-regarded team behind Globus.  It is also commendable that the authors have made the code for their reference implementation freely available.

I find it hard to understand the transfer rate data shown in Fig. 8.  These distributions show transfer rates that differ by six orders of magnitude!  And as slow as ~100 bytes/sec, which is more like old acoustic-coupled modem transfer rates.  I understand the upper envelope, in that you cannot exceed a transfer rate whose numerator is larger than the total amount of data transferred.  But I do not understand the lower envelope unless there is some cut-off or time-out being applied such that transfers that take too long are aborted (and not counted).  Also, it might make more sense to label the x-axis in powers of ten incremented by 3, e.g., 10^0, 10^3, 10^6, so that they map more obviously onto KB, MB, GB, etc..  Lastly, for Fig. 8a it is remarked that there is only one incoming transfer, but it is not at all visible in the plot.

Given that several of the authors are active in the National Data Services Consortium, I was disappointed that NDS is not mentioned, e.g., as a deployment platform for the reference implementation.  One could imagine community engagement through customization of the reference implementation and sharing of those customizations through the NDS Labs Workbench.


Instruction: You are one of the reviewers, your task is to write a review for the article. You will be given the title of the article, the number of the round in which the article is located, and your order among the reviewers.

Input:
Title: THE MODERN RESEARCH DATA PORTAL: A DESIGN PATTERN FOR NETWORKED, DATA-INTENSIVE SCIENCE
Review round: 1
Reviewer: 2

Output:
Basic reporting: The article is well written in clear, professional language with a good introduction and relevant set of references.

One comment on Figure 1 is the lack of explanation of the perfSONAR network monitoring devices. These are explained in the MRDP Figure 2 but are either unnecessary for the LRDP diagram in Figure 1 - they would surely not have been present in most legacy implementations – or should be explained.
Experimental design: The research issues are well explained and very relevant to the management and movement of the increasingly large amounts of experimental data being generated in large-scale facilities.

The code examples are well explained and relevant and the physical exemplars are interesting and well-chosen.
Validity of the findings: The evaluation of MRDP adoption is brief but very interesting.

The only example I would question is the inclusion of the Cornell Advanced Computing (CAC) archive. This was not described as an example of an MRDP design pattern and seems to be a different case. Does the Cornell high-speed internal network separate data transfer from general network traffic? If so, that would be very interesting but this was not made clear.
Additional comments: The paper is clearly written with interesting and relevant examples. There is now a real need for something like the MRDP design pattern and this paper shows how the necessary elements for authentication and authorization, data file management and fast data movement can be achieved with Science DMZs and a service like Globus Online.

I have made a couple of comments for the authors to consider clarifying but these should not prevent publication of this useful and important paper.


Instruction: You are one of the reviewers, your task is to write a review for the article. You will be given the title of the article, the number of the round in which the article is located, and your order among the reviewers.

Input:
Title: THE MODERN RESEARCH DATA PORTAL: A DESIGN PATTERN FOR NETWORKED, DATA-INTENSIVE SCIENCE
Review round: 1
Reviewer: 3

Output:
Basic reporting: Chard et al present a distributed systems software engineering pattern for building data intensive science portals.

This work presented is not necessarily new. However, this is the culmination of much work in this space by the Globus team, and other groups developing software infrastructure for global science. This publication to formalise this pattern, present real world examples, and demonstrate example code is very worthwhile and and very much worthy of publication as a complete whole.

The publication is well written and provides both a code walkthrough, example data portal and Jupyter notebook example.
Experimental design: The authors present a model that formalises the intersection of the science DMZ, high performance file transfer services, HTTP portals. The illustrate the success of this model by demonstrating successful implementations and providing statistics about their uptake, and allow authors to test and reuse by providing code and examples. I consider the experimental design and demonstration extensive for this type of publication.
Validity of the findings: The case made by the authors is valid and demonstrated.

I would suggest that using the terms “legacy research data portal” and “modern research data portal” is loaded and the authors may want to reconsider this naming.
Additional comments: No additional comments


Instruction: You are one of the reviewers, your task is to write a review for the article. You will be given the title of the article, the number of the round in which the article is located, and your order among the reviewers.

Input:
Title: A REDESIGN OF OGC SYMBOLOGY ENCODING STANDARD FOR SHARING CARTOGRAPHY
Review round: 1
Reviewer: 1

Output:
Basic reporting: The paper needs a native English language revision to make it clearer.
It is well organized and this helps understanding the main contents.

The cited literature references related with OGC standards and portrayal interoperability of such standards  are relevant.
Figures and schema help understanding the main contents and well exemplify the potentiality of the proposal.
Experimental design: The faced problem to allow on the user side of an SDI personalizing the style of visualized maps according to user cartographic preferences is original and relevant.
Also the proposal to enable such a feature by fostering the compliance with OGC standards for interoperability of SDIs as far as portrayal interoperability is a correct way to approach the problem.
Finally the implementation architecture of the proposal within the OrbisGIS is well schematized and the example shown provide an idea of the potentiality of the approach.
Validity of the findings: The validity of the proposal for personalization of SDI and the novelty of the results could be exploited for designing geoportals customized to user categories.
Additional comments: The faced problem to allow on the user side of an SDI personalizing the style of visualized maps according to user cartographic preferences is original and relevant.
Also the proposal to enable such a feature by fostering the compliance with OGC standards for interoperability of SDIs as far as portrayal interoperability is a correct way to approach the problem.
The proposal should make clearer the difference between choice of the styles of a thematic maps, with respect to choice of the  graphic representation of a thematic map, for example using coremi.
Finally the authors needs to explicit how far the proposal can go in the personalization of the cartographic style, for example if anamorphic maps could be a possible choice of the style of a map.

The paper needs a native English speaker revision.


Instruction: You are one of the reviewers, your task is to write a review for the article. You will be given the title of the article, the number of the round in which the article is located, and your order among the reviewers.

Input:
Title: A REDESIGN OF OGC SYMBOLOGY ENCODING STANDARD FOR SHARING CARTOGRAPHY
Review round: 1
Reviewer: 2

Output:
Basic reporting: In the introduction, your discussion on support of visualization in current SDIs is a good point, and relevant is the consideration of INSPIRE, but your conclusion that portrayal is considered a concern of second zone could sound too strong, given for instance the “data specification on geology theme” (INSPIRE D2.8.11.4_v3.0 - http://inspire.ec.europa.eu/id/document/tg/ge), where mature styling practices from the geological community are exhaustively considered and included in the guideline document (pages 106-130).
Figures are well designed and of great support to the reader. I have just some points about Figure 7 and related text (“use case catalog” paragraph, lines 230-242). In the figure you depict the “catalog” and the “map service” as two separate components, and I agree that this is a good interpretation of functionalities, but what you describe in the text concerns a single endpoint, the actual WMS implementation (“the storage point to discover, …” line 239), and this is a bit confusing with respect to the figure, where the arrow from “my customized version of style2” and the “map service” is hiding the put operation which the reader would expect towards the style catalog, while now he could interpret the current arrow having the same meaning of corresponding arrow in Figure 5 (getMap request with posted user style). Hoping to catch the overall idea, I would suggest you to solve this issue stressing the separation of functional modules. If you agree, you could slightly modify both the figure and the text: 1. in figure 7 I would add a second arrow from “my customized version of style2” directed towards “style catalog”. 2. the text could suggest the following lifecycle of the style: the user gets style2 from catalog, then he makes modifications to the style with his software (style editor) and sends a getMap request to WMS with the the customized style, repeating modifications to the style till satisfactory results are obtained (that is: as you suggested in the use case, the user does not want to download data, so he uses WMS capabilities of interpreting style), and at that point the style is put into the catalog.
A second very minor suggestion for the same figure is then to add the label “style editor” just like in Figure 5.
Minor issues
Line 265: something is missing, at least an hyphen, but also “mapServer” within brackets has an unclear meaning: maybe a missing reference? Please check.
Line 426: please check the question mark at the end of the line.
Line 480: a closing bracket is missing.
Experimental design: I have some points regarding the very interesting discussion of the OGC SE and related standards.
You well clarify some weaknesses of the OGC standard, but the sentence at line 269 (“ambigous explanation in the specification”) could be better supported at least by one example, that I would not assume to be the possibility to obtain the same result by two different functionalities, on the contrary I would expect an example of two different outcomes driven by the same usage of the standard.
Test suite: a  very important aspect is highlighted saying that a test suite is missing – lines 271, 272 -, but I would ask to better clarify if and how the current study could contribute to this point, that the same standard admits to be critical considering that the “Conformance verification of the output of a system implementing SE requires in most cases visual interpretation” (OGC SE, 05-077r4, Annex A).
Even if the request from different communities of alternative encodings to OGC standards is a fact (not confined to the cartographic community, see for instance, about SWE, the discussion about JSON encoding at http://www.ogcnetwork.net/node/1646), I would ask you some reference to support your comments in this direction at line 316, that could serve to better identify the actual gaps and be of interest to the reader.
Validity of the findings: In section “New symbolization capabilities”, the reader could benefit by some indication of the gap between the actual implementation of SE in contrast to the new symbolization capabilities you are discussing. For example Unit Of Measure suggested by OGC are confined to “metre”, “foot”, “pixel”; Transformations are actually (if I’m not wrong) just Displacement (=Translation?), Rotation and Scale, while other affine transformations are missing; etc.
Additional comments: No additional comments


Instruction: You are one of the reviewers, your task is to write a review for the article. You will be given the title of the article, the number of the round in which the article is located, and your order among the reviewers.

Input:
Title: SUSTAINABLE COMPUTATIONAL SCIENCE: THE RESCIENCE INITIATIVE
Review round: 1
Reviewer: 1

Output:
Basic reporting: See general comments.
Experimental design: See general comments.
Validity of the findings: See general comments.
Additional comments: This article provides an informative discussion of the ReScience journal and its motivations. Created in 2015, ReScience publishes replication attempts of previous computational analyses. The initiative is noteworthy as a project that aims to improve the replicability of science and produce open source implementations of past closed science. In addition, the initiative was innovative in adopting GitHub as the submission, review, and publication platform. This design leads to unprecedented transparency in the review process and presumably helps encourages productive and actionable exchanges between reviewers and authors.

I do take issue with some aspects of ReScience's design. However, given that this article is a retrospective examination, modifying the past design is not feasible. Therefore, some of my comments should be considered advice for the future or discussion points to consider, rather than obligatory changes.

The following two statements are worrisome in that they seem to take a defeatist attitude towards computational reproducibility:

> In particular, if both authors and reviewers have essential libraries of their community installed on their computers, they may not notice that these libraries are actually dependencies of the submitted code. While solutions to this problem evidently exist (ReScience could, for example, request that authors make their software work on a standard computational environment supplied in the form of a virtual machine), they represent an additional effort to authors and therefore discourage them from submitting replication work to ReScience.

> the code newly produced for ReScience will likely cease to be functional at some point in the future. Therefore, the long-term value of a ReScience publication is not the actual code but the accompanying article. The combination of the original article and the replication article provide a complete and consistent description of the original work, as evidenced by the fact that replication was possible.

It is highly problematic if ReScience replications are not reproducible (i.e. they cannot be rerun by other researchers in the future). First, enhancing a replication will become difficult. Software and data analyses are never fully complete. Contributions to a replication may continue indefinitely. Furthermore, the replications will often contain useful open source implementations that are adaptable to new problems. Reproducibility will make repurposing the code more feasible. Finally, the value of the initiative is diminished if independent parties cannot verify a replication. Therefore I think it's a bit prescriptive and shortsighted to consider software reproducibility as a secondary concern.

While there is not a one-size-fits-all solution to reproducible computational analyses, there are many approaches that help remedy the situation. For example, the shelf life for software that makes no attempt to control its environment is often only a few months. However, even just specifying the versions of the explicit dependencies, say as a conda environment, could increase shelf live to several years. Alternatively, containerizing the entire computational environment, e.g. via Docker, could increase the shelf life to decades. I believe that ReScience should strongly encourage controlled environments and dependency versioning. For example, as a code reviewer, I would in most cases refuse to approve a data analysis codebase without versioned inputs and software.

On a related point, the study does not cite continuous analysis (https://doi.org/10.1038/nbt.3780). Continuous Analysis (controlled environments with automated execution via continuous integration) would potentially be very helpful for ReScience repositories. It would help ensure the reproducibility of analyses and traceability of the results. Continuous Analysis can be quite complex to configure, so this is mostly a suggested reference and consideration going forward.

> It offers unlimited free use to Open Source projects, defined as projects whose contents are accessible to everyone.

When discussing the pricing model of GitHub, I recommend writing "It offers unlimited free use to public projects" rather than stating "to Open Source". It's important not to further the misconception that public repositories without an open license are open source. Currently, GitHub's [pricing page](https://github.com/pricing) states:

> GitHub is free to use for public and open source projects. Work together across unlimited private repositories with a paid plan.

Can ReScience implementations use code from the original study or are authors required to entirely rewrite the code to encourage true replicability?

The study is lacking a data-driven summary of ReScience's past operations. I'd recommend tables and/or figures that express ReSceince's history up to this point. For example, what about a table with all ReScience articles thus far? The table could show the date and info of the original study as well as link to the ReScience replication. Additionally, some quantitative measure of how much total review (in the form of GitHub comments) has occurred would be informative. What's the distribution of codebase sizes, language choices, study lifespans, etcetera, across ReScience articles? In short, this article could do a better job providing quantitative measures of ReScience's historical growth and mechanics.

To close, my favorite quote from the study is:

> The holy grail of computational science is therefore a reproducible replication of reproducible original work.


Instruction: You are one of the reviewers, your task is to write a review for the article. You will be given the title of the article, the number of the round in which the article is located, and your order among the reviewers.

Input:
Title: SUSTAINABLE COMPUTATIONAL SCIENCE: THE RESCIENCE INITIATIVE
Review round: 1
Reviewer: 2

Output:
Basic reporting: The abstract names a certain  'James Buckheit'. It should be rather 'Jonathan Buckheit'.

This is a superbly lucid article.

It describes a project that is exemplary and inspiring.

Because of the unusual nature of the article -- it describes a new scientific publishing effort -- the usual PeerJ reviewer checklist is not appropriate.

I have not published at ReScience (But now I intend to!) and so I cannot vouch for the veracity of any of the statements about the procedures. The description is very clear -- I wasn't even able to spot typos, except for the issue mentioned in line one of this review.
Experimental design: The intellectual principles of science are modeled in an exemplary way by this paper.
So as a scientific project this is first-rate. It is not however relevant to judge this by the PeerJ experiment design checklist.
Validity of the findings: This paper is necessarily a list of experiences setting up a new scientific publication process.

I am not able to judge the literal truth of the claims about the journal and its operations, however, the description makes me believe it all.

Perhaps someone among the reviewers actually HAS used ReScience and CAN say that things actually work this way.
Additional comments: BRAVI! HOORAY!


Instruction: You are one of the reviewers, your task is to write a review for the article. You will be given the title of the article, the number of the round in which the article is located, and your order among the reviewers.

Input:
Title: COST-EFFICIENT ENACTMENT OF STREAM PROCESSING TOPOLOGIES
Review round: 1
Reviewer: 1

Output:
Basic reporting: The authors present an stream processing engine that uses docker containers on top of VM's. Docker containers promise to provide more flexible deployment of stream operators under varying data arrival scenarios.

A deployment algorithm has to optimize resource usage while taking SLA requirements into account. The authors present a heuristic to this NP-hard optimization problem. The authors evaluate their novel BTU-based optimization approach with a more naive threshold-based approach. They can show that their new method is more cost efficient.
Experimental design: For the evaluation the authors simulate input data (i.e. sensor data) using tools from their VISP Testbed. Their stream processing engine is runs on a private OpenStack deployment. For the evaluation the authors compare their topology enactment algorithm that optimizes Billing Time Units with a simple algorithm that uses thresholds on the incoming queues to determine if VM's should be added or removed to the tolopogy. The experiments are performed with different arrival patterns and variable data volume levels. For comparison the the authors compute various metrics, including paid Billing Time Units and various Quality-of-Service metrics. The author convincingly performed the evaluation on a near real world example (real-world example but simulated data) using variable data load scenarios. The evaluation metrics are reasonable and provide insights into the behavior of their algorithm.
Validity of the findings: no comment
Additional comments: This is a well written paper!

I miss some discussion regarding I/O. How would a high IO to compute ratio impact the deployment strategy?

It is not clear to me if the software the authors developed is available for use, ideally as open source.


Instruction: You are one of the reviewers, your task is to write a review for the article. You will be given the title of the article, the number of the round in which the article is located, and your order among the reviewers.

Input:
Title: COST-EFFICIENT ENACTMENT OF STREAM PROCESSING TOPOLOGIES
Review round: 1
Reviewer: 2

Output:
Basic reporting: * the paper is clearly structured and easy to read
* sufficient background is presented and the studied related work meets the expected standards
* most figures are easy to read, except Figure 8, which is impossible to read on b/w printout and Figure 9 and 10, which are totally overloaded
Experimental design: * the paper studies a very interesting topic (elastic scheduling considering the monetary cost), which is novel and not studied in detail in the existing related work

* the methods are clearly described and easy to follow

Negative points:
* I see a clear gap between the title of the paper, its abstract, and content:
** the abstract and the paper itself are focusing on different goals. While the abstract speaks about "maximizing the resource usage", the paper tries to minimize the costs, although the authors point out that these goals are not necessarily exchangable
** the title contains "utilizing container technologies", I don't see how a container technology helps to solve the problem described and what is the difference to the use of a thread / process per operator like previous solutions do;
   for me the container technology is an implementation detail and not relevant for the contribution of the paper

* I don't understand the described approach completely and have multiple questions:
** how do the algorithms Algorithm 1,2, and 3 contribute to the goal of minimizing the number of reconfigurations and the overall cost? I understand that these algorithms choose the optimal parallelization degree per operator,
   but not how they influence the cost or minimize the number of reconfigurations. The authors should describe this more clearly.
** the core contribution for me is the approach illustrated in Figure 5, but this figure is only described very shortly and informal. Moreover, the approach provided in this Figure answers the problem only partially:
   E.g. shouldn't you undo all changes if you can not release the host in order to minimize the number of reconfigurations?

* All described algorithms have "magic" values including "scalingThreshold" (Algorithm 1), CF (Algorithm 2), QL, W1, W2, W3, and W4, which are not discussed or evaluated in detail.

Based on the points raised above, I cannot judge if the approach above really solves the described challenge or if the measured cost reduction is only a positive "side effect".
Validity of the findings: * The used workload pattern are too simplistic for a meaningful evaluation and the authors themself raise some concerns regarding practical applicability (line 658ff).
Additional comments: Minor points:
* line 67/68: I would recommend to add Apache YARN and Kubernetes to this list: YARN is the most important cluster manager used in practice today and Kubernetes gets more and more importance overall
* line 69ff: I totally disagree to this statement. Mesos and similar frameworks do not decide how much resources are used by the data streaming system. Both rely on a two-level architecture,
  where the framework like Spark, Storm, or Flink decides how much resources is needed. YARN or Mesos only ensures all running frameworks get sufficient resources and fairly share them between multiple users.
  For this task, simple methods like BinPacking work quite well (especially given the intended scalability of Mesos or YARN).
* line 661ff: Also in private cloud environments cost efficiency is important to the user. Typically, the resource costs are cross-charged to different business units of the same company and you need to minimize the cost for your department.
* line 670ff: I disagree that elasticity has not reached established products: Spark Streaming has Dynamic Allocation (https://issues.apache.org/jira/browse/SPARK-12133),
  Flink supports elastic scaling of stateless flows, Heron recently published a paper on elastic scalability on VLDB2017, and also some of the research about System S reached the final product.
* your related work does not reflect any works on cost efficiency of data management solutions, you can consider referencing one of the following:
** Florescu et al. "Rethinking Cost and Performance of Database Systems.", SIGMOD Records 2009.
** Kossmann et al. "An Evaluation of Alternative Architectures for Transaction Processing in the Cloud.", SIGMOD 2010.
** Lim et al. "How to fit when no one size fits.", CIDR 2013.
** Kllapi et al. "Schedule Optimization for Data Processing Flows on the Cloud.", SIGMOD 2011.
** Setty et al. "Cost-Effective Resource Allocation for Deploying Pub/Sub on Cloud.", ICDCS 2014.


Instruction: You are one of the reviewers, your task is to write a review for the article. You will be given the title of the article, the number of the round in which the article is located, and your order among the reviewers.

Input:
Title: COST-EFFICIENT ENACTMENT OF STREAM PROCESSING TOPOLOGIES
Review round: 2
Reviewer: 1

Output:
Basic reporting: * the paper is clearly structured and easy to read
* sufficient background is presented and the studied related work meets the expected standards
* all figures and tables are easy to read and understand
* the presented level of detail about related work meets the expected quality
Experimental design: * the paper studies a very interesting topic (elastic scheduling considering the monetary cost), which is novel and not studied in detail in the existing related work
* the methods are clearly described and easy to follow
* all my previous comments regarding the description of the approach, its parameters and the title of the work have been addressed
Validity of the findings: * the evaluation scenarios now meet the state of the art and allow to judge the quality of the proposed solution
Additional comments: I think the paper has improved significantly since the earlier version I reviewed and can now be published. A few things the authors could consider for the final version of the paper:

* I personally would prefer to use another kind of measure for the QoS, which is more familiar to the generic reader. You could replace your compliance measure by different latency percentiles e.g. the 99th, 98th and 90th percentile of the end to end latency
* you motivate the used BTU interval 60 minutes with the fact that AWS uses the same interval. Unfourtnately, AWS changed its price model during your paper was in review [1]. You maybe should consider to remove this statement from your paper.

[1] https://aws.amazon.com/blogs/aws/new-per-second-billing-for-ec2-instances-and-ebs-volumes/


Instruction: You are one of the reviewers, your task is to write a review for the article. You will be given the title of the article, the number of the round in which the article is located, and your order among the reviewers.

Input:
Title: A GPU-BASED SOLUTION FOR FAST CALCULATION OF THE BETWEENNESS CENTRALITY IN LARGE WEIGHTED NETWORKS
Review round: 1
Reviewer: 1

Output:
Basic reporting: * In many places, authors cite some work without explaining what’s actually done in that work -- it'd be quite hard for a reader without a background on BC. For example, line 221 says “In this paper, we apply the novel warp-centric method (Hong et al., 2011), which allocates a warp rather than a thread to one node”. Could you please elaborate on that? Another example is line 208: “McLaughlin et al. figured out an
excellent work-efficient technique to solve this problem (McLaughlin and Bader, 2014)”. The following few sentences are not enough to present that work. I strongly recommend having a separate “related work” section before the “GPU-based algorithm” section that summarizes those previous works in detail. For each work that you summarize, please stress the differences and similarities of your work.

* Pseuducodes should help understand the algorithm. No need to give every single detail (like Alg. 1) to show the mechanics. Please shorten those. Using a different tex package that does not show “end if” and “end for”s would also be great (indentations do the same thing).

* Algorithm section basically tells the mechanics of the proposed solution without any high-level explanation or intuition. Please provide a high-level explanation of your algorithm first, then give the details.

* Background is not detailed enough, please give the pseudocode for Dijkstra’s alg.

* ln 149: explain coarse-grain parallelization, which is also used in exps (ln 245).

* So many grammar mistakes (esp articles), some are below. Multiple proofreadings needed.
- ln 23: publicly
- ln 38: While Brandes .. -> Brandes ..
- ln 44: millions of active
- ln 68: characteristics
- ln 93: Brandes
- ln 99: how to adapt for directed?
- ln 102: the minimum length of all paths connecting s and t -> the length of the shortest path connecting s and t
- ln 104: an undirected graph
- ln 113: for an unweighted .. for a weighted ..
- ln 118: an unweighted
- ln 120: tree is
- ln 126: the shortest path problems. Dijsktra algorithm
- ln 129: s to an

* Please format the last two lines of Tab. 2 to highlight that they're speedups (not runtimes).

* Replacing Fig 4 by a table (like Tab. 2) might be better. The current indirection for the algorithm variants (A, B, C ..) makes it hard to understand.
Experimental design: Regarding the motivation:

* ln 17: Not sure if the weighted networks are more prevalent than the unweighted ones. Yes, there exist attributes on the edges in many real-world data, but capturing those with the weights is a hard problem and not always possible. Please reword that.

* ln 49: It cannot be said that GPGPU computing provides excellent parallelization for graph/network algorithms. It is always nontrivial to get good performances for the irregular computations (that's why this work is interesting).

For the experiments:

* ER and Kronecker graphs are not realistic models and do not have the community structure as in the real-world data, please check "Community Structure and Scale-free Collections of Erdős-Rényi Graphs" by Seshadhri et al. In addition, uniformly assigning random edge weights is not justified -- being used by previous works cannot be a reason.

* The largest network has ~15M edges, that's not big enough to test the possible memory constraints of GPU.

* I wonder if the results would be different for the networks that have integer weights and double weights. No information is given for the used data. I suspect double weighted networks might be cheaper for BC computation since there would be less number of shortest paths between any pair of nodes.
Validity of the findings: - ln 99: how to adapt for directed? Please give more detail or don't say that.

- There is a contribution in this work, but the presentation must be improved significantly and a few issues about the exp design should be addressed.
Additional comments: Ignoring the presentation and the data used in exps, I believe that authors provide a nice combination of the existing techniques (for unweighted BC) and adapt them for the weighted BC computation.


Instruction: You are one of the reviewers, your task is to write a review for the article. You will be given the title of the article, the number of the round in which the article is located, and your order among the reviewers.

Input:
Title: A GPU-BASED SOLUTION FOR FAST CALCULATION OF THE BETWEENNESS CENTRALITY IN LARGE WEIGHTED NETWORKS
Review round: 1
Reviewer: 2

Output:
Basic reporting: This manuscript has many minor grammatical issues and would greatly benefit from a review by a native English speaking colleague. Here are some examples:

- The abstract contains the phrase "can arrive the performance of." This phrase should be replaced with "attains a" or "achieves a" when mentioning the speedup of the algorithm.
- The use of the word "while" on line 38 is confusing.
- Line 50: I would prefer the term "applications" (or a synonym of it) to "issues"
- Line 52: I would rephrase this sentence as "CUDA, developed by NVIDIA, is the most popular..."
- Line 130: "According to be settled or not" could be rephrased as "According to whether or not a node is settled"

The reference supplied by this article are sufficient, although I would add a reference to the following paper:

E. Solomonik et al., "Scaling betweenness centrality using communication-efficient sparse matrix multiplication."

Also, on Line 75 there is a citation to "Merrill et al., 2015" which does not match any of the references. Perhaps you meant the paper from Jia et al. in 2011?
Experimental design: The research question of this work is well-defined. One particular aspect of the work I would like to see a clearer explanation of is the use of a lock. Locks are particularly problematic in CUDA as they're not only error-prone but also can halt entire warps instead of individual threads as they would on a CPU. Interestingly, CUDA 9.0's capabilities might have a big influence of this choice of algorithm.

In fact, CUDA 9.0 may also come into play when attempting to figure out the appropriate value of WARP_SIZE, as CUDA 9.0 supports cooperative groups. Since you used CUDA 8.0 (which is obviously fine), do you see a way that you could determine the best value of WARP_SIZE dynamically without having a significant performance impact?
Validity of the findings: The depth of experiments and high quality of results are the strongest point of this submission. The conclusions drawn by the authors support those of prior work and appear reasonable.
Additional comments: No additional comments


Instruction: You are one of the reviewers, your task is to write a review for the article. You will be given the title of the article, the number of the round in which the article is located, and your order among the reviewers.

Input:
Title: A GPU-BASED SOLUTION FOR FAST CALCULATION OF THE BETWEENNESS CENTRALITY IN LARGE WEIGHTED NETWORKS
Review round: 1
Reviewer: 3

Output:
Basic reporting: The paper describes an efficient GPU algorithm for computing betweenness centrality on large-scale weighted graph, and shows performance advantage via extensive experiments and performance analysis. Overall, the idea is clearly presented with sufficient literature references and background/context. That said, the English expression of a lot of the details of algorithm description and performance analysis needs to be improved. Most figures and tables are self-explanatory, one general suggestion is that the author should consider putting relevant figure/table at the same page as the texts.
Experimental design: The research question is well-defined and the open-sourced implementation will greatly help the community to reproduce and improve on this work.

On the experimental design, I have two high-level comments:
1) Comparing to other betweenness centrality papers, especially [McLaughlin 2014], the datasets in this paper is not large enough. I suggest the author to include at least one graph with more than 1 million nodes for each synthesis dataset and real-world dataset.

2) The current results analysis is not organized well enough. Could the authors restructure the section into something similar to "overall performance", "analysis on different traversal strategies", "analysis on different datasets"? (Not necessarily the one I suggested, but just structure the section in this way will help audience understand the experiments better)

3) For the sequential and multi-thread CPU implementations, are they the state-of-the-art implementation? Please add more description and/or citation information of these two algorithms.

Also, on page 10/14:308, the authors claim that R-Mat and real-life scale-free graphs have similar topology. I have different opinion on that, since R-Mat graphs are more scale-free than real-world graphs, real-world scale-free graphs usually have lower edge factor. The authors should consider adding analysis according to different edge factors.
Validity of the findings: The main problem is that the paper's method is not innovative enough as both the graph traversal part and workload balancing strategy part is based on previous methods. But I think the topic is worth exploring, since it is true that no work has been done on weighted graph betweenness centrality on the GPU.

The main things this paper could improve are as follows:

1) It does not try another method which simultaneously does graph traversal and can potentially speedup betweenness centrality: 10.1109/ICPADS.2015.10
If the authors think there is technical reason this method cannot be applied, they should include this in the literature review and specify the reason it is not applied in this paper.

2) For the load-balancing strategy, the authors tried Merrill's method (referred to as work-efficient method) and warp-centric method, but they miss the other strategy which were used in both [Davidson 2014] and [Wang et al. 2016]'s work. The merge-path based load-balanced search has better performance on scale-free graphs on all traversal-based graph algorithms, which I believe includes betweenness centrality too.

3) One minor pick is that I think with small modification, delta-stepping could also level information and number of shortest paths information. In that case, using delta-stepping with a smaller number of buckets could also potentially increase the performance. The authors should provide more details on the reason of not using delta-stepping method.
Additional comments: I put several grammar comments and minor technical comments in the attached pdf file.


Instruction: You are one of the reviewers, your task is to write a review for the article. You will be given the title of the article, the number of the round in which the article is located, and your order among the reviewers.

Input:
Title: A GPU-BASED SOLUTION FOR FAST CALCULATION OF THE BETWEENNESS CENTRALITY IN LARGE WEIGHTED NETWORKS
Review round: 2
Reviewer: 1

Output:
Basic reporting: No comment.
Experimental design: No comment.
Validity of the findings: No comment.
Additional comments: I'm satisfied with the revision. Organization and presentation are well done, and the paper is comprehensive now for a general CS audience.


Instruction: You are one of the reviewers, your task is to write a review for the article. You will be given the title of the article, the number of the round in which the article is located, and your order among the reviewers.

Input:
Title: A GPU-BASED SOLUTION FOR FAST CALCULATION OF THE BETWEENNESS CENTRALITY IN LARGE WEIGHTED NETWORKS
Review round: 2
Reviewer: 2

Output:
Basic reporting: The revised manuscript is better structured and has greatly benefited from the review of native English speaker(s). The authors took care to cite additional references and provide additional background based on the first set of reviews.
Experimental design: no comment
Validity of the findings: no comment
Additional comments: Overall I thought the authors did a fantastic job of restructuring and cleaning up this manuscript. It now provides a thorough understanding of the complications of computing between centrality for weighted networks, with a nice history of recent work in this area. I feel that this manuscript is now ready for publication.


Instruction: You are one of the reviewers, your task is to write a review for the article. You will be given the title of the article, the number of the round in which the article is located, and your order among the reviewers.

Input:
Title: A GPU-BASED SOLUTION FOR FAST CALCULATION OF THE BETWEENNESS CENTRALITY IN LARGE WEIGHTED NETWORKS
Review round: 2
Reviewer: 3

Output:
Basic reporting: The author has made significant improvement on writing and the structure of the paper. It added more citations, explanations to previous unclear descriptions, as well as more experiments and data sets.
Experimental design: Previously my main concerns were: 1) not comparing with the SOTA GPU weighted-BC algorithm; and 2) not having large graph data sets.
In the revised version, the author proved that there is no weighted-BC implementation on the GPU, and it is difficult to modify any unweighted-BC implementation. Instead, work in this paper could serve as a good baseline weighted-BC algorithm.
The revised paper also included two large graphs (one synthesized, one real). However, no CPU performance is presented, which is strange since usually size of graph should not be a limiting factor for CPU algorithms. Author should add some explanation to this.

In general, I really like the re-structured experiment section.
Validity of the findings: I think with the revision and the open-sourced implementation, the paper could serve as a baseline for weighted-BC algorithm on the GPU.
Additional comments: Thanks for the revision and carefully language improvements. The revised version looks much better.


Instruction: You are one of the reviewers, your task is to write a review for the article. You will be given the title of the article, the number of the round in which the article is located, and your order among the reviewers.

Input:
Title: SOLVING THE INVERSE HEAT CONDUCTION PROBLEM USING NVLINK CAPABLE POWER ARCHITECTURE
Review round: 1
Reviewer: 1

Output:
Basic reporting: The authors have presented a solution to inverse heat conduction problem, IHCP, for cylindrical geometry with the help of GPU and CUDA.  They attain a tremendous speed-up of 120 over with 4 GPUs and 20 cores.
Although the authors call it a 2-D problem, in essence it is  a 3-D problem with axial symmetry.
The direct heat conduction problem, whose temperatures are used to estimate heat transfer coefficients could be more completely described for understanding rather than incomplete description ( boundary values at z=0, z=L , missing) given in the paper.

The authors are, however,  commended for achieving such a high speed-up with just 4 GPUs, under CUDA environment and for achieving high parallelism to solve this problem.

The paper is well written, graphs are clear, the background has been sufficiently provided.
Experimental design: The research problem, IHCP, although well defined, could be described mathematically correctly by stating the region of equation and boundary conditions as mentioned  in another section of the review.

The massive parallelism achieved through shared memory architecture is well described.

Bench marking is very well done with various GPUs and  a CPU and is described as well as depicted through graphs and text in detail.

It would be better if authors could provide more explanations on the experimental setup and data results. For example, the configurations of the hardware and the details on how the simulations are run.
Validity of the findings: Even the  DHCP are are not much attempted in the literature,  with massively parallelism achieved,   through GPUs and CUDA environment, the authors attempting an inverse heat conduction problem with GPU and CUDA is pioneering.

Conclusion are well stated and clear,  and amazing results are obtained with the usage of small number (4) of GPUs with larger of cores size (20).
Additional comments: The authors have attempted a very useful and pioneering work in this paper.  They have achieved a tremendously high speed-up of 120 with just 4 GPUs under CUDA environment.  The authors are commended this work.
The description of the boundary conditions although clear at the circular boundary r=R, they are not correctly stated at the boundaries z=0, z=L, a region is cited instead of the boundaries.  The authors are recommended to state the region (0<z<L, 0<r<R) for equation number 1 as well in addition to correcting the boundary values at z=0, z=L.


Instruction: You are one of the reviewers, your task is to write a review for the article. You will be given the title of the article, the number of the round in which the article is located, and your order among the reviewers.

Input:
Title: SOLVING THE INVERSE HEAT CONDUCTION PROBLEM USING NVLINK CAPABLE POWER ARCHITECTURE
Review round: 1
Reviewer: 2

Output:
Basic reporting: -To put all measured data of the experiments in the paper is redundant, there should be a good design on how to present data efficiently

- All tables (1,2,3,4 and 5 ) include huge amount of numbers, it is not necessary to put all the population sizes in the paper, some important points of the population size can be picked to show the trend of changing execution time or memory transfer time.
Also what is the difference between showing using 1 or 2 or 3 or 4 GPUs or 5-10-20 cores, it is only for the purpose of showing the trend, you can pick only 2 sizes

-Table 1 and table 3 would be good to see the speed up factor, in table 3 speed up factor comparing CPU vs. GPU is important
Experimental design: -Why  two sets of multi GPUs with two architectural set up  are used? (Geforce, P100) where do you use it in your analysis or does that help to draw any conclusion based on these two architectural configurations? for example you can compare 2 GPUs from one configuration to the another

-The memory Transfer time in Fig 6 and Fig 7 in two different GPU configurations is different, why any explanation or analysis?

-The NVLink experimental part needs more explanation and analysis.
Validity of the findings: no comment
Additional comments: No additional comments


Instruction: You are one of the reviewers, your task is to write a review for the article. You will be given the title of the article, the number of the round in which the article is located, and your order among the reviewers.

Input:
Title: SOLVING THE INVERSE HEAT CONDUCTION PROBLEM USING NVLINK CAPABLE POWER ARCHITECTURE
Review round: 2
Reviewer: 1

Output:
Basic reporting: The paper is well written in clear and concise form. The Problem of Inverse Heat conduction for estimating the heat transfer coefficients, which are many times not accurately measured or available., is very well described mathematically, and solved for a finite solid cylinder.

The good background is provided as to what other attempts have been made to determine the coefficients by other methods and why the author chose this technique to determine or estimate the coefficients through this GPU technology

The results are clearly graphed and discussed.
Experimental design: Experimental design has been clearly stated and implemented.

However algorithm description is missing. It would be helpful to understand the experimental design if the authors could add algorithm description.
Validity of the findings: Results are very well depicted through graphs, and performance comparisons have been made with different core sizes (5, 10, 20) as well as through various number of GPUs (1,2,3,4).  Good results  and high performances have been achieved (120 Times).
Additional comments: Well written paper.  Tackling of inverse heat conduction to estimate various heat transfer coefficients is difficult but adeptly solved.  The authors deserve credits for handling a time consuming and difficult problem through GPU technology.  The three dimensional problem is described as  2-dimensional problem through  axial cylindrical geometry.  It is  remarkably handled, graphed, comparative performance  depicted through various GPUs/CPUs.


Instruction: You are one of the reviewers, your task is to write a review for the article. You will be given the title of the article, the number of the round in which the article is located, and your order among the reviewers.

Input:
Title: SOLVING THE INVERSE HEAT CONDUCTION PROBLEM USING NVLINK CAPABLE POWER ARCHITECTURE
Review round: 2
Reviewer: 2

Output:
Basic reporting: Comparing to last version the representation of the experimental data improved and less redundant experimental data is in the paper. Also speed up comparison is added.
Please edit Table 1 caption in the manuscript the last sentence should be :
 the last column shows the speed up total time.
Experimental design: It is good that more detailed mathematical background for the problem is added to paper, this helps to understand the inverse heat conduction problem better.
Data transfer rate section is convincingly reconstructed and explanation added that improves the result discussion on that part.
Validity of the findings: Paper shows promising speed-up results using multi-GPU in inverse heat conduction.
Additional comments: No additional comments


Instruction: You are one of the reviewers, your task is to write a review for the article. You will be given the title of the article, the number of the round in which the article is located, and your order among the reviewers.

Input:
Title: INTEGRATING USER EXPERIENCE PRACTICES INTO SOFTWARE DEVELOPMENT PROCESSES: IMPLICATIONS OF THE UX CHARACTERISTICS
Review round: 1
Reviewer: 1

Output:
Basic reporting: Clear, unambiguous, professional English language used throughout.
The basic reporting is in professional English

Intro & background to show context
Too much is context in the form of prior research (literature) . The Introduction gives background literature; this contextual information is also interwoven throughout the Results where the main information should be the response data and code results, and the Discussion is largely more literature review and citations rather than the authors emphasizing the meaning of the data as presented in the Results. Too little context is given about the respondents, their roles and contributions to products, their interactions across “disciplinary” lines in their companies.
,
It would help if the authors explicate from the start what they see as the differences – in discussing results – between functional requirements, quality requirements, usability, and experiential requirements (that fall outside the scope of the prior terms). Is emotion/affect the same for all types of software. Is perceived waiting time an emotion? Does it differ for different software? User tasks and task-related expectations? Also are features and quality mutually exclusive as the authors seem to imply in the body of the manuscript (speaking of features in a way that tacitly disparages them on the sub-text level).

Structure conforms to PeerJ standard, discipline norm, or improved for clarity.
The main section headings conform to standards. But the material within the sections diverges from what readers usually expect (e.g. a lot of literature review/citations woven into Results). The Intro lit review is more a laundry list of studies that may have something to do with the topic – but it is not organized and presented as supporting studies for key arguments that a reader will later see emerge from the interview findings. The authors’ arguments are buried in regard to points unique to their findings. Overall, the piece comes across as an extensive literature review into which data from the study are fit. It should be other way around. The numbering for subsections is not standard and should be changed. Sub-section numbering should begin anew within each major section.

Figures are relevant, high quality, well labelled & described. Not applicable.

Raw data supplied. Good appendices.
Experimental design: Original primary research within Scope of the journal.
Very good research topic and good interview questions. They are original and within the scope of the journal.

Research question well defined, relevant & meaningful. It is stated how research fills an identified knowledge gap.

The research problem and questions are meaningful, and they relate to an important gap that has to be addressed. The aspects of the manuscript that need to be revised do not lie its focus on a problem in the field but in its design, execution and presentation of data analysis .Unfortunately, upon finishing the article, I did not have a clear idea of UX challenges that are unique from other quality issues in software development and evaluation – the authors’ stated purpose in the article. For example, which software quality issues are measurable and how did software specialists arrive at the metrics? What UX issues are different so that they defy the same approach to measurement? What did respondents say that gives evidence for claiming such differences?

From the article readers get a clear idea of challenges related to UX without such a comparison but only at a high level – not the detailed level promised in the Abstract and Intro. Readers would not be able to succinctly state at a detailed level what the most important overlaps are between UX and other software quality issues.

Rigorous investigation performed to a high technical & ethical standard.
Rigorous investigation is not apparent in the analysis of data.

Methods described with sufficient detail & information to replicate.
Methods applicable to qualitative research are presented and justified well. It isn’t unclear how the coding was applied or analyzed. Were there statistical outcomes from the analysis? One advantage of coding is that you can derive statistics and unite them with qualitatively derived insights. It isn’t clear if codes on “demographics” (role, company, type of software produced, years of experience dealing with UX/development) were used and if analysis of coded data accounted for these demographics to find similarities and divergences across demographic traits. Example of passages and how and why they were coded as they were would help. Importantly, no reliability statistics are provided to show cross-coder or intra-coder reliability.

There also isn’t information on methods used (if at all) for integrating coding outcomes with traditional qualitative analysis of the themes that grew out of the data after iterative readings. I found that, as a reader who wasn’t clear on these methods, the challenges that sub-headed the Results portion seemed to be imposed by the authors on the data based on prior knowledge and literature reviews rather than having emerged from the data.

The authors do not organize Results so that readers clearly can see issues or themes that they derived from the data. Because the section is organized around challenges that are already well known and previously studied, a reader is likely to assume that the authors imposed this thematic categorizing on the data rather than probing the data for relationships that would give rise to themes that offer new insight for the research questions – i.e. overlaps and distinctions between UX and other quality issues in software.

I am a researcher who often seeks studies like this one to cite to support my own empirical research, Yet I find myself unable to use the Results for support. There isn’t enough evidence of who perceived what and why. The authors express results about “Practitioners” or Participants” as the agents of responses but that is too vague in terms of show where certain types of respondents agree and diverge. The authors cross-reference their chart of participants (e.g. C1 as the agent) for some of the Results. But the hardship falls on readers to trace back to the label, to make the distinctions and to draw the inferences themselves about similarities and differences.

Finally, the authors talk about results from a qualitative research perspective but I do not see the methods they used for deriving themes qualitatively (to complement/supplement the codes) – e.g. affinity diagrams, post-it clustering. They also talk about challenges to validity that, in fact, are threats to quantitative research. The authors need to address qualitative research threats to validity (namely imposing categories on the data rather than letting the data lead to the themes  and reliability scores in coding (missing in this manuscript).
Validity of the findings: I did not find the distinction between usability and UX clear. Nor was I entirely clear what significant point the authors were trying to make by saying that the two are distinct. I believe they are. But I’d like the authors to answer the “so what?” question. Ultimately they should relate the issue to the relationship between UX and other software quality issues (there are more quality issues than usability; plus they should show how usability – when software engineers adopted it as a quality issue – is often defined by them differently from how usability experts define it. Evidence needs to be drawn from the data, differentiating by respondent role, company and software.

I do not believe that the authors answered the question of how to deal with the questions of measurable requirements for UX – in the bigger picture of  UX requirements being like other quality measures in software so that UX is more integrated into the development cycle.

A key problem is that experiential (subjective) requirements have not yet been measured at all or not to a standard that software teams use. The authors “answer” is to have more measurable UX requirements. That issue has been plaguing us for at least 20 years (see some of the soft goals literature in Requirements Engineering and Aspects computing (cross-cutting goals). I’m not sure if the data from interviews gives insight into details of measures beyond calling for the need for them. I’d suggest seeing what the data do reveal that is insightful and can address and highlight it.

The authors should also try to avoid what I’d call “sleight of hands” in reporting results. In lines 807+ they say “Still practitioners from 6 out of 8 companies stated they often come into projects only in later stages…” Two companies in the study include all technical respondents. The framing of the sentence may lead readers to think that in 25% of the companies this isn’t so – unless they refer back to the chart and see the composition of respondents by company. Similarly consultancies by definition may only contract for coming in at the end, making this phenomenon a function of being a consultancy – or even seeing that user-centeredness should be outsourced rather than integrated into a multi-disciplinary software team.
Additional comments: I do not want to close the door on this manuscript. The data are very rich and, as such, are important to analyze for insight and novelty more rigourously and with more nuances. I believe that the piece needs a major rewrite – going back to the methods for analyzing the data, restructuring the Results to show qualitatively derived themes that grow, and writing a Discussion that show how insights from this study  Overlaps with existing knowledge should be brought in early in the research literature. A small initial sub-section in Results may enumerate what the data from the interviews show that is already known. The rest of the subsections in Results should be insightful themes.

I also suggest that the authors write with more sense of audience/readers. Cut down the number of points made so that readers come away impressed by the 2 or 3 main insights that the data reveal and a strong sense of why they are significant for the larger issue of software quality


Instruction: You are one of the reviewers, your task is to write a review for the article. You will be given the title of the article, the number of the round in which the article is located, and your order among the reviewers.

Input:
Title: INTEGRATING USER EXPERIENCE PRACTICES INTO SOFTWARE DEVELOPMENT PROCESSES: IMPLICATIONS OF THE UX CHARACTERISTICS
Review round: 1
Reviewer: 2

Output:
Basic reporting: The mainly proposes to investigate how user experience is perceived and defined while identifying the main challenges in incorporating UX in the software development life-cycle.  The authors investigated 17 practitioners from 8 companies to identify 11 challenges that these practitioners are facing when trying to incorporate UX in their daily work. and processes.  The authors reviewed the literature on UX and suggested to classify the perception of UX under three different categories, Work on UX, work on UX in Agile and work on UX/Usability.  The 11 challenges are well discussed.
Experimental design: The design of the exploratory and qualitative study that have been conducted is discussed. However various key information are missing:

- How the companies have been selected and why these companies and not others. Why they have different profiles and business focus. UX is really subjective and the perceptions may differ drastically from one company to another one, UX design-oriented firm, service design, mobile and Web companies are really very different, even the profile of their UX designers and developers is sometimes totally different.  Table 1 need to incorporate more data that gives a true and realistic picture of the companies.

-  Again why semi-structured interviews? Such method for interview may bring very divergent data, especially that the goal here was to build a consensus around the 11 challenges! A screening scenario that detail how the interview have been conducted is needed

- The data were analyzed using a inductive and deductive approaches, thematic analysis. Why this method again? Usually, we also consider triangularisation to consolidate data collected from different sources, for example it was good if at each company you tried to understand the point of view of developers, projects managers and seniors staff and why not external stakeholders. Still the data collected was honestly and objectively analyzed.
Validity of the findings: - Validly threats reported are not really very convincing as there was many missing information as presented above.

A major effort need to made on this section, I suggest that the authors look at to the Common Industry Format on how and what we should document in UX/usability studies ISO/IEC 25062:2006(en)
Software engineering — Software product Quality Requirements and Evaluation (SQuaRE) — Common Industry Format (CIF) for usability test reports

The second big concern with the paper is that many of the challenges reported are well known. They have been largely discussed in the scientific literature. Various workshops have discussed how and why UX/Usability should be considered in the llifecycle.
Additional comments: I suggested when revising this paper to discuss your findings while contrasting them with the findings from previous studies and the conclusions of previous workshops.  The topic of integrating usability in the development life cycle have been around since 1998, see CHI workshop on use cases and task modeling. What is the differences and what is new here?  The last section of the paper (discussion) bring some useful information especially for practitioners. What should be done by researchers to overcome these concerns.

Are these challenges totally independent from SE methods such as Agile? Agile is very close to UX somehow, at least they share some common concerns engaging users/consumers.

List of references need to be extended, check recent publications 2015 and 2016 for ACM and IEEE digital library


Instruction: You are one of the reviewers, your task is to write a review for the article. You will be given the title of the article, the number of the round in which the article is located, and your order among the reviewers.

Input:
Title: INTEGRATING USER EXPERIENCE PRACTICES INTO SOFTWARE DEVELOPMENT PROCESSES: IMPLICATIONS OF THE UX CHARACTERISTICS
Review round: 2
Reviewer: 1

Output:
Basic reporting: The paper uses clear, unambiguous and professional english in general
There are further literature references that could be included, but the list is sizeable
The structure is clear and appropriate, with suitable figures. Some raw data appears in the text
The paper does not have hypotheses but does have research questions which are answered with the work presented
The paper does define terms, as used in the paper but part of my unease is because these definitions are not necessarily used more widely, and particularly within the participant group for the study
Experimental design: The experimental design, in this case an interview qualitative study, seems to have been well-designed, and performed in a rigorous fashion as befits a qualitative study.
Validity of the findings: Conclusions are well stated and clear, and are linked to the data. However the background and literature context of the work, together with the interpretation of the findings differs from my understanding and experience of practice and my knowledge of the UX field (more below). This disturbs me but the design and execution of the study itself appears to be sound. The interpretation of the findings makes me uneasy, but this may be due to differing countries (the countries for the companies are not listed in Table 1) and UX cultures.
Additional comments: Based on the title and abstract for this paper I would expect to find it useful and interesting. Indeed I did find some of it useful, which is why I have recommended only minor changes, but your interpretation of the data and the tenor of the discussion does not sit easily with my own experience of software practice and UX. My experience is largely based in UK and US, but with enough experience and understanding of international practice to know that different countries have quite different development cultures, and in particular UX. To support my minor corrections recommendation, I would suggest:

1. that the location (and possibly global distribution?) of the companies be included in Table 1.
2. that more be made of the comparison of UX with other software qualities, as I felt that this was a bit eclipsed by the UX model.
3. that the strong separation you emphasise between usability and user experience be better explained. UX is not just hedonic (pleasurable), and usability is not just about measurement.
4. clarify when you're talking about an agile team and when you're talking about a traditional or safety-critical situation as UX is very different in different application areas and development models
5. Smaller points: clarify Table 2 headings (education level? experience of?)
6. check for typos (there are a few)

The following comments are an attempt to explain my unease around the tone of the paper and its interpretation of practitioner comments. I would be pleased if you're able to respond to these in your writing.

The notion that context and application determine how important is user experience compared to usability, for example, does not feature in the paper. The context-dependent nature of UX is included, because it is one of the characteristics, but the implications of that seem to be missing in the overall tenor of the discussion. User experience is important for all products, but the exact nature of usability and user experience goals, their measurement and their trade-off is context-dependent. I think this is probably what you mean when you say that they are categories of experience, but the treatment seems to imply that it is problematic. Having qualities of software that are difficult to measure quantitatively is not necessarily negative. UX is not an exact science and it requires dealing with people and real world contexts. Compromises and pragmatism are needed. That is the nature of UX. For example “limited access to end users can negatively impact UX measurement” p26 is true and will continue to be true for lots of reasons, but there are pragmatic ways to handle that situation, and good UX practitioners and educators know that and apply those work-arounds.

The paper presents a sound study of UX in practice, with well-reasoned findings. But the picture painted by the findings doesn’t surprise me. For example, in practice, UX, interaction design and usability have all been used in overlapping ways, so the fact that practitioners don’t know what UX is (according to the definition used here) doesn’t surprise me. However it does not mean that practitioners don’t understand the concept underlying UX. The problems of evaluating UX characteristics such as fear, excitement, expectation are well-known, and are being studied in research.

1.	The terms “User experience” “interaction design” and even “usability” have been used in practice in overlapping ways for many years due to a range of reasons. This finding is not a surprise, and indeed if you asked practitioners and academics for a definition of software engineering or agile you will equally get differing views beyond a top level definition agreement.
2.	Evaluating user experience is not easy because you are trying to ‘measure’ how someone ‘feel’, although ‘helpful’ is a user experience characteristic and that’s arguably not hedonic (a term used in the paper in several places to describe user experience)


Instruction: You are one of the reviewers, your task is to write a review for the article. You will be given the title of the article, the number of the round in which the article is located, and your order among the reviewers.

Input:
Title: INTEGRATING USER EXPERIENCE PRACTICES INTO SOFTWARE DEVELOPMENT PROCESSES: IMPLICATIONS OF THE UX CHARACTERISTICS
Review round: 3
Reviewer: 1

Output:
Basic reporting: The paper fulfils these requirements
Experimental design: The paper fulfils these requirements
Validity of the findings: The paper fulfils this set of requirements
Additional comments: Thank you for your revisions, which have improved the paper considerably. The presentation is clearer and the claims made are better justified.

In particular I am pleased to see an explicit acknowledgement that you are focusing on Hassenzahl's model for your analysis. My previous discomfort with your paper stemmed in part from your implicit reliance on this model, which was not clearly acknowledged, as this is only one view.

I have one small suggestion, which is to include the locations of companies G and F when they are first mentioned. This struck me as an odd omission.


Instruction: You are one of the reviewers, your task is to write a review for the article. You will be given the title of the article, the number of the round in which the article is located, and your order among the reviewers.

Input:
Title: THE APPROPRIATION OF GITHUB FOR CURATION
Review round: 1
Reviewer: 1

Output:
Basic reporting: * I'd find it helpful if added some examples of resources that are curated (somewhere around page 1 line 40).
* P2,l53: why are personal needs extrinsic?
* There are several smaller errors in the English, e.g. line 211: "participant" -> participate
* Is it possible that you show an overview of your coding schema at the beginning of the results section?
Experimental design: * why do you only do open coding? Why not the further steps of Grounded Theory?
* The relative counting of the occurrences of codes that you use in the results section should be explained here as well.
* I'm also confused by the claim that you use open coding and then your codes are the categories found be other studies on motivation on OS projects. Wasn't it more a preexisting categorisation?
Validity of the findings: * the high kappa indicates a good agreement and hence generalisability of the codes
Additional comments: - While I found the discussion interesting and well-informed, I think it could be better integrated with the results of the interview. Some parts of it feel like they could have written without the interviews.
- The conclusions are very short. Could you please summarise what we should actually conclude? What do we learn and what effect should/could that have on software engineering practice?


Instruction: You are one of the reviewers, your task is to write a review for the article. You will be given the title of the article, the number of the round in which the article is located, and your order among the reviewers.

Input:
Title: THE APPROPRIATION OF GITHUB FOR CURATION
Review round: 1
Reviewer: 2

Output:
Basic reporting: PROs:
* The paper is generally well written, and discusses an interesting and timely topic which has not been extensively studied in the past.

CONs:
* At various points in the paper, the authors seem to use language with a certain amount of personal bias or over-emphasis, for example in the statement "[...] provides them with a perfect starting point" on page 8 the word "perfect" might be considered as over-emphasizing. This is not a critical comment, but it does make a subtle difference, and it would be advisable to use cautious and hedging language wherever possible. Also, the authors should carefully review whether all of the presented conclusions are actually drawn from and backed by the survey data collected.

Style/language (minor):
* abstract: "motivates software developers to engaged" -> "motivates software developers to engage"
Experimental design: * The methodology of the paper is primarily based on quoting exemplary study responses for different dimensions/categories of the survey. It is not entirely clear how the authors came up with the dimensions that are discussed in the study. Which of the two came first - 1) were the dimensions defined beforehand and the survey results are then used to discuss/confirm these dimensions, or 2) are the dimensions derived directly from the survey results? Generally, it may be advisable to be a bit more precise and specific about the methodology and how the survey has been constructed. Some details are provided in the supplemental material (e.g., chat messages exchanged with the respondents), but including the actually survey questions in the text (or appendix) would help making the paper more self-contained.

* In addition to the qualitative analysis of the free-text study answers, it would be interesting to see a basic quantitative analysis to provide an overview of the numeric results of the study. Due to the way the results are currently summarized in the paper, the presentation seems fairly lengthy and it is hard for the reader to extract the key insights. Some of the discussed results appear straight-forward or at least not very surprising, but if we had some numbers for these individual results, they could become more meaningful (e.g., X% of respondents gave a similar answer, etc).
Validity of the findings: * I generally wonder if any kind of cross-correlation has been performed. For each of the quoted statements the paper lists the survey respondents (P1-P16) who provided the answer, which in itself does not provide a great added benefit (the paper would be equally valid and valuable if the reference to the respondents were entirely left out). An interesting aspect, though, would be to see how the individual respondents provided answers to different combinations of survey questions. Maybe there is a common theme among the answers? Are any of the survey questions in any way related to one another? (e.g., if a respondent provides answer X to question A, they are likely to provide answer Y to question B) It would be interesting to analyze these aspects more systematically.

* In my view, the Discussion section should be more focused on discussing the results and limitations particularly related to the presented contribution (survey). In its current form, the section seems a bit like an extension of the Background section, rather than a specific discussion and critical reflection of the presented results. This is evidenced by the fact that most of the paragraphs/statements in the Discussion are backed by external resources. Using external references is not a problem in itself, but the paper should draw a clear line between discussion of related work / background, and discussion of the results and limitations of the approach itself.
Additional comments: No additional comments