Instruction: You are one of the reviewers, your task is to write a review for the article. You will be given the title of the article, the number of the round in which the article is located, and your order among the reviewers.

Input:
Title: SOFTWARE CITATION PRINCIPLES
Review round: 1
Reviewer: 1

Output:
Basic reporting: This paper argues for the adoption of software citations, basically for the same reasons that data citations have been advocated: Because software packages have become so important in science, they should be appropriately credited and referenced, which can be achieved if we extend the concept of citations in scientific articles.

The paper tackles an important problem and provides a very valuable foundation for further discussions and future implementations. In particular, the six software citation principles presented in the beginning are very valuable.

While I think this is a very useful paper, I see the following problems with the current version:

- The logical connections between the different parts of the paper are not clear. For example, how do the "basic requirements" of Table 2 map to the "principles" in Section 1? And are the principles presented in the beginning the *outcome* of this research (this is what lines 29+ seem to suggest) or are they *assumptions* of the approach to be discussed and evaluated (this is how they seem to be treated in Section 5).

- It is not clear whether the authors consider software citations to be just the first step towards a long-term vision, or whether they consider software citations the overall final solution. It seems that the first is the case, but the long-term vision is not discussed.
Experimental design: This paper doesn't have an experimental design in the strict sense, as it is more of a position paper than a research article. The PeerJ guidelines state that "the submission should clearly define the research question", which is currently not the case for this paper. However, I see a research question between the lines, which could be phrased as "how can citations be applied to software products to account for their increasing importance in scientific research?". By making such a research question explicit, the PeerJ requirements can be met.

The use cases described in Section 3 form the main method to answer the implicit research question. Given the importance of these use cases for this paper, I think reference [18] with the details about these use cases should be published in a more official way, either on a site like ArXiv or FigShare, or maybe even as supplemental material or appendix for this paper. I think having these use cases in the appendix of this paper would be the best solution, which would also increase the impact of these use cases and would strengthen the paper in general.
Validity of the findings: As noted above, the logical structure of the paper is unclear, and therefore it is not clear what exactly the main findings are. I think the main findings should be the principles stated in the beginning, and the use cases would then form the main method to arrive at these findings. Alternatively, one could label the principles as "assumptions" and apply the use case study to validate these assumptions. In any case, I think the findings are valid but they should be labeled as preliminary, as it is not possible to arrive at a final conclusion for such kinds of questions. Only the future can tell whether software citations really work out.
Additional comments: Below are some more specific comments.

line 88: "which will be presented ..." > "which was presented ..."

line 89: "We expect that this discussion may lead to a second, final version": So this paper is not the final version? Please clarify.

line 97: "the need for credit for application software underscores the need to overhaul the system of credit for all research products.": Very true and very important. But then maybe the concept of a "citation" is obsolete altogether, and we shouldn't be advocating "software citations", but something new altogether? Some more discussion on this would be valuable, I think. This relates to my comment above about a long-term vision.

line 238: "Citation is a record of software that is important to a research outcome": This is another important issue, but not much more is said about this. What makes a software "important" to a research outcome? Or, more generally, what is the semantics of a software citation?

line 254: "In addition, if the software authors ask that a paper should be cited, that should typically be respected": I think I disagree with that. Whether a paper should be cited or not should *not* depend on whether the given software package tells you to do so. Such pointers can be useful as suggestions, but not more than that in my opinion.

Lines 308+: You don't imply that this list is complete, but wouldn't an identifier of a software *branch* be another important type of identifier? For example, the Windows "version" of a software package. Also, I see in what situation one would use (1) or (2) with respect to software citations, but what is the purpose of (3) in the context of citations?

Line 325: "other software may be available as a service": This makes a huge difference, doesn't it? I think some more discussion on this would be helpful. For example, with software services it can be hard to find out what the official name of the software is, let alone the developers or version numbers. Also, does a software that is only available as a service (e.g. REST API) violate the Accessibility principle (5)?

I find it a bit weird to quote passages from the same paper like "The Accessibility principle (5) states that “....”" (line 329 and others).

In the discussion about Unique identification (lines 292+), I miss the discussion of hash values as version identifiers, as applied by Git for instance. As the first author is affiliated with GitHub, I assume this aspect was left out intentionally, but why? As basically all software is nowadays maintained with version control systems such as Git, these identifiers come for free and are by design unique. So, I would think they are an important aspect to discuss for software citations.


Instruction: You are one of the reviewers, your task is to write a review for the article. You will be given the title of the article, the number of the round in which the article is located, and your order among the reviewers.

Input:
Title: SOFTWARE CITATION PRINCIPLES
Review round: 1
Reviewer: 2

Output:
Basic reporting: [The submission must adhere to all PeerJ Computer Science policies]

This was not specifically checked by me, but I see no indication that it violates any reasonable policies.

[The article must be written in English ...]

Then paper is written in clear and respectful English.  I found it easy to read.

[The article should include sufficient introduction and background...]

Apart from the principles themselves, I found the background provided to be very useful, relevant and well-referenced.

[The structure of the submitted article should conform to one of the templates...]

I believe the variation in structure from the norms suggested paper is well suited to its purpose.  (See also my response at "Validity of findings".)

[Figures should be relevant to the content of the article...]

The two tables are relevant and helpful, though I did have a slight problem with the legibility of my printed copy of Table 2 (page 5).  The distinction between solid and open circles was not immediately clear to me, and I might suggest using a slightly larger version of these symbols.

[The submission should be ‘self-contained,’ should represent an appropriate ‘unit of publication’...]

The submission appropriately addresses the topic indicated by its title ("principles"), and material deferred to separate publication ("technical solutions" and "examples") seems to be an entirely reasonable separation of concerns.

[All appropriate raw data has been made available in accordance with our Data Sharing policy]

N/A

[Formal results should include clear definitions of all terms and theorems, and detailed proofs]

N/A
Experimental design: N/A - see "Validity of findings"
Validity of the findings: The paper does not present a specific research finding, and as such does not feature a research question and experimental procedure that are expected of such papers.  However, I think it makes in important contribution to the acceptance of software as part of the process and output of research activity, and reflects a consensus formed among a significant community of researchers.  As such, I believe it justifies a place in the formally published literature.

The motivation for the work is clearly stated, and the paper adds clarity and depth to discussion of the role of software in research, and its attendant citation as such.

The process by which the proposed principles were arrived at is described across sections 1-4, and relevant information is referenced, so the principles articulated have a clear provenance and basis in research practice.

(I note that some research fields recognize "method papers", and I would judge this submission by the the standards of such papers rather than as a presentation of a specific research finding.)
Additional comments: The following comments are offered for the authors' consideration.  I don't think any affect the fundamental value of the paper as presented.

Lines 32-35: as a justification for treating software differently from data, I found this passage raised more questions than answers, and that the comments here were uncompelling and unsupported.  For example, some published data is updated on a frequency comparable with some software (e.g. the bi-monthly release schedule of FlyBase is comparable with many software projects - http://flybase.org/static_pages/docs/release_schedule.html).  Also, I think the distinction between data and software is less clear-cut than the authors suggest (e.g. is a schema or stylesheet data or software?).  I would see the difference between software and data as more akin to the difference between research methods and research findings, but that is just a personal opinion.  My suggestion would be that this justification is not really necessary to the overall purpose of the paper, and might be dropped without significant loss of value.

Line 57, et seq: [Nit] I would find this easier to read if the sentence introducing the list of reasons were moved to the same page as the list itself.

Line 88: "will be presented" refers to an event now in the past:  change to "was presented"?

Line 113: [Nit] "work using software" appears to be very broadly scoped, and could apply to just about any activity using a computer.  Suggest "research using software".

Table 2, page 5: I had a problem with the legibility of my printed copy of this table.  The distinction between solid and open circles was not immediately clear to me.  I might suggest using a slightly larger version of these symbols, or maybe more distinct symbol shapes.

Table 2, page 5: [Comment] This table contains a lot of information, and much food for debate.  I was surprised by some ommissions, which may mean that I wasn't fully understanding the concepts being presented:

(a) I find it hard to imagine any of the use cases not benefiting from availability of a "Description", especially 1, 2, 9, and 16.
(b) I would have expected "Indexed citations" to be useful for 5: getting credit.
(c) For publishing a software paper, I would expect all of the basic requirements other than "Indexed citations" to be at least beneficial.
(d) For 12: build a software catalog/registry, would there not be a role for indexed citations?
(e) For 16: Store software try, I'd expect pretty much all of the Basic requirements other than "Indexed citations" to be relevant, and particularly "Description" and "Keywords".
(f) I think there is one use-case missing, which I think is commonly faced by researchers: viz. "Evaluate suitability of software for a given task" - I think this is distinct from discovery, and typically involves making a choice between different available software packages, or to use an existing package, modify an existing package or develop something new.

Line 175: [Nit] "in(1)"  Missing space?

Lines 241-242: "Similarly, the software metadata that is recorded as part of data provenance should be a superset of the metadata recorded as part of software citation."  I found this non sequitur (or am not seeing the logical connection).  I can imagine reasons to cite software that does not form part of the provenance chain of results presented (e.g. alternatives considered).

Lines 245-247: "In general, we intend the software citation principles to cover the minimum of what is necessary for software citation for the purpose of software identification. Other use cases (e.g., provenance, reproducibility) may lead to additional requirements (i.e., enhanced metadata)."  This statement feels to me to be at odds with table 2, which presents "Use cases and basic metadata requirements for software citation", which includes "reproducibility" as a use-case.  This leaves me wondering if the use cases in table 2 are going beyond what is strictly needed for the purposes of software citation.

Line 327-328: "When software exists as both source code and another type, we recommend that the source code be cited".  The term "exists" here doesn't seem quite right to me.  E.g., proprietary software may "exist" as source code, but be unavailable.  Suggest "exists and is accessible" or just "is accessible"?

Line 348: "Updates to this document".  This paper is being presented for consideration as a part of the formal academic record - as such, I don't think "this document" is something which may be updated.  Suggest "Updates to these principles".


Instruction: You are one of the reviewers, your task is to write a review for the article. You will be given the title of the article, the number of the round in which the article is located, and your order among the reviewers.

Input:
Title: SOFTWARE CITATION PRINCIPLES
Review round: 2
Reviewer: 1

Output:
Basic reporting: The authors followed most of my suggestions, and where they did not, they mostly
presented convincing arguments. I like in particular that there is now an
appendix with the use cases.

However, there are two aspects of the latest version that I find not very
convincing:

1. The listed differences between software and data (lines 100-114) are not
convincing in my opinion:

- "Software can be used to express or explain concepts, unlike data": I don't
  understand this.

- "Software is updated more frequently that papers or data" (typo: "that"
  instead of "than"): This was questioned also by the second reviewer, and I
  don't think the issue is resolved. The footnote is unsupported and seems to
  partly contradict the main point. (I can imagine that maybe only *popular*
  software tends to be updated frequently, whereas *popular* dataset are often
  not.)

- "Software suffers from a different type of bit rot than data": I am not
  convinced that this is really a different *type* of bit rot. In particular
  datasets in proprietary formats seem to suffer from exactly the same problems.

- "Software is frequently built to use other software": True, though similar
   things can occur in datasets too, in particular with respect to Linked Data,
   where datasets typically use a number of third-party ontologies and
   vocabularies.

- "Software is generally smaller than data": Another unsupported claim, and I am
  not fully convinced that this is true. There must be a large number of very
  small spreadsheets of data. So, on average data might be smaller than
  software. The difference is certainly true on the upper end of the scale
  though: the largest datasets are larger than the largest pieces of software.

- "Software teams can be large and multidisciplinary, ...": I think this applies
  to data too.

- "The lifetime of software is generally not as long as that of data.": Again
  unsupported, and again I can imagine this to be false. Many datasets, in
  particular small ones, might be stillbirths (i.e. they are never used). The
  same might be true for software, but possibly to a lesser degree. And software
  might commonly evolve, whereas data might more often be replaced by a
  different and newer dataset, in which case evolving software would live
  longer.

2. I am also not convinced by the arguments against hashes as identifiers:

- "commit references are not guaranteed to be permanent": identifiers consisting
  of strong hash values *are* guaranteed to be permanent in the sense that they
  can forever only represent what they were meant to represent, i.e. the bits
  and bytes on which the hash was calculated (which, in the case of Git, is the
  entire version history).

- "A repository address and version number does not guarantee that the software
  is available at a particular (resolvable) URL". It depends. A URL like
https://github.com/torvalds/linux/commit/a0cba2179ea4c1820fce2ee046b6ed90ecc56196
  *does* come with a hash and can be resolved to get to the actual software.
  There is no guarantee that github.com will be available forever, but this is
  also true for publisher sites and therefore DOIs.

- "A particular version number/commit reference may not represent a “preferred”
  point at which to cite the software from the perspective of the package
  authors.": Hashes certainly don't prevent you from defining preferred versions
  for future reference. But they come with the *possibility* to identify all
  versions, which I would in fact consider an important feature. If I want to
  make my research perfectly reproducible, I should be able to cite the exact
  version that I used, whether that happens to be a preferred version or not.

- "When we talk about a “Unique identifier” in the document, we are referring to
  unique, persistent, and machine-actionable identifiers such as a DOI, ARK, or
  PURL; we do not believe a repository address with a version number is
  sufficient." [from the rebuttal letter]: Identifiers that include hashes are
  in important ways *more* unique, *more* persistent, and *more*
  machine-actionable than DOIs or PURLs. Strong hashing algorithms guarantee
  uniqueness (no other input can be constructed for the same hash), are in
  practice probably more persistent (for example, there exist independent
  complete copies of all GitHub content, so even if github.com closes for good,
  the cited software could still be found), and can be automatically verified
  against their supposed content. (The above holds for *strong* hash functions,
  and Git's SHA-1 is no longer considered strong - even though not a single
  collision has been found until now - but this doesn't invalidate the general
  points.)
Experimental design: No further comments.
Validity of the findings: No further comments.
Additional comments: No additional comments


Instruction: You are one of the reviewers, your task is to write a review for the article. You will be given the title of the article, the number of the round in which the article is located, and your order among the reviewers.

Input:
Title: ELECTRONIC LABORATORY NOTEBOOKS IN A PUBLIC–PRIVATE PARTNERSHIP
Review round: 1
Reviewer: 1

Output:
Basic reporting: Language is clear and the article is generally well written.
Experimental design: The experimental design was clear. The authors wrangle with a very difficult problem of how to construct a user friendly electronic notebook and how to test the users satisfaction with the design.
Validity of the findings: The conclusions that the authors draw based on their study are valid. However, the study could be improved by having a more sophisticated ENL system to test. Particular features of the ENL interface could be isolated and the user experience tested. For example, drawing pictures or version control (e.g. per git model).  However, the reviewer understands that this is a lot of work and may be outside of the resources available to the authors. This should be considered in the future.
Additional comments: Plots in Fig. 4 should be overlayed and on the same axis


Instruction: You are one of the reviewers, your task is to write a review for the article. You will be given the title of the article, the number of the round in which the article is located, and your order among the reviewers.

Input:
Title: ELECTRONIC LABORATORY NOTEBOOKS IN A PUBLIC–PRIVATE PARTNERSHIP
Review round: 1
Reviewer: 2

Output:
Basic reporting: The article is well written and contains sufficient related work.
Experimental design: The methodology is well documented.
Validity of the findings: The findings seem valid and conclusions are appropriate.
Additional comments: The paper presents an experience of implementing an ELN in a large collaborative project. The implementation methodology and user experiences and feedback are adequately presented. I think such experiences would be useful for others trying to develop or adopt an ELN. My comments are mostly regarding clarification and some are suggestions.

Suggestions/clarification points:

- (Table 1) What does 'Chemical (sub)structure search' mean? Expand on that little.
- (Line 122) 'The most important items of the collected user requirement are listed in Table 2'. How was the importance of an item determined?
- (Table 2) What does GLP stand for? Good Laboratory Practice?
- (Line 128) What does 'chemical and biological search' mean? This might be obvious to people in biological field, but not to me (computer science background).
- (Table 2) How were requirements classified as core vs. non-core?
- (Table 2) Explain 'chemical and Biological notebook'. Do you just mean a notebook for chemical and biological experiments? If so, rephrase.
- (Line 139) 'two vendors were selected'. Based on what?
- (Line 144) 'The final decision was based on the higher number of positive features in the chosen system'. Just the higher number of features does not seem like a good criteria. Were all features equally important? Perhaps explain little more on the choice here. It would be useful to know the key features that led to choosing one vendor over the other.
- (Table 3) The fifth point ('Users should take responsibility ...') is not clear. It does not read like a feature either.
- (Table 3) 'The potential to continually use the ELN after five year ...' How was this evaluated? Was it just based on cost?
- (Line 174) 'The initially predefined templates were rarely adopted.' I can see that happening. Was there any common format/pattern among users that could be used to revise the default templates? If so, maybe the templates could be revised based on ELN usage for first few months?
- (Line 195) What was defined as acceptable performance?
- (Line 196) Expand on the 'denial of access' case. Was it due to just heavy unexpected usage of ELN cloud or was it due to an external factor?
- (Line 213) Here you say 100 users overall but in Supplement Article S2 you mention the number of overall users was less than 80. In introduction you say 90 bench scientist. Having a para on study participants explaining study participant demographics (age, gender, title, and previous familiarity with ELN) would help. In that para you can explain the overall number of users and number of parallel users, number of users who took the survey, and so forth.
- (Table 4) Are survey results different for users who were familiar with ELN vs. who weren't? If the results are different, it is worth pointing out.
- (Table 4) Mention number of users in parenthesis when saying 'Most user' and 'Many users'. E.g., 'Most user (n=?)'.
- (Table 4) Just curious, did the frequent users who realize an increase in quality of documentation said they would recommend ELN?
- (Table 4) Among users who were not satisfied (seventh point), was there any common theme/reason for their dissatisfaction?
- (Line 262) 'For many users'. How many exactly? Give number, e.g., 'For many users (n=?)'
- (Line 321) '... not to the quality of results.' Not just results. Isn't it also about quality of documentation/reproducibility? It was good to see this paragraph.

- Suggestion for a discussion point: In this experiment it seems, to me, the users were forced to i) change the way they document their experiments, and ii) document their experiments on a computer instead of a paper notebook. Would it have been an easier transition if a paper notebook was printed with a template (from the ELN) and the users were first asked to use the paper notebook (with template from ELN), and then later transition to ELN. This of course would be a speculation (permitted by PeerJ). It might be interesting to add a small para on this from the author's perspective based on their experience.

- (Line 458) Very appropriate quote from one of the users. Incorporate more of such quotes from users throughout the paper wherever applicable. You do have many quotes from users (Survey supplement).

Minor things:

- Some table captions are below the table, some are on top. Convention is to keep caption for tables on top and for figures on bottom. You can choose to keep table captions on top or bottom, but be consistent.
- Show number of users (n) in table/figure captions. E.g., Table 4 (how many users responded to the survey?), Figure 4 (number of users might have varied over time, but you can show median or minimum number of users).
- Bad break for Table 2 and Table 3. It's easier to read if the entire table is on one page.
- (Line 18) typo. feedback
- (Line 46) 'Ioannidis et al., 2014' not 'Ioannidis, et al., 2014'.  No comma between first author and et al. Fix this for other similar citations as well (Line 54, Line 57, ..).
- (Table 3) typo. First point. '... selected solution should *be* intuitive'
- (Line 101) Figure 3 is referenced before Figure 2.
- (Line 240) typo. '... or biweekly *mails* about the ELN ...'
- (Line 208) typo. weekends
- (Line 234) in Aug/Sep 2015 *were* related ...
- (Line 242) subsequent decline in Nov 2014 (no and) is correlated ...
- (Table 4) typo in fifth point. skeptical
- (Table 6) change 'about searching' to 'about the quality of search results' (if thats what you mean, if not clarify what do you mean by searching).


Instruction: You are one of the reviewers, your task is to write a review for the article. You will be given the title of the article, the number of the round in which the article is located, and your order among the reviewers.

Input:
Title: ELECTRONIC LABORATORY NOTEBOOKS IN A PUBLIC–PRIVATE PARTNERSHIP
Review round: 1
Reviewer: 3

Output:
Basic reporting: This is an experience paper that describes the rollout of an electronic laboratory notebook (ELN) system used in a collaborative research program that identifies new mechanisms for antibiotic delivery. The introduction provides good motivation as to why an ELN is important; however, there is no related work section that compares and contrasts the design choices made in the implementation of this ELN compared to others. If there are other examples of similar systems, it would extremely helpful to compare and contrast your design choices to the choices of prior work.
Experimental design: Since this an experience paper, it doesn’t seem that this work answers any particular research question in a traditional sense. This paper provides many details as to how the ELN was implemented, but there was no formal experimental design.
Validity of the findings: The results presented in this paper come in two forms: quantitative (number of experiments performed, number of support tickets) and qualitative (user survey that contrasts experience with expectations). The high level results provided from the survey are quite helpful (Table 4), but I feel some details are missing. It would be helpful to have another table that describes some statistics about the user studies – how many users, distribution of PC vs Mac vs Linux users. Some of this is provide in lines 213 – 222 on page 13, but the numbers aren’t precise – it would be better to pull these out into a table when possible. Finally, some analytics on the survey results could provide some more meaningful insights into precisely why the users felt the system was frustrating to use. This would help to better justify the findings in tables 5 and 6.
Additional comments: Overall, I found this work pretty interesting. There is currently a large push in the medical and scientific communities to digitize health and experiment data. This paper provides a nice overview of the challenges and pitfalls one can face when implementing a large scale collaborative system for sharing experiment findings. The information presented here should be quite helpful to others attempting to implement a similar system.
The supplemental files included with this submission are quite helpful, but I was left with a couple of questions regarding the actual implementation of the ELN. Was data actually entered into spreadsheets such as the ones presented in tables S3 and S4? It might be really helpful to include a couple of screenshots from the actual ELN being used in a browser to give the reader a better understanding as to what the users were dealing with. Currently, I’m not sure how the spreadsheets fit within the web-based GUI used for experiment reporting.
As I mentioned in my comments above, I think you could do a better job quantifying the results from the user surveys. Where quantified results are available, you should present them. When you present the high level results in tables 4 – 6, it would be good to have numbers to bring out how important the findings are. For example, “Windows users were unhappy about the performance of the system” – if there were a table that showed how many users used which platform, we could better understand how big of an impact this really is. Ideally, you should present the data in tables 4 – 6 in histogram format rather than in the currently presented qualitative manner.
Overall, I think the community should find this paper valuable but some improvements can be made to the presentation of results.


Instruction: You are one of the reviewers, your task is to write a review for the article. You will be given the title of the article, the number of the round in which the article is located, and your order among the reviewers.

Input:
Title: ELECTRONIC LABORATORY NOTEBOOKS IN A PUBLIC–PRIVATE PARTNERSHIP
Review round: 2
Reviewer: 1

Output:
Basic reporting: Pass. The article reads well, is clear and connects new knowledge with the broader field of knowledge.
Experimental design: Pass. Experimental design is reasonable and presented well.
Validity of the findings: Pass. Findings are reasonable and interesting.
Additional comments: No comments.


Instruction: You are one of the reviewers, your task is to write a review for the article. You will be given the title of the article, the number of the round in which the article is located, and your order among the reviewers.

Input:
Title: ELECTRONIC LABORATORY NOTEBOOKS IN A PUBLIC–PRIVATE PARTNERSHIP
Review round: 2
Reviewer: 2

Output:
Basic reporting: The authors addressed my comments.
Experimental design: N/A
Validity of the findings: N/A
Additional comments: Thank you for your response letter with details on specific changes performed to address my comments.


Instruction: You are one of the reviewers, your task is to write a review for the article. You will be given the title of the article, the number of the round in which the article is located, and your order among the reviewers.

Input:
Title: ELECTRONIC LABORATORY NOTEBOOKS IN A PUBLIC–PRIVATE PARTNERSHIP
Review round: 2
Reviewer: 3

Output:
Basic reporting: No comments.
Experimental design: No comments.
Validity of the findings: No comments.
Additional comments: After reading through the revisions to the original manuscript, I feel that all of my original comments have been adequately  addressed. I recommend accepting this version of the paper.


Instruction: You are one of the reviewers, your task is to write a review for the article. You will be given the title of the article, the number of the round in which the article is located, and your order among the reviewers.

Input:
Title: INVESTIGATING THE EFFECT OF THE REALITY GAP ON THE HUMAN PSYCHOPHYSIOLOGICAL STATE IN THE CONTEXT OF HUMAN-SWARM INTERACTION
Review round: 1
Reviewer: 1

Output:
Basic reporting: No Comments
Experimental design: No Comments
Validity of the findings: No Comments
Additional comments: The paper shows an interesting topic related to human-being psychophysiological reactions toward activities which have a collaboration between human and machine devices. In reality, there is a big difference of results achieved from simulation environment and reality, especially in matters related to human. Because human has a heterogeneity reaction mechanism even in the same environment, the reactions are dependent on task quantity, under pressure or not, the order of sequence of tasks and so on. Authors indicate that virtual reality devices can be a solution contributes to reducing feeling differences in a comparison between simulation environments and reality. It plays an important role for other study or applicable fields regarding human psychophysiology, but it also would demand another measurement of the impact of virtual reality environment to the awareness of participants.
I am interested in the research topic as well as its application, I suggest to accept the paper.


Instruction: You are one of the reviewers, your task is to write a review for the article. You will be given the title of the article, the number of the round in which the article is located, and your order among the reviewers.

Input:
Title: INVESTIGATING THE EFFECT OF THE REALITY GAP ON THE HUMAN PSYCHOPHYSIOLOGICAL STATE IN THE CONTEXT OF HUMAN-SWARM INTERACTION
Review round: 1
Reviewer: 2

Output:
Basic reporting: No comments
Experimental design: No comments
Validity of the findings: No comments
Additional comments: The paper presents an interesting method on reducing (eliminating) the reality gap on human interactions with robots. I think that it is very important for researchers on virtual (or augmented) reality communities.

Also, the questions has been well designed, and the hypothetical testing has been well done.


Instruction: You are one of the reviewers, your task is to write a review for the article. You will be given the title of the article, the number of the round in which the article is located, and your order among the reviewers.

Input:
Title: INVESTIGATING THE EFFECT OF THE REALITY GAP ON THE HUMAN PSYCHOPHYSIOLOGICAL STATE IN THE CONTEXT OF HUMAN-SWARM INTERACTION
Review round: 1
Reviewer: 3

Output:
Basic reporting: This paper describes a set of experiments in which human subjects observe either a "flat screen" simulated robotic swarm, a robotic swarm rendered in a virtual environment, and a real robotic swarm. The aim of the study is to measure the reaction of the subjects to the three different types of swarm using their psychophysiological states, as well as questionnaires.

From my perspective, the paper is extremely well written. The subject is introduced in a very clear way, with reference to relevant literature. My knowledge of this field (human-robot interaction) is limited, but by reading the literature review from this paper, I got a sufficiently clear idea of the state of the art. I can not say whether the authors missed to cite recent important papers.

The paper is structured very well, with clearly stated hypotheses, very good description of methods, results, and conclusions. I really appreciated the justifications for the statistical tests used which significantly help the reader to digest the results. The figures are fine.
Experimental design: I am not used to work with human subjects, so I have no experience with these type of methods. It seems to me that the experimental design is rigorous and solid enough to provide answers to the initial hypothesis. The factor gender is not balanced. Not sure if this matters, probably not, but adding few words on this would be better.
As I have already said, methods, analysis, and results are very well explained.
Validity of the findings: As far as I can say, this is a novel study, which shows new data concerning the human-swarm interaction. Data discussed in the paper looks to me robust, statistically sound, and rigorously controlled.
Additional comments: I have only minor further comments:
1) It would be nice to say few words on why the interaction with a swarm should be different from the interaction with any other single robot system. If no difference should be made, then your results can be extended to any robotic system, and you could link your study to a broad literature.

2) One part that I found a bit week is about the motivations of the work. The paper, from the title, promise human-swarm interaction. For any expert in swarm robotics, the interest is focused on what the swarm does (or does not). In this work, the actions of the swarm comes after (in term of significance) the response of the subject, which ultimately does not "interact" with the swarm (the subject observes the swarm). I think this secondary role of the behaviour of the swarm, and the non interaction between human observer and swarm has to be made more explicit in the introduction. Otherwise some reader may interpret the "human-swarm interaction" in the title in a wrong way.

3) Point 2 has to be further discussed in the conclusions, when it is time to clarify the relevance of this study for human-swarm interaction. To what extent, an analysis of an observation-based scenario relates to more complex scenarios where the human subjects interact with the swarm? Please, expand this point in the conclusion.


Instruction: You are one of the reviewers, your task is to write a review for the article. You will be given the title of the article, the number of the round in which the article is located, and your order among the reviewers.

Input:
Title: INVESTIGATING THE EFFECT OF THE REALITY GAP ON THE HUMAN PSYCHOPHYSIOLOGICAL STATE IN THE CONTEXT OF HUMAN-SWARM INTERACTION
Review round: 1
Reviewer: 4

Output:
Basic reporting: No Comments
Experimental design: No Comments
Validity of the findings: No Comments
Additional comments: This paper describes an investigation that is timely and interesting. Indeed, the question of whether a simulation can be regarded as equivalent to reality in robotics research has been an object of debate for many years to date. And still one can find nowadays papers accepted in prestigious journals that validate their results in simulation only. This is especially relevant in situations in which the human-robot interaction plays a major role, and this so-called reality gap can make a substantial difference in the psychological effects on the interacting human.

The paper addresses this issue in the context of a recent research field, namely human-swarm interaction. A field in which research in general, and most previous similar studies in particular, are conducted primarily in simulation. For all these reasons, the user study, as described in the paper, is welcome and its hypothesis are relevant and original.

The paper is extremely well-written including all kind of details that make the experiments fully replicable. A strong point of the methodology is the use of both subjective self-reported measures based on questionnaires along with more objective psychophysiological measures (based on skin conductance and heart rate). The experimental procedure is in general correct as well as the statistical analysis of the data. The conclusions are, in principle, reasonably well supported by the experimental results.

However, I have two major concerns regarding the significance of the conclusions.

- The first one refers to the nature of the interaction itself in the experiments. The subjects were instructed to "supervise" the swarm of robots, i.e. "watch attentively" for 60 s. Their behavior was a simple random walk with obstacle avoidance. This watching activity is far from any reciprocal action or influence and, consequently, it is questionable whether this can be considered as an interaction at all. In consequence the validity of their conclusions as pertaining to robot-swarm _interaction_ is limited. Moreover, given the low-level of involvement of the subjects in the robot actions, the expected response of the sympathetic nervous system should be low since this situation will generate little stress in the human. The authors should have made a greater effort to come up with a proper interaction scenario with a greater implication of the participant.

- My second concern is a consequence of the first one and questions the relevance of the study to swarm robotics itself. Since the subjects are essentially watching, the fact that it is robots what they are watching could be psychologically irrelevant, to the extent that they could have been watching non-robotic swarms like a school of fish in a tank, a set of billiard balls moving randomly on a table, or even a complex mechanism with several moving parts with no resemblance to a swarm at all. Would in these cases the results have been the same due to the reality gap? The answer for me is unclear and additional experiments should have been conducted with this kind of conditions as control cases.

Addressing these two questions would require additional experimental work, but I believe it would greatly enhance the merit and impact of research. In this case, the questionnaires should be improved. Asking directly the subjects to select directly their level of valence or arousal is not good practice and there exist much better questionnaires that provide these levels by asking a set of more natural questions about their interaction experience. Probably these questions would not make much sense in the first place since as an interaction experience it was rather poor.

Still the paper could be accepted without additional experimental work if the following recommendations are incorporated:

- The authors write that there is no social interaction between humans and robot swarms. This may be true but they should clarify the notion of human-swarm interaction, describe the nature of the interactions addressed in previous related studies, and justify why their watching action can still be valid as an interaction in this context.

-  The authors should discuss the second point above and justify why, under the given circumstances, the study is relevant to swarm robotics and the fact that robots or even swarms were used is not accessory.

- Apparently the order of the three conditions was random for each participant (please clarify this more explicitly). It would be interesting to know whether there are significant differences among the same condition in different order; given the rather passive role of the subjects, the third time they watch the same situation a weaker reaction may result in any case due to habituation.

- In experiments in social HRI results tend to be different depending of the gender of the participants. Having results separated and analyzed by gender would add value to the study.

- When reporting the literature about the reality gap in social HRI the term "more enjoyable" is repeated again and again. The authors should be more precise in their reporting and specify what psychological effect was measured in each case.

- Since the reader is referred to Garattoni et al. (2015) for details about the software infrastructure and it is a non-archival publication, a web link should be provided.


Instruction: You are one of the reviewers, your task is to write a review for the article. You will be given the title of the article, the number of the round in which the article is located, and your order among the reviewers.

Input:
Title: INVESTIGATING THE EFFECT OF THE REALITY GAP ON THE HUMAN PSYCHOPHYSIOLOGICAL STATE IN THE CONTEXT OF HUMAN-SWARM INTERACTION
Review round: 2
Reviewer: 1

Output:
Basic reporting: No comments.
Experimental design: No comments.
Validity of the findings: No comments.
Additional comments: The authors have convincingly replied to my previous comments and have satisfactorily incorporated my recommendations, including additional analysis regarding the gender effect and that of the order of the conditions.
Regarding my comment about a link to Garattoni et al. (2015), the link provided seems not to work. I got the following error using three browsers. Please check.

The requested URL /IridiaTrSeries/link/IridiaTr2015-004.pdf. was not found on this server.


Instruction: You are one of the reviewers, your task is to write a review for the article. You will be given the title of the article, the number of the round in which the article is located, and your order among the reviewers.

Input:
Title: DEEP LEARNING FOR CONSTRUCTING MICROBLOG BEHAVIOR REPRESENTATION TO IDENTIFY SOCIAL MEDIA USER’S PERSONALITY
Review round: 1
Reviewer: 1

Output:
Basic reporting: The paper is basically well-organized and well-written. However, there are several grammar mistakes I noticed that lead to unclarity of readers' understanding. They are listed as follows:
1. In abstract, "which could unsupervised extract Linguistic Representation Feature Vector (LRFV) from text published on Sina Micro-blog actively." where "unsupervised" should be "unsupervisedly", or the authors can change the structure of the sentence. Same goes to others.
2."Collecting volunteers’ IDs of Sina Micro-blog, we crawled their micro 119
blog data through Sina Micro-blog API. The micro-blog data collected consists the users’ all blogs 120 and their basic status information, such as..." Obviously "crawled" should be replaced with words like "obtained"and "consists" should be "is consist of" since consist is an intransitive verb.
3. In 5.1 "there are weak correlation..." should be uncorrected use.

I think you need to ask a native for revision of English.
Experimental design: The authors describe their methods of study pretty clearly and successfully. The key point of this study lies largely on how good is the "dictionary". I actually looked into the "LIWC" 1 years ago. This software is based on English key word counting process and neglects so many important grammars. Thus the correct rate of it is just so so.

I noticed your decryption of your "dictionary". My question is how precise your algorithm performs, and in what way is yours different with previous methods?

Also I would like to know why you choose linear regression for prediction.
Validity of the findings: How you would consider the performance of the proposed algorithm valid? I have seen enough description on this part.
Additional comments: The paper is basically well-organized and well-written with interesting results and findings.  Still it need some kind of revision. Details can be found in separated parts.


Instruction: You are one of the reviewers, your task is to write a review for the article. You will be given the title of the article, the number of the round in which the article is located, and your order among the reviewers.

Input:
Title: DEEP LEARNING FOR CONSTRUCTING MICROBLOG BEHAVIOR REPRESENTATION TO IDENTIFY SOCIAL MEDIA USER’S PERSONALITY
Review round: 1
Reviewer: 2

Output:
Basic reporting: The notation might be ambiguous: author first use the word 'dimension' for the 5 traits (line 208), then use 'dimension' to represent the number of the features for prediction. It would be better if two term could be distinguished.

Figure 3 is only a deep architecture of stacked autoencoders. However, authors should include the exact structure that used in their study, instead of a general structure (for example, number of units in each layer, and number of layers). Or at least a table of the network used should be represented. You can find examples in the famous Alex net paper.

Sample size is relatively small in this setting (1552). With this sample size, deep learning is not so necessary. It is better to explain how and why deep learning works in such small size, compared to other algorithms.
Experimental design: 1. In line 215-216, Authors trained five personality prediction models based on linear regression algorithm using corresponding linguistic behavior feature vectors and labels. Does these mean the five linear models are trained separately? This might lose the information between five dimensions.

2. Line 275 mentions sigmoid function is used as activation function of hidden layers. There are many choice for the nonlinearity units in the network. Is there any experiment with other units, or are there any literatures suggest the preference?

3. Line 124 mentions: If the total number of one’s micro-blogs is more than 500, this volunteer is a valid sample. This is reasonable, but might cause bias in the experiment. Authors might be more careful of this step and make more discussion for the potential outcome of this step.

4. The dimensionality of linguistic behavior feature vector is set according to the optimal result of prediction model. Did authors try first generate high dimensional data, then use some statistical method for feature selection, e.g. lasso?
Validity of the findings: 1. In the section 'Evaluation measures', authors did not mention explicitly what is "prediction". Does the data split into training and testing data? Did testing data used in feature extraction?

2. Authors study the relationship of the number of features and the prediction performance. There is an interesting pattern in figure 4b that 300 is a local minimum, while does not shown in the 4a. Authors might add more discussion of why that happens.
Additional comments: No additional comments


Instruction: You are one of the reviewers, your task is to write a review for the article. You will be given the title of the article, the number of the round in which the article is located, and your order among the reviewers.

Input:
Title: DEEP LEARNING FOR CONSTRUCTING MICROBLOG BEHAVIOR REPRESENTATION TO IDENTIFY SOCIAL MEDIA USER’S PERSONALITY
Review round: 1
Reviewer: 3

Output:
Basic reporting: No Comments
Experimental design: No Comments
Validity of the findings: No Comments
Additional comments: In this paper, the authors use deep learning methods to study the social media user's personality problem. Both the topic and technique is quite new.
My major comments are as follows,

1. For data collection, linguistic behavior is extracted based on the FFT methods. By applying FFT, we always assume the signal is stationary. However, whether micro-blog data is stationary signal or not is still highly questionable. Since micro-blog data does not only depend on user's thoughts, ideas and feelings. It is also strongly influenced by public events, like recently death of college student Zexi Wei raises questions on Baidu's ethics, which typical make the data non-stationary.

2. In method, Stacked Autoencoders is used as a deep learning method for dimension reduction, which is an unsupervised method.  The author mentioned that compared with supervised methods unsupervised method is more objective, efficient, comprehensive and universal. However, normally supervised dimension reduction method would achieve a better performance than unsupervised methods. It is better to make a further discussion and compare different unsupervised and supervised dimension reduction methods in result.

3. In result, the authors use different parameters for the demission of feature vector. What kind of principle has been used here for parameter selection? If the authors select the parameters corresponding to the best result, it would be unfair to the other methods in the comparison and also unable to be used in practical. And also, how many components are used in the PCA? Again, what principle has been used for parameter selection in PCA? Furthermore, the introduction of the SAE is not sufficient? There is no definition of function f and g.  What does \theta and \theta' stand for in Eq. (4) and Eq. (5)? How to determine the demission of x'? How to training the W, b, W', b in Eq. (4) and (5)? As the authors mentioned in line 200, the number of layer would be decided according to the optimal value of many experiments? Which kind of experiments will be done? Is it time consuming or did we need additional data in these experiments. Finally, it should also mentioned the time and memory consuming for the method of Stacked Autoencoders in this application, and make a comparison with the others, like PCA. In all, a lot of technique detail has not been described, which makes the readers unable to get a comprehensive understanding of the proposed method.

Minor questions:
1. Does all the users have their micro-blogs account 3 years ago?
2. Eq. (9) and (10) is not so strict, Eq. (9) is in a vector form and Eq. (10) is in a scalar form.


Instruction: You are one of the reviewers, your task is to write a review for the article. You will be given the title of the article, the number of the round in which the article is located, and your order among the reviewers.

Input:
Title: DEEP LEARNING FOR CONSTRUCTING MICROBLOG BEHAVIOR REPRESENTATION TO IDENTIFY SOCIAL MEDIA USER’S PERSONALITY
Review round: 2
Reviewer: 1

Output:
Basic reporting: No further comments.
Experimental design: No further comments.
Validity of the findings: No further comments.
Additional comments: After revision, this paper meets the standard for publishing. I would suggest accept this article.


Instruction: You are one of the reviewers, your task is to write a review for the article. You will be given the title of the article, the number of the round in which the article is located, and your order among the reviewers.

Input:
Title: DEEP LEARNING FOR CONSTRUCTING MICROBLOG BEHAVIOR REPRESENTATION TO IDENTIFY SOCIAL MEDIA USER’S PERSONALITY
Review round: 2
Reviewer: 2

Output:
Basic reporting: No Comments
Experimental design: No Comments
Validity of the findings: No Comments
Additional comments: I’m the third reviewer for the previous version of the manuscript.
It’s great honor for me here to be the reviewer again for the revision. But I still should say,  the authors did not answer my questions well.

My first question is why FFT is used for feature extraction. In the reply the authors showed me more detail about the result after FFT.  However the authors still did not make any explanation about the reason of using FFT.

The second question about supervised and unsupervised methods. The authors mentioned that, "for supervised methods, the directivity of dimension reduction results is stronger"; and the reason for using unsupervised method is that "there are some limitations of obtaining labeled data". In fact, after the dimension reduction, the label information is still needed for regression. So logically, the label information must be known for the proposed method. Since this, I did not find any reason here for using unsupervised deep learning method in dimension reduction instead of some supervised methods.

The last question is about the principle of parameter selection. Since there are so many parameters need to be determined in the proposed method, parameter selection would be key important for the success of the method. Here, the authors determine the value of the parameters by the optimal prediction results. This principle is the most direct way for parameter selection. However it is easy to lead to the parameters overfitting to the dataset, which makes the results harder to believe. Cross validation is an alternative approach, which will leads to more computation. But at least it should be done on selection of the most important parameters.


Instruction: You are one of the reviewers, your task is to write a review for the article. You will be given the title of the article, the number of the round in which the article is located, and your order among the reviewers.

Input:
Title: DEEP LEARNING FOR CONSTRUCTING MICROBLOG BEHAVIOR REPRESENTATION TO IDENTIFY SOCIAL MEDIA USER’S PERSONALITY
Review round: 3
Reviewer: 1

Output:
Basic reporting: No Comments
Experimental design: No Comments
Validity of the findings: No Comments
Additional comments: In the revision, I see a great effect has been made by the authors. All the questions I asked has been answered well. The quality of the manuscript has been improved a lot. So I suggest to accept it for PeerJ.


Instruction: You are one of the reviewers, your task is to write a review for the article. You will be given the title of the article, the number of the round in which the article is located, and your order among the reviewers.

Input:
Title: GRNSIGHT: A WEB APPLICATION AND SERVICE FOR VISUALIZING MODELS OF SMALL- TO MEDIUM-SCALE GENE REGULATORY NETWORKS
Review round: 1
Reviewer: 1

Output:
Basic reporting: No Comment
Experimental design: Dahlquist et al present GRNsight, a service for visualising gene regulatory networks of the ‘small-to-medium-scale’. The tool is available via a web application and the source code is also provided. On receiving a manuscript such as this I immediately try and use the application and I compare it to my current preferred tools in the domain. The web application loads quickly and is reasonably intuitive to use. It is not so clear that the force algorithm is disabled on specific nodes once that node has been manually relocated to a new position. This may be described under the help section, but I could not find it although it is mentioned in the manuscript (line 217). The visual style of the network is good. Certain behaviours of the visualisation are not compatible with internet explorer (which the authors do not mention is supported – line 239). However, the only platform for which mouse over on links revealed the underlying weights was firefox on windows 7. Neither safari, chrome not firefox on OSX or Chrome on windows 7 showed the weights. I would like to see an option on the tool for showing or hiding all weights.
Validity of the findings: My two preferred tools for laying out networks are BioTapestry (http://www.biotapestry.org/) and YeD (http://www.yworks.com/products/yed) . In neither case could I see a simple way to import the information from the excel file specified by the authors as output from the GRNmap package to reproduce the visualisation that they show in GRNsight. The specific issues are weighted networks – neither of these packages provide a direct method of importing these data from a matrix in excel. However GRNsight itself does not work directly from a matrix – rather it parses that network through JSON. The central constraint for GRNsight is the file format output by GRNmap. I could write a parser for that matrix into a format such as GraphML and then import that in to YeD and have far more tools available to me.

The authors themselves recognise this to some degree and argue that their tool is aimed at doing one thing well. I agree that this tool does present a network from GRNmap well. However, given the existence of 47 other tools which already do something similar it must be an exceptionally good tool. I am familiar with cytoscape, but not Gephi and so tried Gephi to visualise a network similar to that presented here. It is possible, but as the authors state non-trivial.

Whilst I agree that simplicity is key (cognitive load – line 103) I am less clear about the ‘understanding the biological results of the model’ enabled by GRNsight. The authors discuss these on lines 281-289 giving generalised interpretations of issues such as feed forward motifs, highest in-degree and the regulatory chains. Yet as far as I can see, these are all determined by visual inspection. Furthermore none of these rely on the weights, which the distinguishing feature of GRNsight. I would be able to identify these more easily in a system such as YeD and do not require the weight information to do so – a simple sif file format giving the directionality between two nodes is sufficient.

The visualisation of the weight parameters described in lines 290-322 is the key behaviour here. Figure 5 D,E show clearly the impact of the addition of this weight information, yet E is the clearest visualisation and the only node within it that is located in anything close to its original position is Ace2 – almost every other node has been moved by hand. This reveals a flaw in the implementation of the force-spring layout algorithm as applied in GRNsight. Given the small area of the view port and the constraint that all nodes remain within it, the layout is sub optimal as the force-spring cannot reach its most relaxed state. Furthermore when looking at figure 5 I am always drawn to panels C and F as being the most informative view. Thus I would argue that the force-spring algorithm used here does not provide much benefit to the layout of the network other than separating nodes from one another. The useful layout requires manipulation. Taking the network from E and recreating it in YeD revealed that just a hierarchical clustering and layout gets you closer to E without any manual intervention.
Additional comments: Ultimately the tool presented here is useful for interpreting the results of GRNmap, but I would be unlikely to use it in any other situation. As it does not accept a standard input file type, the output of any other network analysis package requires conversion in to the matrix format required here. Similarly, the tool provides no export function (the option in the File menu remained stubbornly greyed out) and so I can’t take a network from GRNsight and utilise it elsewhere. I also can’t use GRNsight to convert the GRNmap format to something I might like to use elsewhere. GRNmap itself looks to be a very interesting package and I would like to explore it further, but I would be looking at converting its output into something I could use in a number of other pipelines.

The authors refer to future features coming in version 2 (lines 323-329). I encourage them to consider implementing at least one standard filetype for displaying graph data within their tool. Be it sif, graphml or even gml, it would significantly increase the utility of the tool as it currently exists. Alternatively this tool not be considered stand alone from GRNmap.

My comments are rather focussed on the tool and its usability, less on the manuscript itself. The manuscript is generally well written and clear. It’s weakness lies in the arguments about biological insight derived from the visualisation. If I were to parse the matrix into a graphml file format, I could visualise these networks (complete with weightings and line endings etc) in a wide array of tools and – I would argue – extract more biological value from the interpretation. I would encourage the authors to expand on this section of the manuscript if possible.


Instruction: You are one of the reviewers, your task is to write a review for the article. You will be given the title of the article, the number of the round in which the article is located, and your order among the reviewers.

Input:
Title: GRNSIGHT: A WEB APPLICATION AND SERVICE FOR VISUALIZING MODELS OF SMALL- TO MEDIUM-SCALE GENE REGULATORY NETWORKS
Review round: 1
Reviewer: 2

Output:
Basic reporting: The article "GRNsight: a web application and service for visualizing models of small- to medium-scale gene regulatory networks" is well written and documented. I have found disappointing the level of review of other similar network visualisation tools that exist out there. They only provide comparison agains Cytoscape and Gephi. They do not refer (at least in the introduction) to the Cytoscape.js web application, only to the console application. This to me raises concerns in an otherwise exhaustive work, providing ground for not having extensively researched other similar tools in the field, which is quite crowded. A search in the BioJS registry (http://biojs.io/; an NPM based repository of biological web application) of the word "network" retrieves 12 components. Not all of them are necessarily relevant for this publication, but at least some of them should be compared against and a case should be made so that it is made apparent how GRNsight provides valuable new functionality that is not redundant.

You do mention Cytoscape.js in line 175 but the justification of creating GRNsight because of the future possibility of implementing other D3.js visualisations is not well founded in my opinion. The http://biojs.io/d/biojs-vis-interactions-d3 network visualisation component does use already D3 for network visualisation.

The supplementary figures are very helpful to the understanding of the article.
Experimental design: N/A
Validity of the findings: I am not sure that the visualisation of 75 nodes or 150 edges is the kind of magnitude that would be valuable to many potential users. I would find it more impressive if the capacity to render nodes was in the order of thousands (even though this may be impractical to visualise and some data reductions might be necessary).

To me the bits that I have found most useful for the tool are:
- The visualisation is pleasing and intuitive
- The documentation is extensive and easy to read
- The demos allow users to quickly grasp the functionality
- The article is clear and the results show the relevance of the functionality
- The emphasis on testing and best practice are well appreciated although not complete, see below
- Networks can be uploaded via an xlsx file

I would have also liked some more emphasis on the "findable" and "reusable" aspects of open source software principles. I do believe some mention to "FAIR" principles could be useful: Findable, Accessible, Interoperable, Reusable. This has been done with data sharing (http://www.nature.com/articles/sdata201618) but this applies to software.
Additional comments: Bioinformatics web tools like GRNsight are published in the literature but they are not made accessible via a centralised repository like the BioJS registry. I would thoroughly recommend authors to make accessible their tool via the BioJS registry. The requirements for making a package accessible through the BioJS registry are minimal:

- The source code has be made available via GitHub
- It has to be made available in the Node.js Package Manager (NPM), the package manager for JavaScript
- The "biojs" keyword has to be included in the "package.json" file of NPM

If the authors had searched the BioJS registry for the keyword "network" they would have found components that could have at least compared against. By not checking the BioJS registry and not including GRNsight in it they have missed the opportunity to increase the exposure of their tool and potentially lose valuable engagement with a community of reference who might even keen to contribute to the code. It is thus important to appear that this tool is not coded in isolation.


Instruction: You are one of the reviewers, your task is to write a review for the article. You will be given the title of the article, the number of the round in which the article is located, and your order among the reviewers.

Input:
Title: ITERATIVE GUIDED IMAGE FUSION
Review round: 1
Reviewer: 1

Output:
Basic reporting: No comments, see attachment for minor tracked changes.
Experimental design: No comments
Validity of the findings: No comments
Additional comments: This new and important fusion scheme enables combining the fine details found in image intensified (nighttime) scenes with the lower detailed but higher contrast thermal imagery, while reducing certain types of noise. Performs as well as other, current fusion approaches. Its computational efficiency allows it to be possibly incorporated into (near) real-time imaging systems.


Instruction: You are one of the reviewers, your task is to write a review for the article. You will be given the title of the article, the number of the round in which the article is located, and your order among the reviewers.

Input:
Title: ITERATIVE GUIDED IMAGE FUSION
Review round: 1
Reviewer: 2

Output:
Basic reporting: The papers present a novel multi-scale image fusion algorithm that can fuse images of different modality. The multi-scale image fusion algorithm use guided filter to extract coarse scale geometric structure and fine scale details (i.e. the image residual)  from images at different level of details (scale in scale space sense). Furthermore, saliency maps are used to select appropriate information from the images to be fused.  Guided filter is applied to the saliency maps to enforce spatially consistency in saliency map. The main contribution in the paper is the novel GF based multi-scale image fusion algorithm for fusion of images of different modality, and the extensive evaluation of the presented algorithm against other state-of-the-art multiscale fusion algorithms.

The paper reads well, the background materials and the own contributions are presented in a clear and balanced manner. The background contains a large number of application of image fusion which point out the importance of the problem at hand. A large number of alternative – older and newer – multi-scale image fusion methods are discussed (or mentioned). Bilateral filter, joint bilateral filter and guided filter are presented in details. The guided filter is presented in a short, but detailed manner, and is very well written. Some details in the presentation of the iterative guided filter is unclear.

Comments:
The terms resolution/multiresolution/lower resolution are used without being defined. Commonly multiresolution reference to some kind of image pyramid – for example the Gaussian image pyramid and resolution is commonly used for the numbers of pixel in the image. In this paper resolution is instead used in a rather general Gaussian scale space meaning – resolution is the level of details present in the image (i.e. the sigma in linear Gaussian scale space  - inner scale).  Also the terms base layer and detail layer are less frequently used in the literature. Commonly an image can be decomposed into geometry (base layer) and texture (detail layer). The detail (texture) layer is also commonly referred to as residual image/layer.

The GF filter is defined as
O = GF(I,G;r,e)
where I is an image input, G is a guidance image, r is the window size parameter and e is a regularization parameter.  The parameters in the IGF is not explicitly given. As it seems the image input is X_0 in all iteration while X_i is updated in the iteration. Explicitly stating the parameters in the IGF would clarify the issue.

The key features in the saliency map computation is the local contrast. Combining two images with potentially different contrast magnitude into BW_X and BW_Y can be non-trivial. Does all the test image pairs used in the fusion have the same contrast magnitude or have the images been normalized?
Experimental design: No Comments
Validity of the findings: The presented method is evaluated using 4 objective metrics. As the author points out no standard universal metric for objectively evaluating image fusion exists. Instead a number of metrics exists that is commonly used in evaluation of image fusion- of which 4 has been selected. The data set used in the evaluation is 12 pairs of images (Visual + IR (LWIR?) or NIR). The dataset used in the evaluation is rather small, the scene are quite similar and the modality combination the same in almost all pairs. It would be interesting to evaluate the performance with image of different modalities (i.e. there the information correlation between the images varies: Vis + SWIR, Vis + MWIR and Vis + LWIR). The results on the dataset are very promising and the method achieve state-of-the-art results.
Additional comments: Overall a very well written paper, containing sufficient novel ideas, presented in a clear way and sufficiently evaluated against state-of-the-arts algorithms on a limited dataset.


Instruction: You are one of the reviewers, your task is to write a review for the article. You will be given the title of the article, the number of the round in which the article is located, and your order among the reviewers.

Input:
Title: ITERATIVE GUIDED IMAGE FUSION
Review round: 1
Reviewer: 3

Output:
Basic reporting: The article is well written and sound in its descriptions. It articulates the large amount of technical detail very clearly to the audience and provides sufficient background for understanding the basic concepts.

Minor comments:
-In the introduction, it might be helpful to provide example (or small description) of what encompasses image noise and/or image artifacts given that fusion aims to eliminate such features. I understand there is not always strict definition for these constructs but some motivating examples may be helpful.

-Section 2.1 into 2.2, there is no example of what can be used as a guidance image G until lines 152-153, although it is talked about in lines 143-144. This may be fixed simply by moving what is described within the parentheses to the earlier mention. Also, are there times when something other than the identical input image is used for G? If so, and if it is of simple enough of a statement to add in, it might be helpful to mention what other type of image might be used.
Experimental design: The newly proposed method is very well described in the text. In particular, I find the 1-5 iterative breakdown in lines 265-276 very helpful in promoting understanding of the steps of the process. Additionally, the test imagery and process for evaluation of the new and classic fusion techniques is clear and valid to traditional study of image fusion.

Minor comment:
-If at all possible, it would be nice to view the images at a larger size in the figures for ease of viewing, however because I think it is important to show all image conditions as you do in the paper now, this may be an impossible trade-off between size and space.
Validity of the findings: The findings, using classic, valid methods for image fusion evaluation, are very clear and in strong favor of the new technique. The summary tables are helpful in clarifying results.
Additional comments: minor edits:
-"artifacts" vs "artefacts" consistency throughout
-Line 415 C2 should be subscripted


Instruction: You are one of the reviewers, your task is to write a review for the article. You will be given the title of the article, the number of the round in which the article is located, and your order among the reviewers.

Input:
Title: DECENTRALIZED PROVENANCE-AWARE PUBLISHING WITH NANOPUBLICATIONS
Review round: 1
Reviewer: 1

Output:
Basic reporting: In this paper the authors propose a low level infrastructure for distributing and decentralizing nanopublication datasets. As a part of the infrastructure, the authors also define tools for validating the nanopublications, publishing them and reviewing them. The approach is evaluated with 5 datasets spread around the globe.

The paper is well written and relevant for the Peerj journal, but I have several concerns with its originality and novelty. When I was reading the paper I had the feeling that I had already reviewed this work. By the time I reached section 2 I realized that indeed I had, because I was a reviewer of the ISWC paper from which this work is derived. I have thoroughly compared both versions, and my conclusion is that the contribution of this paper is exactly the same as the one described in the previous paper. More details are provided on this version (e.g., the part on how the servers propagate the nanopublication has been added), which is great (and I asked about this in my original review, so it's fantastic to see that the authors have acknowledged some of my comments), but I fail to see what does this work add to the previous original paper: the approach remains the same, the introduction and conclusions are almost a copy-paste of the original one and even the two evaluations are almost identical: table 1 has 2 more datasets but are not used in the evaluations, and figure 7 has been redrawn, but that's all. The web interface of the nanopublication validator is new, but that is not a significant contribution to become a new publication.

I already accepted the original paper because it was (and it is) a nice work. I am marking a "major review" decision because I want to give the authors and opportunity to explain to me why this paper is a significant contribution compared to the original one. As it stands now, I think that the paper should not be accepted in the journal.

Below I add additional comments from my original review which I think also apply to the current paper:

- The authors state that a nanopublication is never edited, just added. Then, what would happen if an author wanted to delete or retract a given published fact? How would the deletion affect the other copies among the server network?

- The data used for the evaluations i.e., for building figure 7 and 8 doesn't seem to be available. However the authors reference the datasets, so I don't think it is a major issue. Additionally, all the code is on Github, which is great.
Experimental design: All the comments regarding the originality of the research are included above. The methods and evaluations are clearly described in the paper.
Validity of the findings: The validity of findings is identical to the original (non extended) publication, because the evaluation is almost the same.
Additional comments: No additional comments


Instruction: You are one of the reviewers, your task is to write a review for the article. You will be given the title of the article, the number of the round in which the article is located, and your order among the reviewers.

Input:
Title: DECENTRALIZED PROVENANCE-AWARE PUBLISHING WITH NANOPUBLICATIONS
Review round: 1
Reviewer: 2

Output:
Basic reporting: The authors adhere to the policies of the PeerJ Computer Science Journal.
Experimental design: The empirical evaluation conducted by the authors is clearly defined in my opinion. The setting used is clearly defined, and the results obtained are analysed in an objective manner.
Validity of the findings: The finding reported in the results are justified by the empirical evaluation. There is however the aspect related to reliability that has not been covered in the empirical validation (see comment 3 in the general comments to the authors).
Additional comments: The authors tackle in this paper the problem of publishing datasets in decentralized environments with distributed control over the publishing and the retrieval of datasets. The main motivation being the fact that there is currently no effective and reliable means for publishing datasets that are referenced from within scholarly publications.

The paper has the merit of reporting a practical solution. I enjoyed reading it as it is grounded with real examples all a long together with analyses that justify the choices made by the authors when elaborating their solution. That said, below are comments that I think are worthwhile addressing or at least discussing in the paper.

C1. The authors stated two main requirements: Having a reliable mechanism for hosting and referencing datasets, but also the ability to reference and retrieve datasets at different granularity levels. With respect to the second requirements, the authors did not discuss the process by which a datasets is transformed into a set of nanopublications. The model proposed by the authors support nano-publication and nano-publication indexes, which references other nano-publication. The question the reader may ask is how nan-publication and their container nano-publications indexes are obtained given a dataset.

C2. A second aspect that is related to the first comment is the heterogeneity of datasets. Scientific datasets are usually heterogeneous, and the most part of it is not stored in the form of RDF, but instead in CSV, relational, etc. Can the solution proposed by the authors cater for this kind of datasets?

C3. Regarding reliability, it is not clear on how the dataset is replicated among the server. In other words, what is the replication scheme used to ensure the availability of the datasets.

C4. I also note that in the evaluation section, the problem of reliability has not been examined.

C5. The publication of datasets should in general be accompanied with the publication of metadata within catalogues that inform prospective users with their availability. I think that this aspect is worth discussing in the paper.

Minor comments.

C6. At the end of the introduction, the authors state that the article is an extended version of a previous article. The authors need to state clearly what is new in the paper submitted to the PeerJ.

C7. The URIs used by the authors contain the artifact code which obtained by analyzing the data content. This raises the question as to the cost that this operation may incur when building the URIs. This aspect was touched on in the paper, but briefly, and need further discussion.

C8 In Figure 2 and the text that explain the Figure, the author use the expression “propagation”. Later this term is explained, but until then it is not clear for the reader what is meant by propagation.

C9. In page 5, 1st paragraph towards the end: reminder -> remainder


Instruction: You are one of the reviewers, your task is to write a review for the article. You will be given the title of the article, the number of the round in which the article is located, and your order among the reviewers.

Input:
Title: DECENTRALIZED PROVENANCE-AWARE PUBLISHING WITH NANOPUBLICATIONS
Review round: 1
Reviewer: 3

Output:
Basic reporting: The submission is very clearly written and well presented. It is mostly self-contained, although some aspects could benefit from some additional background information.

1. Although past literature regarding trusty URIs was cited, I did not understand how they fit into the nanopublication server network (section 3.2) until I re-read the paper from ESWC. So the idea is that the proposed approach prerequisites the ability of having trusty URIs for all the nanopublications exchanged within the network. It is kind of what figure 2 shows and what section 3.2 says. But could have been made clearer. Also, does this mean that all the nanopublications exchange in this network must be published using the nanopublication java library, or something similar? Then I would say the schematic architecture is a bit too high level for readers to understand these key points. A technical architecture diagram, at least for representing an instance implementation of the proposal, could be helpful.

2. On line 257 the authors claim that there is an assumption that all nanopublications should be all similar in size, which makes it one of the advantages of the proposed approach. As someone who do know about nanopublications, I am not sure this is a universal true. Have the authors got some citations to support this statement, or is this a kind of best practices expected from nanopublication users?
Experimental design: The hypothesis of the submission is: Can we create a decentralized,  reliable, trustworthy, and scalable system for publishing, retrieving, and archiving datasets in the form of  sets of nanopublications based on existing Web standards and infrastructure? Therefore, we expect the experiment to evaluate the following aspects of the proposed system: its reliability, trustworthiness, and scalability. However, in my opinion the experiments are not completely satisfactory.

In terms of evaluation1, it does not seem ideal to take more than 13 hours to load and populate the test datasets. Obviously it will probably take marginal time to load and populate a singular nanopublication. But as the authors say, it’s often that people would publish a collection of nanopublications as a dataset. What would be the typical size then? Would make any sense to evaluate the load time for different sized nanopublication datasets? I would say 13 hours is a very long time even for 200 million triples.

In terms of the second evaluation, what is the SPARQL query used for testing the Virtuoso server? The experiment could use more explanation. Currently, the procedure seems to be very much tailored to the described system, for example, going through the internal journal page etc. Is this a realistic nanopublication retrieval scenario? Also, this did not demonstrate retrieval in a decentralized setting, but simply a stress testing of a single server.

The evaluation section seems a bit incomplete and weak. How about scalability? What would be the scalability bottleneck of the proposed design, the indexes or the propagation procedure? Could the system handle communication of 100 or several hundred nodes?
Validity of the findings: The data could be better explained so that users can understand why something worked well (such as the reliability line 603-617), and how they support the claimed hypothesis. If evaluations are incomplete in terms of supporting the whole hypothesis (such as trustworthiness), then the authors should explain whether the evaluation may be incomplete.
Additional comments: No additional comments


Instruction: You are one of the reviewers, your task is to write a review for the article. You will be given the title of the article, the number of the round in which the article is located, and your order among the reviewers.

Input:
Title: DECENTRALIZED PROVENANCE-AWARE PUBLISHING WITH NANOPUBLICATIONS
Review round: 2
Reviewer: 1

Output:
Basic reporting: The authors seem to have addressed all my comments successfully. They have shown how the paper includes new material from the former ISWC publication, and I think that the response for being able to edit or fix nanopublications makes a lot of sense. I would have referred to that more as "archival" rather than publishing, but I don't think it's a big issue.

What I would like to recommend the authors is to include the link to the repository with the data from the evaluations (https://bitbucket.org/tkuhn/trustypublishing-study/src/). That way anyone interested in the data would be able to access it.
Experimental design: The authors have expanded on the evaluation and discussed some limitations of their approach
Validity of the findings: No comments
Additional comments: No comments


Instruction: You are one of the reviewers, your task is to write a review for the article. You will be given the title of the article, the number of the round in which the article is located, and your order among the reviewers.

Input:
Title: DECENTRALIZED PROVENANCE-AWARE PUBLISHING WITH NANOPUBLICATIONS
Review round: 2
Reviewer: 2

Output:
Basic reporting: The article is well written and make a pragmatic proposal that I think is useful and interesting for the linked data community.
Experimental design: The authors addressed my main concern about the empirical evaluation by  running a new experiment to assess the reliability aspect.
Validity of the findings: The ideas presented in the paper are backed by empirical evaluation, and as such I think that the findings are solid.
Additional comments: I would like to thank the authors for attempting to address my comments. They have managed to address most of them, and ran a new experiment, albeit of a small size, to address the point of reliabaility that I raised. There are 2 comments from the previous round that still need to be addressed by the authors.

C1. The authors did not address my first comment in the previous round. “The authors stated two main requirements: Having a reliable mechanism for hosting and referencing datasets, but also the ability to reference and
retrieve datasets at different granularity levels. With respect to the second
requirements, the authors did not discuss the process by which a datasets is
transformed into a set of nanopublications. The model proposed by the authors
support nano-publication and nano-publication indexes, which references other
nano-publication. The question the reader may ask is how nan-publication and
their container nano-publications indexes are obtained given a dataset”.

C2. In their response to C2 in the previous round, the authors suggest that heterogeneity of the data model used in datasets (e.g., CSV and relational) can be resolved by using existing state of the art technique to translate data in those models into RDF. This approach may be expensive, I was wondering if lightweight approaches which do not attempt to translate the original data, but instead create metadata that describe them using nanopublication would be more realistic and cost effective as a solution. Of course the granularity of retrieval in this case would be a whole dataset, but there are scenarios where this solution would be acceptable. I think that a discussion in these lines that clariy the options to the reader would be helpful, specially that linked data form only a small proportion of available scientific data.


Instruction: You are one of the reviewers, your task is to write a review for the article. You will be given the title of the article, the number of the round in which the article is located, and your order among the reviewers.

Input:
Title: DECENTRALIZED PROVENANCE-AWARE PUBLISHING WITH NANOPUBLICATIONS
Review round: 2
Reviewer: 3

Output:
Basic reporting: No comments
Experimental design: A new evaluation is added to strengthen evaluation.
Validity of the findings: Hypothesis has been adjusted according to the findings.
Additional comments: I am happy with the responses from the authors and would therefore recommend acceptance of its publication.


Instruction: You are one of the reviewers, your task is to write a review for the article. You will be given the title of the article, the number of the round in which the article is located, and your order among the reviewers.

Input:
Title: UNCERTAINTY MODELING PROCESS FOR SEMANTIC TECHNOLOGY
Review round: 1
Reviewer: 1

Output:
Basic reporting: 1) The paper claims (in the abstract) that little has been done for the engineering of probabilistic ontologies.  The statement is false based on industry, government, and the ETWURG.  If you check the references and remove your own, the most recent paper is 2010?  Clearly there is a mismatch between the paper preparation and the submission; or you choose not investigate the literature?
2) The use cases are too simple and not well vetted for applicability
3) The process should follow definitions in Figure 2 to make it easier for the reader to follow.
4) Math of uncertainty and the RV discussion is hard to discern how created, but is more associated with a discussion with BN.  Some effort could be coordinated between a BN representation for the uncertainty quantification, that then is described in an ontology. The ontology then supports the query of analysis.
Experimental design: Too limited to discern viability
Validity of the findings: Does not seem reproducible of complete to validate
Additional comments: GRAMMAR and CONCEPTS
INTRO
1. “Uncertainty is fundamental across a wide range of domains to which semantic technology is applied”
-- awkward sentence
2. “This increase in complexity has been a major obstacle to penetration of expressive probabilistic representations into real-world applications.”
- awkward; the complexity is not a reason what it has not been reported in real-world literature (that comes from reporting on products).
3. “the literature contains little guidance on how to model a probabilistic ontology.” Seriously?  Your references are 7-years old.
4. “use a probabilistic ontology in real world problems, the first step is creating and documenting the model (TBox).”
- what is the model – mathematical? Linguistic? Etc.
5. “t needs to be populated (A-Box).”  - What is A, what is T?
6. “possible to query the model given the specific situation provided by the data available.”
- OK, a “query” is a way to access a system, but here you assume it is human or a machine
http://www.britannica.com/technology/query-language

UNCERTAINTY MODELING
7. Three stages? How can you build the model, without the parameters?
8. Uncertainty types – see the URREF
9. “Disciplines” – that is the word. Analysis and Design are separate concepts. In most engineering applications, you design first and then do the analysis of the design against testing.  Not sure what the “unified process” is, but seems like it is myopically applied.
10. “in Figure 2, the requirements discipline (blue circle)” is labeled as Goals and not requirements. Do goals consist of queries and evidence?  Seems like a disconnect? The other circisl do have some association with the terms.
12. Figure 2, LPD not described

FRAUD
13. What are the fraud types?   All that is listed in “fake competition.” Where is a reference to a document that explores fraud?
14. “the prob ontology designed by experts transforms millions of items into dozens” – HOW?
15. 1. Goal: evidence – looks like a social network diagram
16. 2. Goal, query: evidence – question should be a “criminal history” extracted from a police database
17. RTM – you end with this, but it should be the start of the requirements section.
18. Figure 4 is good
19. Categories for development should follow Figure 2. Thus, there is no category “iimplementation” in Figure 2.
20. how does Figure 6 create the RV?
21. there is no example of the coordination of the math of the RV. For example “The Improper Procurement MFrag has the resident RV isImproperProcurement(procurement),” is not a RV.
22. If you have a LPD, you must have a GPD.  It seems that you should look at the efforts for distributed fusion that works with graphs for distribution fusion.
23. Can you provide a easy to follow RTM matrix? There are no matrices presented
24. “The purpose of importance analysis is to verify that selected parts of the model behave as intended.”
This does not seem to follow as importance sampling is a decision process.  You probable want to discuss “verification” that the model works as intended.
25. Too many references to Carvalho (2011) are required to learn from this presentation alone.
26. Need to highlight the scenario as a separate section as a use case. There is not enough detail to work through the examples as scenarios to follow the process. Numbers in Figure 9, 10 are not described in relation to the scenario.
27. The numbers for the scenarios are not compared to be able to learn from the examples. A combined table would be useful.

CONCLUSIONS
28. The quick look at the scenarios does not seem to convey the viability of the approach. More is needed to make the claim.
29. The second paragraph in the conclusions should go in the intro
30. “the next step” is part of future work.
31.”Future work” should be a discussion section before the conclusions.