Instruction: You are one of the reviewers, your task is to write a review for the article. You will be given the title of the article, the number of the round in which the article is located, and your order among the reviewers.

Input:
Title: PARALLELISATION OF EQUATION-BASED SIMULATION PROGRAMS ON HETEROGENEOUS COMPUTING SYSTEMS
Review round: 1
Reviewer: 1

Output:
Basic reporting: Overall, the quality of presentation and use of language was high.  The manuscript is well-written and well-structured.  The motivation for the work was clearly explained.  The diagrams are nicely made and used to good effect in explaining the work.  There was no  trouble following the flow of the presentation.

While I did not read the manuscript carefully with the intent of identifying all mistakes  in grammar, I found some minor uses of language that I suggest could be improved in the  following way:

In the abstract on lines 24 and 25 - the joining of OpenMP API and OpenCL framework to utilised is somewhat grammatically incorrect.  May I suggest breaking the sentence up into two sentences.  Perhaps something along the lines of "...*the* DAE Tools modeling software  in *the* C99 language using two APIs/frameworks for parallelism:  *The* OpenMP API is used for parallelization on general purpose processors, and the OpenCL framework is used for parallelization on streaming processors and heterogeneous systems."

30 - "is discussed" instead of just "discussed"

109 - "equation expressions" instead of "equations expressions" (no s on equation)

118 - "the" before "DAE Tools context"

123 - "as shown in the Results section" instead of "as it has been shown in the Results  section"

141 - comma after OpenCL kernels

165 - comma before which

576 - 583 - "a" in front of every "multi-core CPU"

576 - I believe it should read Cases 1 and 2 instead of Cases 1 and 3, judging from Table  1

578 - Likewise Cases 3 and 4 instead of Cases 2 and 4

584,etc. - replace "on" with "at" before every "clock frequency"

605 - This experiment is discretized using finite differences, so perhaps "grid points"  instead of "elements"

647 - Colon replaced by comma, as well as a comma after "linear systems"

649, 652, etc. - all cases of "speed-up" should be "speed-ups"

673 - comma before "each requiring"

673 - "resolution" instead of "the resolution"

674 - comma before while

690 - "outweigh" instead of "out-weight"

717 - the in front of each item, i.e. "the OpenMP API", "the OpenCL framework"

721 - a/an in front of each item, e.g. "a viscious fluid model"

733 - "the solution of linear systems", i.e. add "the"
Experimental design: 1:

The primary thing that I think needs some revision in this manuscript is the description  of the experiments.  First of all, not all of the details are given in the description.   For example, all of the equations are time-dependent, but the time interval over which the  solutions is evolved does not seem to be given.

Furthermore, the nature of DAE solver and linear solver are not described in the  Benchmarks section.  In the Results section, it eventually is stated that the linear solver is SuperLU, but I did not see any similar statement about the DAE solver.  This  means that the spatial discretization is described (finite-difference or finite-element), but the nature of the time stepping is not known.  Is BDF?  Runge-Kutta?  Is a particular  package being used?  If so, which solver inside the package is chosen?

Furthermore, it is not clear which parts of the system are running on the GPU vs. the CPU.  Clearly at least the equation evaluation is being done on the GPU, but what else?  From the discussion in the Results section and Discussion section, one can infer that it is  probably the case that the linear solver and DAE solver are running on the CPU.  In other  words, the problem is primarily run on the CPU and the evaluation of the equations is done  using the GPU as an "accelerator".  This is in contrast to a common approach in HPC where  the problem remains on the GPU for the length of the problem without fine-grain movement  of data between the CPU and GPU between components (e.g. between the DAE solver and linear  solver).  Each part of the system acts directly on the GPU data.  It would be helpful to  make that relationship clear as early as the Implementation section of the paper.  Under  what conditions is there transfer of data between the CPU and GPU?

In light of the fact that the equation evaluation is (probably) being done in an  accelerator style, there is a question of whether a copy is being done with the integrated  GPU (the Intel HD 530).  Since that device shares memory with the CPU, it is possible to  use a zero-copy buffer to avoid movement from a CPU-owned buffer to a GPU-owned buffer,  which avoids a copy to different locations in the same memory.  Clearly a copy is being  made out of necessity with the GTX 950M, but is one being done with the HD 530?

The performance of streaming architectures is highly influenced by the bandwidth of  memory.  It would be helpful to list the memory bandwidth for each of the architectures  described in the Benchmark section.  In particular, the higher bandwidth of the dedicated  GPU is probably the biggest reason for its superior performance over other cases.   Furthermore, it would also be helpful to list the bus bandwidth between the CPU memory and  the dedicated GPU memory, as this determines the cost of moving data between the CPU and  GPU.  Movement of data between memories is generally *the* highest overhead in heterogeneous systems.

In light of this, it would be helpful to list the final amount of data being moved back  and forth between the CPU and GPU in each benchmark in terms of bytes (or some other  direct measure of the data cost beyond requiring the reader to infer and calculate from  equation unknowns etc.).  The author has already made some effort in the Implementation section to describe the efficient use of memory in the design of the Compute Stack structure.  Some discussion about how that adds up in the final system would be helpful for showing off the design.

2:

The performance of the Compute Stack approach is evaluated relative to the performance of  the Evaluation Tree approach implemented in DAE Tools.  In other words, the comparisons  are being done between two implementations done by the same author in the same software  suite.  We see that the Compute Stack approach outcompetes the Evaluation Tree approach, so it's better than (the author's own) Evaluation Tree implementation, but how good is the Evaluation Tree implementation?  From the manuscript alone, we have no way of knowing how  good of a baseline the Evaluation Tree implementation is.  To put it more simply, the performance results are relative improvements over an unknown (absolute) baseline.  A reader sees that the Compute Stack GPU performance is considerably improved, but improved  over something that was good or bad?

Of course this is being somewhat pedantic, and likely the implementation of the original  Evaluation Tree is fine.  The author says up front that direct use of C/C++ and Fortran  are generally the fastest approach, and it's clear that some loss of performance for the  sake of convenience is acceptable here.  Furthermore, DAE Tools is being successfully used  in applications, so it seems likely that the original Evaluation Tree implementation is  not downright pathological.

Still, it would be very helpful to include something that grounds the performance in more  absolute terms.  If a reference can be made to another paper that compares performance  with other software or that otherwise discusses the performance of the software, that would resolve the matter.

In my opinion, a particularly nice approach would be to go so far as to compare the  performance to a direct C/C++ implementation.  For the test problems used in this paper, a  C/C++ or direct OpenCL implementation would be straight-forward, and it is likely that the author is using a package for the DAE solve (such as something like SUNDIALS) that can accept such direct implementations already.  Of course, the direct implementation would probably be somewhat faster, but everyone expects and accepts that one is making a trade-off for convenience here.  But it would clear up the issue that one is showing a speedup over something that is ultimately unknown, and it would also show that the cost of the trade-off for convenience is acceptable as a bonus.

3:

What versions of the compilers, APIs, and frameworks are being used?
Validity of the findings: In the Discussion section, there is some speculation about the underlying causes for the  performance differences.  For example, in the first paragraph the author points out that  Evaluation Trees use a large number of non-contiguous blocks of memory that cannot fit  into cache, that a large number of virtual function evaluations are required, and that  there could be pipelining of instructions and hiding of memory latency in the Compute  Stack case.  Everything that the author states is plausible, but it is important to make  very clear that all the suggestions are plausible but not proven.  In general, performance  analysis of codes is very tricky, and attributing causes in a definitive way is very  difficult.  I do not think a fine-grained attribution of causes is needed for  demonstrating the value of this work in any case, so it is best to mostly avoid it.  I do not think the Discussion section would be any less for having most of the speculation about causes simply removed.  If it is kept, it should be stated explicitly that the attribution is hypothetical.

However, I take this opportunity to emphasize again that further detail about the movement  of data between the CPU and GPU in the Implementation section would make the nature of the  system more clear.  The author highlights this particular cost in the Discussion section,  noting appropriately that such a cost is a probable cause but not a proven one.  However,  this is the first place in the manuscript where this aspect of the system is enunciated and it is assumed that the reader understood this all along.
Additional comments: Reviewers were instructed that the paper should be evaluated strictly on scientific grounds and not on impact, but I do believe that with some modest refinement this paper will make a valuable contribution to the literature.


Instruction: You are one of the reviewers, your task is to write a review for the article. You will be given the title of the article, the number of the round in which the article is located, and your order among the reviewers.

Input:
Title: PARALLELISATION OF EQUATION-BASED SIMULATION PROGRAMS ON HETEROGENEOUS COMPUTING SYSTEMS
Review round: 1
Reviewer: 2

Output:
Basic reporting: The English is clear and the explanation almost unambiguous; the state of the art is well presented. The goal and structure of the article are both good, although the Results section could be improved.

In particular, there is some confusion between the "types of computational setups that were benchmarked" (a list of 5 items) compared to the Table 1 named "computational setups", which has 12 items. I suggest to stick to the 12-items description, by extending the list in the text and by integrating the information about the proportions of work done. The same 12-items convention should be used in the Conclusions. Moreover, case #5 (i.e., CPU + OpenCL) is named "CPU" which is absolutely confusing, since it resembles case #1 (i.e., simple sequential CPU execution).  A different naming is strongly suggested (e.g., CPU + OpenCL).

A few minors:
- Abstract: "It is found that the sequential Compute Stack approach is up to one order of magnitude faster than the old approach for evaluation of equations". It is not clear what "old" means in this context. The same comment is valid for the Discussion ("The Compute Stack approach brings significant performance improvements compared to the old one").
- Page 2/26: Procesing -> Processing
- Page 17/26: "u is the velocity field" <- symbol "u" is a vector and should be bold (see Equation 6).
- All execution time tables: please report the unit (seconds?)
- Bibliography: openmodelica -> OpenModelica. Also add: "46th Conference on Simulation and Modelling of the Scandinavian Simulation Society (SIMS2005), Trondheim, Norway"
Experimental design: The paper focuses on the comparison of DAE programs implemented using different computing architectures (i.e., CPU and GPU), including heterogeneous solutions (e.g., CPU+GPU). Moreover, the author compares two alternative formalisms for equations encoding, i.e., classic evaluation trees (ET) and computing stacks (CS).  The research falls within the aims and scope of PeerJ Computer Science, and it is interesting for researchers involved in the development of high-performance scientific simulations.
Validity of the findings: According to the author, CSs are more adeguate than ETs on streaming processors since the evaluation of ETs "typically requires recursive function calls which are not supported on streaming processors". However, this sentence is true only in the case of OpenCL code, because Nvidia's CUDA supports recursive calls since the Fermi architecture. The author should clarify/correct this passage since CUDA could, in principle, execute ETs on the GPU.

CSs seem to be more efficient (as shown in Table 4) than ETs. Moreover, GPUs (even a modest integrated Intel) provide a decent speedup with respect to multithreaded code. However, the result with heterogeneous computation (i.e., GPU + CPU) is actually surprising, because it is supposed to be the fastest option since the CPU is generally idle during kernels' execution. The speed-up should be at least similar, but it is actually one half in the results shown in Table 7. This is an unexpected and interesting result that should be investigated better. The author states that "[t]he probable reason is that the time required to copy input arrays to multiple devices, start multiple OpenCL kernels, and copy the results back out-weight the higher theoretical computing capacity of multiple devices". However, the memory copy performed with simple GPU-based computation (i.e., case 7) should be more intensive, since *all* data is transferred back and forth over the (slow) PCI-ex bus. I suggest to investigate the issues of heterogeneous computation more carefully before making such a claim.

The speedup provided by GPU with respect to single-threaded sequential execution is puzzling as well. Even though it is true that the (theoretical) peak power is 33.6 and 36.56 GFlops for the CPU and GPU (respectively), the former can be reached only by using intenstive multi-threading. Thus, the 640 units of the GPU should provide a better speedup and instructions throughput than CPU-based single threaded execution. Maybe some optimization in the source code (e.g., by leveraging cached __constant memory for non-mutable data) could improve the results.
Additional comments: No additional comments


Instruction: You are one of the reviewers, your task is to write a review for the article. You will be given the title of the article, the number of the round in which the article is located, and your order among the reviewers.

Input:
Title: PARALLELISATION OF EQUATION-BASED SIMULATION PROGRAMS ON HETEROGENEOUS COMPUTING SYSTEMS
Review round: 1
Reviewer: 3

Output:
Basic reporting: One of my biggest concerns is that the scope of "equation-based simulations" is simultaneously quite large and also quite vague. Do you mean "differential equations", or even more specifically "partial differential equations" or "ordinary differential equations"? The terminology used does not seem like a standard one to me, since all simulations are "equation-based" in some way. Similarly, the very first sentence seems like a big understatement—are there any engineering problems that aren't described using a system of non-linear (partial)-differential and/or algebraic equations?
Experimental design: The Benchmark 1 section references "Appendix C", but I cannot find any appendices.

I have a few questions about the method:
a) A single float "value" field is used in adComputeStackItem_t is used for constants; does this limit the complexity of operations? Or would operations with more than one constant just be broken into multiple structs?

b) To clarify, parallelizing the evaluation of an adComputeStackEvaluator_t object uses a single CPU thread to calculate a single equation? Does this also extend to the GPU parallelization, where a single GPU thread handles a single equation?

c) How are equations set up using the Compute Stack approach? Does something automatically generate the code shown, or is this done manually?

Regarding how the results are shown/discussed, I think it is a little confusing to choose the "CPU Seq" benchmark as the baseline for showing speedup etc.; instead, wouldn't one of the cases using the Evaluation Tree approach be a better baseline, since that is the prior method against which you are comparing? This issue also relates to how the methods are presented; in my first read, it was not 100% clear that the Compute Stack method is new and the Evaluation Tree approach is the existing approach against with CS is being compared. So, some more clarity on the objectives of the study would be helpful.

In the Benchmarks section, I think that each benchmarks is missing quite a bit of detail on how the governing equations are actually solved. In particular, benchmarks 2 and 3 evidently use the finite element method, but no additional detail is given. Furthermore, there is not enough discussion of the other components required to perform the benchmark studies, such as the linear solver (which appears in Figure 6 prior to its first mention).

(Minor comment: the description of the Method of Manufactured Solutions technique is a bit too strong; it is a good method for creating an analytical solution for verification purposes, but I don't think you can say it ensures "all coding mistakes which prevent the equations from being solved correctly have been identified"...)
Validity of the findings: While I don't doubt the validity of the findings, and think that the work is useful due to the improvement in performance shown, the manuscript does not quite explain why the method behaves as observed. For example, why do the performance trends shown happen?

Furthermore, how well does this method actually parallelize on the systems considered (thinking about, e.g., thread divergence on GPUs, vectorization efficiency on the SIMD devices)? The performance offered on the GPU benchmarks is not actually that high compared with the CPU-based ones, so I wonder how well this approach is actually suited for GPU execution. Some analysis and discussion of these topics would strengthen the manuscript from its current form, which mostly just shows the results of the method without much critical analysis. (There are some speculative comments in the Discussion section on page 23, but these aren't supported by any analysis.)
Additional comments: I appreciate the availability of the benchmark code listed in the Appendix, but for longevity purposes I suggest that it be archived using a more permanent service (e.g., Zenodo) and cited in the manuscript.

Furthermore, it isn't clear whether the DAE Tools software (which the research relies on) is openly available. At minimum it should be cited appropriately in the manuscript, including authors, version used, location where accessible; in the best case, the version used should be archived as described above for maximum reproducibility of the study.

Minor comment: the colors used for Figure 6 could be chosen a little better for accessibility (i.e., colorblindness).


Instruction: You are one of the reviewers, your task is to write a review for the article. You will be given the title of the article, the number of the round in which the article is located, and your order among the reviewers.

Input:
Title: PARALLELISATION OF EQUATION-BASED SIMULATION PROGRAMS ON HETEROGENEOUS COMPUTING SYSTEMS
Review round: 2
Reviewer: 1

Output:
Basic reporting: As with the original submission, the overall presentation of this submission is excellent, the structure is well-organized and clear, and the use of English is of high quality.

I only make the following minor comment.  English rules about whether to use "the" with nouns, particularly proper nouns that are capitalized, are somewhat murky.  However, I believe the most common convention is to refer to cases, figures, and tables, etc. without "the" but using a capital letter, e.g. "Case 1" as opposed to "the Case 1".  In particular, I believe the most common convention for "Case" follows the same rules that the manuscript already uses with tables, such as "in Table 2" on line 745.

Whether a particular usage could be considered incorrect is perhaps debatable, but certainly a convention should be applied consistently throughout a manuscript, at least as long as doing so does not hurt clarity or simplicity.  It is a minor issue, but some lines use "the" with Case(s), such as lines 619 and 622, where as some do not, such as line 762.  Furthermore, some lines do not capitalize "case", such as lines 751 and 754.

Some situations are tricky, such as "to the base Case 1" in line 767.  The most technically correct usage would probably be "to the base case, Case 1,...", but that seems unnecessarily obtuse for the sake of rule following, and it seems fine to leave such cases as is.
Experimental design: The author has done a very thorough job of revising the experimental design and its presentation.  All of my questions about the first submission have been well addressed.  In particular, taking the extra effort to compare with plain C++ is much appreciated.  In my opinion, the current state of the manuscript is quite satisfactory.
Validity of the findings: I sympathize that the author had to balance somewhat competing demands from the reviewers about attribution of causes.  In my opinion, the current presentation is well balanced in that regard.  Overall, all of my concerns with the first manuscript have been well addressed and there are no further ones.
Additional comments: No additional comments


Instruction: You are one of the reviewers, your task is to write a review for the article. You will be given the title of the article, the number of the round in which the article is located, and your order among the reviewers.

Input:
Title: PARALLELISATION OF EQUATION-BASED SIMULATION PROGRAMS ON HETEROGENEOUS COMPUTING SYSTEMS
Review round: 2
Reviewer: 2

Output:
Basic reporting: The paper is now clear, self-contained, well written and fluid to read.
Experimental design: The paper is in scope with the journal. The research question is relevant for GPU-powered simulation based on ODEs.
Validity of the findings: The concerns expressed in the previous review have been addressed; however, the new analyses introduce new complexity and raised new doubts. Specifically, the author should provide a minimal explanation why, in Case #7, the actual speed-up achieved during the evaluation of residuals is 11.85x while the theoretical speed-up is just 8.52x. In general, performances higher than the theoretical peak processing power are unexpected and deserve additional analysis
Additional comments: No additional comments


Instruction: You are one of the reviewers, your task is to write a review for the article. You will be given the title of the article, the number of the round in which the article is located, and your order among the reviewers.

Input:
Title: PARALLELISATION OF EQUATION-BASED SIMULATION PROGRAMS ON HETEROGENEOUS COMPUTING SYSTEMS
Review round: 2
Reviewer: 3

Output:
Basic reporting: no comment
Experimental design: no comment
Validity of the findings: no comment
Additional comments: Overall the author has satisfactorily addressed my original comments.


Instruction: You are one of the reviewers, your task is to write a review for the article. You will be given the title of the article, the number of the round in which the article is located, and your order among the reviewers.

Input:
Title: VERIFIABILITY IN COMPUTER-AIDED RESEARCH: THE ROLE OF DIGITAL SCIENTIFIC NOTATIONS AT THE HUMAN-COMPUTER INTERFACE
Review round: 1
Reviewer: 1

Output:
Basic reporting: The paper is well written, but I have concerns about overall structure and about the lack of evidence to back up some of its statements - please see the general notes below.
Experimental design: Not applicable.
Validity of the findings: Not applicable.
Additional comments: Thank you for giving me the opportunity to review this paper - I enjoyed reading it, and believe that the work is both interesting and valuable, but have three reservations about it in its present form:

1. Even upon second reading, I felt that two papers had been combined to make one: a general discussion of reproducibility (or its lack) in science, and a description of the design of a particular literate programming tool.  I recognize that the first motivates the second, but I think the audience for the two halves of the paper is likely different, and recommend splitting.

2. Claims or implications are made throughout the paper about the scale and severity of the reproducibility crisis in scientific computing and the usability (or lack thereof) of various software tools, but these are only backed up by specific instances or anecdotes.  For example, the author mentions several cases of papers being found to be in error because of computational mistakes, but we simply do not have any solid (quantitative) understanding of how widespread this is - as I note in comments, fewer than 50 of the 5 million papers published 1990-2000 have, to my knowledge, subsequently been withdrawn because of computational errors, which puts our *proven* error rate at 1 in 100,000.  Like the author, I believe that the actual rate is much higher than this, but I have no proof, and am increasingly wary of repeating dire warnings that I cannot substantiate while simultaneously telling people that *their* work should be more reproducible.

3. The second part of the paper (describing the design of Leibniz) uses concepts from programming language analysis and design that will be unfamiliar to most people with backgrounds in other sciences.  This is not to say that those ideas should be removed - they are as necessary to understanding the work as a knowledge of partial differential equations is to understanding a paper about fluid mechanics - but in places I felt that having worked on this tool for so long, the author may have lost sight of how much background knowledge the description assumes or requires.

I hope these comments and the others embedded in the PDF are helpful, and I would be happy to discuss them directly with the author if so desired.

Thanks,
Greg Wilson


Instruction: You are one of the reviewers, your task is to write a review for the article. You will be given the title of the article, the number of the round in which the article is located, and your order among the reviewers.

Input:
Title: VERIFIABILITY IN COMPUTER-AIDED RESEARCH: THE ROLE OF DIGITAL SCIENTIFIC NOTATIONS AT THE HUMAN-COMPUTER INTERFACE
Review round: 1
Reviewer: 2

Output:
Basic reporting: The basic reporting meets requirements in terms of language, citation, figures and scientific notation.
Experimental design: The experimental design is appropriate, although the introduction and background is fairly lengthy. Methods are described with sufficient detail.
Validity of the findings: See main review.
Additional comments: The article by Hinsen focuses on a needs assessment for understanding how users can make mistakes and have difficulty with reproducibility due to the complexity and black-box nature of scientific software. After a fairly lengthy discussion of issues encountered over the authors career, the paper identifies the key role of digital scientific notations at the human-computer interface. A proof of concept implementation of Leibniz as a language for specifying digital scientific notation from models as math equations is one part of a solution. It, however does not entirely solve problems due to the nature of non-determinism in most implementations, needs for a means to specify and understand constants, accuracy and means of implementing the math equations digitally. However, the overall discussion and issues raised are important and the prototype solution via a purposed domain specific language and documentation/code could be a worthwhile path forward. General adoption of such an approach, given the significantly complexity of long term and evolving community scientific/research codes that are growing, highly optimized, and complicated in terms of workflow is unlikely. The scientific validity (with some clarifications noted) appears sufficient and the article appears suitable to join the scholarly literature.

Page 1, lines 38-39: It is stated that computations are fully deterministic. This is typically not true, especially in parallel, due to differing order of operations. Codes can be made fully deterministic, for example with fixed point precision on GPUs, however most often they are not implemented (when optimized) to be fully deterministic. Moreover, reproducibility does not necessarily require determinism.

Page 2, lines 107-108: To this reviewer’s knowledge, even back in 1997, AMBER energies were not dependent in the order of atoms in the coordinate file since they were mapped back to the residue definitions. It may be possible that different order of atoms in the residue definitions could have led to different energies, but this needs to be clarified, i.e. exactly what input file is the author referring to. The issues with implementing force fields is significantly more complicated than simply atom-ordering and there are clear issues about choice of algorithms to implement that may be tricky to figure out without reading the code. For example, the choice of improper angles assigned in AMBER does depend on the alphabetic ordering of the atom types which means if you change a type from CA to ZA that may change the impropers assigned. Moreover, there are issues today with accuracy and speed where many shortcuts are made in the name of performance that may lead to different forces and ultimately results. Even implementation choices such as the value of pi, or the conversion factor from charge to kcal/mol; implementations of cutoffs with or without buffers and automated pairlist updates; use of SHAKE on all bonds (LINCS in Gromacs) versus SHAKE only on hydrogens. I would further note that AMBER has changed in significant ways since 1997 (> 20 years ago). Some issues with energy comparisons among programs is described in JCAMD 31, 147 (2017).

Page 3, lines 125-126: Small effect on energy but due to chaos small differences can be strongly amplified – Do you have evidence for this? Although CHARMM and AMBER by default do not give equivalent energies and forces (although they can be made to get accuracy on forces to 10**-6 if constants are changed), they can give equivalent results on converged conformational sampling in the limits of sufficient sampling. Additionally, equilibrium properties can be converged in MD simulations starting from vastly different initial conditions given sufficient sampling.

Force field errors can occur when re-implemented quite easily leading to anomalous results (for example see recent work of Hayatshahi et al. on dinucleotides JPCB 121, 451 (2017) who could not reproduce the work of Brown et al. JCTC 11, 2315 (2015)).

Page 4, line 163: What is the evidence that “Peer review tends to be shallow”? Provide citation or consider softening the statement – for example, “Given time constraints on reviewers and the complexity of the modern scientific workflow and software that often may involve very large-scale calculations, I would speculate that in some cases initial peer review can be shallow.” Evidence that few reviewers re-do experiments or computations?

In discussion of software only as binary, software not published, you may want to note that in the 90’s publishing of force field parameters became requirement for publication, yet some commercial force fields (Schrodinger) are not published and are proprietary (not available outside of their codes).


Instruction: You are one of the reviewers, your task is to write a review for the article. You will be given the title of the article, the number of the round in which the article is located, and your order among the reviewers.

Input:
Title: VERIFIABILITY IN COMPUTER-AIDED RESEARCH: THE ROLE OF DIGITAL SCIENTIFIC NOTATIONS AT THE HUMAN-COMPUTER INTERFACE
Review round: 2
Reviewer: 1

Output:
Basic reporting: See "general comments".
Experimental design: See "general comments".
Validity of the findings: See "general comments".
Additional comments: Thank you for giving me the opportunity to review the changes to this paper - I am grateful to the author for addressing most of my original comments.  I still find the transition from the general discussion of verifiability to the specific solution in Leibniz awkward, but I recognize that this is a case of "I wouldn't have written it that way", which isn't a legitimate criticism.  The use of the mass-and-spring example throughout is welcome, as is placing the technical discussion of Leibniz in one section; I feel the AMBER anecdote (Section 2.1) could be shortened, but again, that is stylistic rather than substantive.

Recommendation: the paper should be accepted without further changes (other than replacing "literal programming" with "literate programming" on line 373).


Instruction: You are one of the reviewers, your task is to write a review for the article. You will be given the title of the article, the number of the round in which the article is located, and your order among the reviewers.

Input:
Title: VERIFIABILITY IN COMPUTER-AIDED RESEARCH: THE ROLE OF DIGITAL SCIENTIFIC NOTATIONS AT THE HUMAN-COMPUTER INTERFACE
Review round: 2
Reviewer: 2

Output:
Basic reporting: No comment.
Experimental design: The experimental design is appropriate, although the introduction, motivation and background are rather lengthy.
Validity of the findings: The findings appear appropriate.
Additional comments: The revision strongly considered and attempted to address concerns raised in previous review. the revised manuscript is improved, although the intro/motivation/background are a little lengthy.

Line 121 - "and the decision might well have been taken by an inexperienced graduate student" is pure speculation and unnecessary to mention. I would suggest omitting this.

Line 123 - "the feature wasn't documented anywhere else than in the source code of a piece of software, which was never peer reviewed at all." Again, speculation without evidence. Many different people have effectively "peer reviewed" the AMBER software. I would suggest removing the phrase "which was never peer reviewed at all".

Line 126-127 "To the best of my knowledge, no paper and no software documentation mentions this behavior" -- omit. It is true that it is non-trivial to figure this out without reading the AMBER code, but there is documentation out there which can be found on Google.

https://books.google.com/books?id=mPgpMig3tx0C&pg=PA77&lpg=PA77&dq=amber+atom+type+ordering+improper&source=bl&ots=h0EegD4GBZ&sig=KQ2jDNV0e8I6-GpOSAvKOUE9Kaw&hl=en&sa=X&ved=0ahUKEwiK89PYitTbAhUSiIMKHY4aCJ4Q6AEITTAG#v=onepage&q=amber%20atom%20type%20ordering%20improper&f=false

From: A Practical Introduction to the Simulation of Molecular Systems
By Martin J. Field (1999).

Also blog posts on OpenMM: https://github.com/pandegroup/openmm/issues/220

Line 129-131 "In fact, I am not even sure that all software implementing AMBER handles this the same way..." See: "Lessons learned from comparing molecular dynamics engines on the SAMPL5 dataset."
Shirts MR, Klein C, Swails JM, Yin J, Gilson MK, Mobley DL, Case DA, Zhong ED.
J Comput Aided Mol Des. 2017 Jan;31(1):147-161. doi: 10.1007/s10822-016-9977-1.

Like 138 - "It would clearly be less effort for everybody to simply fix the force field definition". The definition is fixed / specific, just poorly documented.


Instruction: You are one of the reviewers, your task is to write a review for the article. You will be given the title of the article, the number of the round in which the article is located, and your order among the reviewers.

Input:
Title: COMBINING ACTIVE LEARNING SUGGESTIONS
Review round: 1
Reviewer: 1

Output:
Basic reporting: Generally speaking, the article is well written and pleasant to read.

A few comments on the structure and on the reporting:

- It might be more natural to place the paragraph on information-density weighting (lines 313-322) in Section 2.1.
- Using rank aggregation to combine multiple AL strategies has been proposed before [1], albeit in the context of active multi-task learning. I believe this should be acknowledged.
- The above reference can be found, e.g., in Burr Settles' book [2], which might also be a useful addition to the references.
- Table 4: "Max Performance" -> the metric is not specified.
- Figures 4 and 5: It would be helpful to indicate which quantiles are represented by the boxes and the whiskers.

[1] R. Reichart et al., Multi-Task Active Learning for Linguistic Annotations, ACL 2008.
[2] B. Settles, Active Learning, 2012, Morgan & Claypool
Experimental design: The research question that the authors seek to address is important and meaningful. Their approach---analyzing a large number of methods on a large number of datasets---is a priori interesting. The experimental methods are well described, and the code is easy to read.

However, I believe that there are important issues in the experimental design, such that it is not possible to gain a clear, objective understanding of the performance of the AL strategies. Hence, in its current state, the paper fails to properly address the research question.

1. The strength (eq. 25) is upper-bounded by 1, assuming that the discrete integrals are positive. However, looking at Fig. 4 and 5, we see a number of instances where the magnitude of the strength is much larger than 1, positively or negatively. This is due to the fact that $m(\max)$ in (25) is the performance achieved using all labeled data, which, in some cases, is *not* the best possible performance. In these cases, the discrete integral can be 0 or negative, and the strength might not even be well-defined. This seriously impacts the interpretability of the results. The following solutions might be explored:

- $m(\max)$ could be redefined as the best performance in the given run (such that by construction the discrete integrals will be positive), or
- multiple runs could be averaged before computing the strength, in order to decrease the variability in the performance, or
- the authors might rely less on strength measurements and more on the learning curves (Fig. 6 and 7), which should then be significantly improved, e.g., by averaging more runs (>=50) and by showing more strategies, or
- other performance measures might be considered, as they might prove to be less sensitive to the "nonpositive integral" problem.

2. There is no meaningful comparison with prior work. For example, I suggest considering the datasets & classifier of [3] and including the results of this paper in the plots, so that it is possible to get a clear comparison between the two.

3. The results seem to depend a lot on the choice of hyperparameters. These dependencies need to be explored more systematically. A few examples:

- Thompson sampling: the authors state that it favors certain heuristics a lot more than others. But this is likely going to be very different if $\tau^2_r$ and the initial $\mu_r$ are set differently.
- Poor performance of information-density weighted strategies: this obviously depends a lot on $\gamma$.
- Is there an effect of the size of the candidate pool?
- Regularization strength in Eq. 22: the choice of $C = 1000$ seems somewhat arbitrary and needs to be better explained.

[3] Y. Baram et al., Online Choice of Active Learning Algorithms, 2004, JMLR.
Validity of the findings: 1. The authors claim that "with the small and medium-sized datasets, the effect of active-learning is minimal". Do the authors suggest that AL should only be used if one has access to a large unlabeled pool? Or that the number of AL iterations has to be sufficiently large? I believe this might be more of a measurement problem: with small datasets, there are fewer iterations, the test set is smaller, and the measured strength is noisier---in conclusion, this might just be an artefact of the experimental procedure. In any case. this should be clarified, perhaps by running additional experiences (e.g., by looking at a small sample of a large dataset, sample whose size matches that of the small datasets).

2. Line 353: "there is no statistical difference between the corresponding heuristic/combiner and passive learning". Clearly, looking e.g. at the median in Fig. 4., there seems to be a difference. Do the authors mean that the difference is not statistically significant?
Additional comments: Small comments:

- You refer multiple times to the "optimal heuristic". I believe "best heuristic" or "heuristic that works best" is more appropriate.
- Line 59: "test example" -> maybe "candidate example"?
- Eq. 5: why the $x$ in $r(x, h)$? The $x$ on the RHS is in a different scope (it is argmax'ed over). Maybe it should be $r(\mathcal{U}, h)$ instead.
- Sec 3.1. A general reference to multi-armed bandits would be welcome after the first sentence. Furthermore, I am not sure that the typical objective of MABs is to "minimize the number of pulls" (lines 132-133).
- Figure 1: why the dotted line? What is the difference with plain lines?
- Eq. 12: the way the function is defined implies that the bandit is able to see a reward for every arm, but that is not the case, I presume? I think the domain of the function should be revised.
- Eq 13: why $| v_r$ on the LHS?
- Algorithm 2, last line: "update the parameters of $B$" -> should be small case $b$.
- Line 201: "consider the KL-divergence" -> between which distributions?
- Line 203: "Algorithm" -> Algorithms
- Line 226: "we could only use one suggestion from one particular heuristic" -> from what I understand, this is not the case for EXP4.
- Line 236: $k(s(x; h))$ -> s(x:h) is a score (i.e., a real number), this is inconsistent with the definition of $k$ in Eq. 20.
- Line 256: Why is it necessary to rescale the ranks? I don't see why this would change the aggregation.
- Line 292: "with an L2 loss" -> I suspect you mean "with an L2 regularizer".
- Eq. 22: $w^T \theta$ should probably be $\theta^T \theta$.
- Line 316: "vetor" -> vector
- Line 393: "Thompson sampling requires" -> assumes
- Line 394 "This can be hard to satisfy" -> This assumption is unrealistic.


Instruction: You are one of the reviewers, your task is to write a review for the article. You will be given the title of the article, the number of the round in which the article is located, and your order among the reviewers.

Input:
Title: COMBINING ACTIVE LEARNING SUGGESTIONS
Review round: 1
Reviewer: 2

Output:
Basic reporting: - The introduction should be improved, highlighting better the motivations and the main contributions of the work.

- Generally speaking, the writing of the paper is good.

- In the session “related work”, the authors should explain better the reasons to only analyze and describe the Uncertainty Sampling and Version Space Reduction strategies. The authors wrote that the other heuristics (query strategies) are very computationally expensive, but they should explain further this situation.

- “Out of the five heuristics discussed, how do we know which one is the most optimal one?” → This question is quite interesting, but I think that the author should consider to rewrite it. According to the Non-Free-lunch theorem, that can be applied to active learning too, there is not a method that performs the best for all possible problems. Maybe, the question should be focused to encounter a method with a “optimal” performance for a particular set of problems.

- “There have been some attempts to combine active learning suggestions in the literature. Baram et al. (2004) used the EXP4 multi-armed bandit algorithm to automate the selection process. Hsu and Lin (2015) studied an improved version, EXP4.P, along with importance weighting to estimate the rewards using only the training set. This paper empirically compares the following four algorithms: Thompson sampling, OC-UCB, kl-UCB, and EXP3++.”

The authors should not restrict the state-of-the-art description to the single-label classification problem. For instance, in the following reference was proposed a method to average several rankings and obtain a consensus for selecting the next query in the context of multi-label learning.

Oscar Reyes, Carlos Morell, Sebastián Ventura, Effective active learning strategy for multi-label learning, Neurocomputing, Volume 273, 2018, Pages 494-508.

- The authors should improve the explanation of the proposed method.

- Does the proposed method fulfill the Condorcet criterion?
Experimental design: - It is not explained how the authors obtained probabilities from the logistic regression base classifier.

- What were the criteria to select the 11 datasets used in the experimental study?. On the other hand, the authors should increase the number of datasets in order to make the study more reliable.
Validity of the findings: - My main concern about this work is that the authors assumed that only one sample is selected in every active learning iteration. However, nowadays this scenario is much less used since the selection of several samples at the same time is preferred in the majority of real-world applications. To select a set of queries in each active learning iteration you can follow either a myopic active learning strategy or a batch-mode active learning one. There are main differences between these two kinds of strategies. The authors should explain this situation in the paper, and portrait some tracks explaining how to adapt the proposed method to this type of scenarios.

- The discussion and conclusions are well stated, they are limited to the results obtained.
Additional comments: Generally speaking, I consider that this work is interesting, but it should be significantly improved before it could be considered for publication. I suggest that authors review the work attending the comments made by the reviewer.


Instruction: You are one of the reviewers, your task is to write a review for the article. You will be given the title of the article, the number of the round in which the article is located, and your order among the reviewers.

Input:
Title: COMBINING ACTIVE LEARNING SUGGESTIONS
Review round: 2
Reviewer: 1

Output:
Basic reporting: The authors have addressed successfully the most of the comments raised by the reviewers. Two minor comments that should be addressed before publishing the paper.

1- “More sophisticated batch-mode active learning heuristics have been proposed, for example by using cluster analysis (Xu et al., 2007) or by maximizing the diversity in a batch (Brinker, 2003). How to aggregate suggestions from these heuristics is an interesting problem for future work.”

There are other interesting and recent batch-mode appoaches that the authors should consider to mention, for example the multi-objective approach proposed in:

Reyes, O., & Ventura, S. (2018). Evolutionary strategy to perform batch-mode active learning on multi-label data. ACM Transactions on Intelligent Systems and Technology (TIST), 9(4), 46.

2- The first reference has a lot of authors, maybe using the first author name followed by “et al” could be ok..
Experimental design: The experimental study is ok.
Validity of the findings: The description of the results and the discussion are ok.
Additional comments: No additional comments


Instruction: You are one of the reviewers, your task is to write a review for the article. You will be given the title of the article, the number of the round in which the article is located, and your order among the reviewers.

Input:
Title: COMPARISON AND BENCHMARK OF NAME-TO-GENDER INFERENCE SERVICES
Review round: 1
Reviewer: 1

Output:
Basic reporting: This paper describes an evaluation of available methods (mostly web APIs) that allow to infer the gender of a person based on its name.
The paper is overall written well and can be easily understood. Related work is described sufficiently (although given my background I cannot guarantee completeness). The paper structure is adequate. Raw data and source code for the paper is available and makes the impression to be complete.

Personally, I find it somewhat strange to cite the address of the raw data of a paper as a regular citation.
Experimental design: The paper pursues a clear and well defined goal. The outcome of the paper can be helpful to researchers in various academic disciplines.

The paper is currently missing a statistical analysis of the findings, i.e., it is unclear which differences are statistically significant. I suggest to add the respective tests, see e.g., "Japkowicz, Nathalie, and Mohak Shah. Evaluating learning algorithms: a classification perspective. Cambridge University Press, 2011."
Validity of the findings: The discussion should go into more detail with respect to bias potentially be implied by data used for the study (e.g., what are the supposed origins of the names, what is the share of Asian/African names, etc...). Looking at the raw data, I have a loose assumption that parts of the dataset (zbmath) are focused on western names. Indeed, it would be also interesting to split the analysis into the different data sources and report results. That would give the reader an impression how strongly results vary for different (sub-) datasets.

The setting of Benchmark 2 is not completely clear to me. Is it "at most 5% of all names" (as in the paper, line 400) or "at most 5% of all classified names"? I would like this Benchmark to be described in more detail.

For the setting in Benchmark 3, reporting the RoC curve would be the appropriate measure to use in my opinion. I strongly suggest to add that to the paper. See: "Bradley, Andrew P. "The use of the area under the ROC curve in the evaluation of machine learning algorithms." Pattern recognition 30.7 (1997): 1145-1159."

Table 4, Table 5, and Table 6 contain very little information. I would suggest to just report mean _and standard deviation_ of the iterations.
Additional comments: Figure 1 and its meaning/relevance should be described in more detail.

Tables use German instead of English floating point separators (i.e. "0,002" instead of "0.002")


Instruction: You are one of the reviewers, your task is to write a review for the article. You will be given the title of the article, the number of the round in which the article is located, and your order among the reviewers.

Input:
Title: COMPARISON AND BENCHMARK OF NAME-TO-GENDER INFERENCE SERVICES
Review round: 1
Reviewer: 2

Output:
Basic reporting: This paper represents a very interesting and worthy contribution to the field of automatic gender inference. Written English quality is consistent and of high quality, and both the paper and the overall argument are well-structured. Its problematization within the context of academic publishing and bibliometrics is both extensive and relatively up-to-date. However, all publications cited in the references section strictly pertain to automatic gender inference; given the importance and sophistication of the performance evaluation metrics and procedures used in this research, reference to seminal works pertaining to predictive analytics and confusion matrices would have been most welcome (at least some of them must have been consulted in doing this research). Finally, both analysis and discussions are directly related to the objective and problematics of the paper.
Experimental design: This article fits nicely within the aims and scope of this journal. The investigation procedure is well described and could easily be replicated. The contribution of this research to the existing body of knowledge is also well stated. In sum, nothing negative to report regarding experimental design.
Validity of the findings: Interpretation of the results is prudent, yet unambiguous. The authors have done a nice work collecting, compiling, and cleaning all the data necessary for the project. Two points need to be emphasized, however. In the data collection phase, the authors used nltk's Named Entity Recognizer to identify names in bibliographic records drawn from WoS. We know from experience that this algorithm has non-negligible efficiency and accuracy problems, especially in comparison to other, freely available alternatives such as Stanford's CoreNLP package. A few comments thereon would be desirable. Also, while the authors do mention the time- and country-dependency of automatic gender detection based on first names, the impact of these characteristics on the performance of the algorithms they evaluate as well as on the significance and scope of their research seems somewhat downplayed. To give one example (surely known by the authors), China represents the second biggest country in the world in terms of research output, but existing gender detection algorithms based on first names often perform poorly when it comes to disambiguating the gender of researchers of Asian origin. The fact that this isn't addressed in the evaluation procedure certainly affects the scope and impact of the research results, and the absence of any thorough discussion thereof reflects somewhat badly on the paper.
Additional comments: No comments.


Instruction: You are one of the reviewers, your task is to write a review for the article. You will be given the title of the article, the number of the round in which the article is located, and your order among the reviewers.

Input:
Title: ENHANCING DISCOVERY IN SPATIAL DATA INFRASTRUCTURES USING A SEARCH ENGINE
Review round: 1
Reviewer: 1

Output:
Basic reporting: The article is clearly written but has some unclearness related to the scope and structure of the content.
Experimental design: The proposed solution seems to be relevant and addresses a significant criticality of the distributed SDIs, but the structure of the article makes its main aim unclear.
Validity of the findings: The architecture and solutions described are solid and of importance in the field of SDIs but conclusions suffer from the not clear focus of the article.
Additional comments: The proposed solution seems to be relevant and addresses a significant criticality of the distributed SDIs. However, the focus / main objective of the article is not always clear. Is the main objective to describe a new conceptual approach/model and architecture for the use of search engines in order to support the discovery?; or is the main focus on the hhypermap software?; or again, about its use within WorldMap?

Probably the authors should immediately explain these aspects and use them as a fil rouge to better organize the subsequent parts of the paper (method, results, discussion) that sometimes do not seem clearly organized.

Main issues:
- it may be useful (but this also depends on the focus) to better separate the general architecture part (SDI + search engine) from specific solutions/implementations made through GeoNode, HHypermal etc. This could help the general understanding of the problem and give the paper a wider relevance
- again with reference to these aspects, the description made of the architecture (in particular for Fig. 2) is minimal and could be developed better
- if the focus of the project wants to be Hhypermap we could better evaluate the fact that it is released as open source, which is only briefly mentioned in the abstract of the article, adding also direct references to the repository. A brief explanation (or clear reference to documentation) on how to install an use the software in this case would be really beneficial.


Instruction: You are one of the reviewers, your task is to write a review for the article. You will be given the title of the article, the number of the round in which the article is located, and your order among the reviewers.

Input:
Title: ENHANCING DISCOVERY IN SPATIAL DATA INFRASTRUCTURES USING A SEARCH ENGINE
Review round: 1
Reviewer: 2

Output:
Basic reporting: Good and precise use of the English language applies.

Introduction and Background demonstrate the authors full grasp of both of the geospatial and library science aspects of the discussed topic, including applicable state-of-the art standards and software projects.

The document structure conforms to the PeerJ standards. Figures are relevant, well drawn and correctly labeled. Links to the project web pages of the described software tools (both for results and used sources) are provided, which point to the respective software repositories.
Experimental design: The research is within the Scope of the journal.

The research question is well defined.

The descriptions of the HyperMap and WorldMap software stacks, the architectural views, description of functionality and examples of input/output (Figure 3) are very concise.

The conclusion, that search engine technology should be embraced in future editions of the CSW OGC standard is sound and would would make a lot of sense.
Validity of the findings: The results are valid and a step forward in the advancement of Science, as the impact on the improvement of the OGC CSW standard.

The developed research code is available for review and reuse from github repositories.

It would be interesting to learn in future reports about the topic (which are highly encouraged) more about the speed improvement (line 165: "about 20/30 times faster" to infer benefits of scaling effects for application scenarios which are currently still prohibitive due to technical limitations (CSW standard definition).
Additional comments: No additional comments


Instruction: You are one of the reviewers, your task is to write a review for the article. You will be given the title of the article, the number of the round in which the article is located, and your order among the reviewers.

Input:
Title: ENHANCING DISCOVERY IN SPATIAL DATA INFRASTRUCTURES USING A SEARCH ENGINE
Review round: 2
Reviewer: 1

Output:
Basic reporting: no comment
Experimental design: no comment
Validity of the findings: no comment
Additional comments: The authors modified the manuscript accordingly to the comments provided, clarifying the highlighted issues, improving the structure and clarifying the objectives of the study.
I recommend this article for publication.


Instruction: You are one of the reviewers, your task is to write a review for the article. You will be given the title of the article, the number of the round in which the article is located, and your order among the reviewers.

Input:
Title: SUPERVISED DEEP LEARNING EMBEDDINGS FOR THE PREDICTION OF CERVICAL CANCER DIAGNOSIS
Review round: 1
Reviewer: 1

Output:
Basic reporting: Section Introduction

-“The two most widespread techniques in screening programs are cytology, either conventional or liquid,
38 and colposcopy [11, 26, 12, 35].” -> This statement appears to be unrelated to the place where it is located in the introduction.

- “We applied the our learning scheme…” -> check this.

Section Methods

- “In many cases, researchers perform these two steps separately, deriving  suboptimal combined models.”-> citation needed

- The description of the proposed method should be improved.

- It is not clear which ideas portrayed are original or whether they have been proposed before in other works.

Section Dataset

-“The hospital anonymized all the records before releasing the dataset The dataset is” -> check this

-“normalization, and we inputed missing data using the average value [4].” -> Why authors used this method to the estimation of missing values?. There are several methods more accurate than the used one.

Section Software and Data Availability

- The URL https://github.com/kelwinfc/cervical-cancer-screening is not working.
- The URL https://archive.ics.uci.edu/ml points to the home of UCI repository and not to the dataset used.
Experimental design: -	A better explanation of the steps carried out to tune the neural network is required.
-	Some parameters configurations taken by the authors are not clear and they require a better explanation. For instance, why authors used RMSProp and no other optimization algorithm.?
Section Results in other applications

-	More than six biomedical datasets should be considered to demonstrate the effectiveness of the method. Also, the results must be analysed by means of non-parametric test to detect significant differences between the configurations.
-	On the other hand, it would be interesting if the authors compare their method to other well-established classification algorithms.
Validity of the findings: Section Discussion

“From a technical point of view, while machine learning techniques have achieved state-of-the-art
results in several applications, the lack of interpretability of the induced models can limit their general
acceptance by the medical community.” -> This statement is partially wrong, due to there are several machine learning approaches that return interpretable models such as rule-based methods and decision tree-based methods. Even, there are several works proposing the extraction of rules from the results of black-bocks-based models.

- The discussion and conclusions are well stated, they are limited to the results obtained.
Additional comments: I consider that this work is interesting, but it should be significantly improved before it could be considered for publication. I suggest that authors review the work attending the comments made by the reviewer.


Instruction: You are one of the reviewers, your task is to write a review for the article. You will be given the title of the article, the number of the round in which the article is located, and your order among the reviewers.

Input:
Title: SUPERVISED DEEP LEARNING EMBEDDINGS FOR THE PREDICTION OF CERVICAL CANCER DIAGNOSIS
Review round: 1
Reviewer: 2

Output:
Basic reporting: The article is well written but there are some typos:

line 67: please replace "we applied the our..." with "we applied our..."
line 76: please replace "data high dimensionality" with "high dimensionality data"
line 144: please replace "autoencoders can to learn" with "autoencoders can learn"
line 174: please replace "behavior related different" with "behavior related to different"
Experimental design: Methods Section of the paper needs some improvements.
As for the dataset, what are the type and numebr of features? Apart from the reference [32], please add a brief description of the dataset.
The paper lacks of some technical details.
First of all, how did you choose the number of features for the dimensionality reduction phase? please clarify
The set of parameters (Table 1) of neural network should be better described.
Please add some implementation details such as:
- programming language
- adopted deep learning framework
- execution times
On line 198, for the first time you speak about a semi-supervided approach: please clarify
On Table 2 and 3, the first row to which subese is referred to?
Validity of the findings: What is the performance gain provided by the dimensionality reduction procedure? please clarify
In order to make the paper more interesting, you should compare your ANN classifier with other well known classification algorithms, such as SVM, Naive Bayes, k-NN, random forest, using both the original dataset and the low dimensional one.
Additional comments: No additional comments


Instruction: You are one of the reviewers, your task is to write a review for the article. You will be given the title of the article, the number of the round in which the article is located, and your order among the reviewers.

Input:
Title: SUPERVISED DEEP LEARNING EMBEDDINGS FOR THE PREDICTION OF CERVICAL CANCER DIAGNOSIS
Review round: 2
Reviewer: 1

Output:
Basic reporting: Section Introduction

“However, many projects that take advantage of dimensionality reduction and classification use suboptimal approaches, where each component is learned separately.” → citation needed.

Section Methods

“We use a Parametric Rectifier Linear Unit (PReLU) [20] as activation function in the hidden layers of our architectures (Figure 1).” → Why the authors use PreLU?, it may be ok, but some justification must be given in this regard.

Section Dataset

- I do’nt understand very well what means bool × int, int x int, etc.
Experimental design: - Statistical comparisons by means of non-parametric test should be carried out in order to draw conclusions more reliable. For example, what is the model that, in average, obtains the more significant results in Table 6?
Validity of the findings: - The discussion and conclusions are well stated, they are limited to the results obtained.
Additional comments: I consider that the work was improved; the authors addressed most of the comments raised in the previous version.


Instruction: You are one of the reviewers, your task is to write a review for the article. You will be given the title of the article, the number of the round in which the article is located, and your order among the reviewers.

Input:
Title: SUPERVISED DEEP LEARNING EMBEDDINGS FOR THE PREDICTION OF CERVICAL CANCER DIAGNOSIS
Review round: 2
Reviewer: 2

Output:
Basic reporting: no comment
Experimental design: no comment
Validity of the findings: no comment
Additional comments: The authors succesfully answered all my remarks, therefore their paper can now be accepted


Instruction: You are one of the reviewers, your task is to write a review for the article. You will be given the title of the article, the number of the round in which the article is located, and your order among the reviewers.

Input:
Title: INTERVAL CODED SCORING: A TOOLBOX FOR INTERPRETABLE SCORING SYSTEMS
Review round: 1
Reviewer: 1

Output:
Basic reporting: No comment
Experimental design: No comment
Validity of the findings: No comment
Additional comments: In this manuscript the authors introduce a Matlab toolbox for ICS. This toolbox provides not only the basic code to perform ICS but also an interactive process with plots and guides to help the users.

This work is well structured, complete and self contained. The toolbox validation section is a plus encouraging the reader to use this new toolbox.

Some minor comments to improve reading of the manuscript:

* In page 4 the authors indicate the possibility of coding interactions. I would recommend adding a sentence offering an example of an interaction in a medical domain.
* In the next page the authors talk about the reweighting process. An example of a solution of ICS with and without this process could help the reader understand why interpretation is a must in some domains.
* A link to the software or directions on where to obtain, public repository, etc. of the toolbox is mandatory.


Instruction: You are one of the reviewers, your task is to write a review for the article. You will be given the title of the article, the number of the round in which the article is located, and your order among the reviewers.

Input:
Title: INTERVAL CODED SCORING: A TOOLBOX FOR INTERPRETABLE SCORING SYSTEMS
Review round: 1
Reviewer: 2

Output:
Basic reporting: There are several use of parenthesis which can be misunderstood. For example, at line 23 of abstract, the authors write “...for a (binary) classification problem...” which can be understood as the model can be applied in binary and non-binary classification problems. However, along the paper the authors use the binary classification. I suggest to clarify it with examples or eliminating the parenthesis.
Experimental design: Some suggestions are made in the attachment.
Validity of the findings: - In section 3, there are some comparisons with classical machine learning methods. Concretely, at line 465 the authors say “… it is clear that the differences are relatively small...”. It is a subjective fact without any other supports. This sentence should be supported with some objective studies or analysis, such us,non-parametrical analysis (e.g. Shaffer test.)
- In order to achieve a better comparison, the authors should add the ranking data and carry out an analysis of them, among their proposals and among the rest of them.
Additional comments: No additional comments


Instruction: You are one of the reviewers, your task is to write a review for the article. You will be given the title of the article, the number of the round in which the article is located, and your order among the reviewers.

Input:
Title: AUTOWIG: AUTOMATIC GENERATION OF PYTHON BINDINGS FOR C++ LIBRARIES
Review round: 1
Reviewer: 1

Output:
Basic reporting: There are some minor grammar issues throughout the manuscript:

1. Missing articles: lines 119 ("a C++ library"), 150 ("the AutoWIG big
   picture"), 212 ("the definition"), 288 ("the previous files"), 212 ("in
   which the definition"), 288 ("class selection for the previous files"), 337
   ("a few problems"), 341 ("if the static overload"), 370 ("most of the
   AutoWIG guidelines"), and 547 ("the C++11 standard") (also note "C++11"
   should be spelled without any spaces) (note this list is not exhaustive).

2. Incorrect pluralization: lines 152 ("represents code abstractions and
   captures code components"), 273 ("name collisions in Python"), Table 2
   ("this enables the choice"), 351, ("it eases code introspection"), Figure 2
   (user tries to set"), 442 ("tries to convert"), 513 ("every type and
   function has to be"), 582 ("AutoWIG translates"), and 584 ("used to
   respectively generate"), (note this list is not exhaustive).

3. Some adverbs should go before the word they modify: lines 259
   ("Boost.Python extensively uses"), 282 ("This allows to easily select"),
   and 442 ("automatically convert").

4. Some other minor grammar issues: lines 20 ("lower computer processing time
   at the cost of high scientist coding time"), 78 ("even if there exists
   classical approaches"), 281 (delete "contains"), Table 2 ("Three plugin
   managers") (numbers that being a sentence should be spelled out), 307,
   delete the comma ("Any non self-contained headers should not be given"),
   334 ("in a basic Python package"), 341 ("has for its first parameter"), 342
   ("all following parameters correspond to another"), Figure 2 ("if a user"),
   and 400, the sentence "assuming that the asg ..." is a fragment.

5. Line 21 "at the opposite, scripting languages" should perhaps be "at the
   opposite end of the spectrum, scripting languages".

6. The way in which the sentence in lines 211-214 is written makes it very
   difficult to read. I would suggest rewriting it as "In order to have access
   to all template class specialization definitions, AutoWIG parses a virtual
   program of undefined template class specialization definitions (e.g., using
   sizeof(std::vector< int >); for forcing a std::vector< int > definition)."

7. The double negative in line 291 ("are also wrapped if they are not
   explicitly marked as non-exportable") makes it difficult to parse. I would
   suggest, "are also wrapped, unless they are explicitly marked as
   non-exportable."

8. The sentence on lines 514-515, "note that if this approach..." is
   confusing. The "if" in particular is confusing, as there doesn't seem to be
   any logical causation to the implication. Perhaps (if I am correctly
   understanding the authors' intention), a better reading would be, "note
   that although this approach is only efficient when exposing a small number
   of functions and objects, it is at the basis of all wrapper tools that
   generate C API code".

9. The citation for (Perez and Granger, 2007) on line 598 is not parenthesized
   correctly.

10. Please add a reference citation for abstract semantic graphs.

11. Are there references that can be added for StatTool and SequenceAnalysis?

12. Please add URLs for the software citations.
Experimental design: 1. For the requirements section, there is no discussion about types and data
   structures. This is distinct from the Pythonic interface. C++ is a typed
   language, whereas Python is dynamic. Furthermore, Python has different
   builtin data types from C++. So any wrapper must answer some questions: how
   are invalid input types handled (C++ disallows this at compile time, but in
   Python any type checks must happen at runtime)? How are Python types such
   as int converted to the respective C++ types, and is any attempt made to
   preserve Python semantics where they differ from C++'s (e.g., Python ints
   are big integers which can be arbitrarily large)? And how are data
   structures handled (will a wrapped function that requires an array or
   vector type support a homogeneous Python list type? Will it support a NumPy
   array? Can a wrapped function that operates in-place operate on a NumPy
   array without copying)? I'm particularly interested in the last point. If a
   wrapper cannot interoperate efficiently with NumPy arrays it will not be
   useful for most scientific applications. My understanding is that AutoWIG
   relies on Boost.Python to handle this logic.

   This is discussed in the findings itself, in section 6.3, although it is
   not clear what containers are supported, and in particular NumPy arrays are
   never mentioned.
Validity of the findings: 1. I did not see the code or data for the example in section 6.4 in the
   autowig repository. If they are in another repository it should be listed
   in the supplementary materials. As it stands this example cannot be
   reproduced.

2. It is disappointing to see a new Python library that does not support
   Python 3. Python 2 support will be sunsetting in 2020, which includes a
   large number of scientific libraries (see
http://www.python3statement.org/). I have seen in the issue tracker that
   this is due to the dependency on scons. In the manuscript the fact that the
   library requires Python 2.7 is not mentioned until the end (in section
   7.4), and no reason is given.

3. In the examples in the docs, the generated files are all checked in to the
   git repository. This makes it difficult to tell which files are supposed to
   be human generated and which are supposed to be machine generated.

4. I found it somewhat confusing/misleading that the intro materials of the
   paper discuss C, C++, and Fortran, but AutoWIG itself only supports C++. I
   understand that AutoWIG is designed in such a way that it could support
   other languages, but that they are not presently supported was not clear
   until the very end, when it was explicitly stated in section 7. The same
   applies for R as a target language.

5. I'm curious what the largest pre-existing C++ library is that AutoWIG has
   been used or attempted to be used on. No indication is given for how large
   the examples from section 6 are, or if wrapping other larger libraries was
   attempted.

6. Are the plots from section 6.4 generated with matplotlib as implied (lines
   460 and 476 and Figures 3 and 4)? If so, how?

7. It would be helpful to see some example of what the Mako templates look
   like, perhaps either as a suggestion of what they might look like for R in
   section 7.2, or show part of the existing templates for Python in section
   4.

8. Section 7.5 (line 608) mentions a tool for parsing compiler errors. Does
   this tool actually exist in AutoWIG, or is this a proposal for future work?
   If it does exist, it should be mentioned earlier. If it doesn't that should
   be made clearer.
Additional comments: The AutoWIG approach to binding generation is promising. I like that it 1) is
designed to be used interactively, and 2) is designed to be used as a Python
library, without inventing a DSL.

1. On line 270, what is the "global name hash"? Is this a reference to the
   Python built-in function "hash" or something else? If it is, will AutoWIG
   still function correctly if hash randomization is enabled (it is on by
   default in Python 3, and can be enabled in Python 2 using the -R flag to
   the interpreter)?

2. The sentence on line 307, "any non self-contained headers should not be
   given to a parser but can nevertheless be considered during parsing using
   relevant search path flags (given by the -I option)," appears
   contradictory, especially since the search path flags are one of the
   arguments to a parser plugin. I cannot suggest an alternative because I am
   really unclear what the authors are trying to say here.

3. It is stated that certain components are not wrapped due to ambiguity (line
   322). Are these ambiguities warned about? How can they be worked around?

4. Is it necessary to set the "default" plugins (such as
   autowig.controller.plugin = 'default', line 371)?

5. Concerning the plugin model, what is the value of the indirection

        autowig.controller['clanglite'] = clanglite_controller
        autowig.controller.plugin = 'clanglite'
        asg = autowig.controller(asg)

   rather than simply

        asg = clanglite_controller(asg)

6. Does AutoWIG write a setup.py or any other build scripts, or does the user
   need to write those manually?

7. Section 6.2 is an interesting meta-example (indeed, I believe this same
   example was mentioned in Anthony Scopatz's 2013 SciPy presentation on
   XDress as an interesting potential thing to wrap). However, since it is a
   meta-example, it is unclear from the outset that this is indeed an example
   and simply a discussion of AutoWIG's limitations, so I would suggest to
   make this clearer somehow.

8. GCC-XML is dismissed because it does not support C++11. Would LLVM still be
   preferred over it if it did support C++11?

9. The link to PySTL in the template.ipynb notebook
   (https://github.com/StatisKit/PySTL) is broken. The link in the code
   (https://github.com/StatisKit/STL) appears to be correct.

10. The conda build commands in the subset.ipynb file failed for me with
    errors like

        failed to get install actions, retrying.  exception was: The following specifications were found to be in conflict:
          - python 3.5.*
          - python-dev -> libdev -> python 2.7*
        Use "conda info <package>" to see the dependencies for each package.

    Everything else in the notebook worked for me, except for some empty
    warning message:

        /Users/aaronmeurer/anaconda3/envs/autowig/lib/python2.7/site-packages/clanglite/autowig_parser.py:860: Warning:
        warnings.warn('', Warning)

    I believe it all ended up using my installed version of clanglite instead
    of the source version, because the build step failed.

11. The subset.ipynb file has a typo that prevents it from running. 'from path
    import path' should be 'from path import Path' in the clanglite_controller
    function.


Instruction: You are one of the reviewers, your task is to write a review for the article. You will be given the title of the article, the number of the round in which the article is located, and your order among the reviewers.

Input:
Title: AUTOWIG: AUTOMATIC GENERATION OF PYTHON BINDINGS FOR C++ LIBRARIES
Review round: 1
Reviewer: 2

Output:
Basic reporting: The paper is well presented. It researches literature and mentions all major
approaches to wrap C++ in Python. Then it describes a new approach based on
LLVM, which is more automatic and can be used with large and complex C++
libraries. The paper describes well how the AutoWIG works and how particularly
difficult C++ constructs are handled and how automatic documentation is
generated. It gives three bigger examples how the library can be used and the
corresponding Jupyter notebooks are available.

A minor correction:

1) e.g std::vector, std::set -> e.g., std::vector, std::set
Experimental design: I have downloaded the Docker image using the following command on Ubuntu 16.04:

	$ docker run -i -t -p 8888:8888 statiskit/autowig:v1.0.0-alpha

And as instructed, I launched the Jupyter notebook from inside the container
using:

	$ jupyter notebook --ip='*' --port=8888 --no-browser

The `basic.ipynb` ran fine. Here are some minor corrections that I noticed:

2)	a basic example of C++ library -> a basic example of a C++ library

3)	A `basic.ProbabilityError` should have been raise -> raised

The `subset.ipynb` failed in the cell [7] with the following error:
```
---------------------------------------------------------------------------
ImportError                               Traceback (most recent call last)
<ipython-input-7-4e28706c39ff> in <module>()
----> 1 get_ipython().run_cell_magic(u'time', u'', u"autowig.controller['clanglite'] = clanglite_controller\nautowig.controller.plugin = 'clanglite'\nasg = autowig.controller(asg)")

/home/main/miniconda/lib/python2.7/site-packages/IPython/core/interactiveshell.pyc in run_cell_magic(self, magic_name, line, cell)
   2115             magic_arg_s = self.var_expand(line, stack_depth)
   2116             with self.builtin_trap:
-> 2117                 result = fn(magic_arg_s, cell)
   2118             return result
   2119

<decorator-gen-60> in time(self, line, cell, local_ns)

/home/main/miniconda/lib/python2.7/site-packages/IPython/core/magic.pyc in <lambda>(f, *a, **k)
    186     # but it's overkill for just that one bit of state.
    187     def magic_deco(arg):
--> 188         call = lambda f, *a, **k: f(*a, **k)
    189
    190         if callable(arg):

/home/main/miniconda/lib/python2.7/site-packages/IPython/core/magics/execution.pyc in time(self, line, cell, local_ns)
   1183         else:
   1184             st = clock2()
-> 1185             exec(code, glob, local_ns)
   1186             end = clock2()
   1187             out = None

<timed exec> in <module>()

<ipython-input-6-316a91bf6e70> in clanglite_controller(asg)
    132
    133     import sys
--> 134     from path import path
    135     for header in (path(sys.prefix)/'include'/'clang').walkfiles('*.h'):
    136         asg[header.abspath()].is_external_dependency = False

ImportError: cannot import name path
```

The `template.ipynb` failed in cell [5] with:
```
scons: Entering directory `/home/main/AutoWIG/doc/examples/STL'
scons: Reading SConscript files ...
scons: done reading SConscript files.
scons: Cleaning targets ...
Removed /home/main/miniconda/include/statiskit/stl/STL.h
scons: done cleaning targets.
scons: Entering directory `/home/main/AutoWIG/doc/examples/STL'
scons: Reading SConscript files ...
scons: done reading SConscript files.
scons: Building targets ...
Install file: "build/src/cpp/STL.h" as "/home/main/miniconda/include/statiskit/stl/STL.h"
Install file: "build/src/cpp/AutoWIG.py" as "/home/main/miniconda/lib/python2.7/site-packages/autowig/site/controller/statiskit_stl.py"
autowig: Generating Boost.Python interface ...
scons: *** [/home/main/miniconda/lib/python2.7/site-packages/autowig/site/ASG/statiskit_stl.pkl] ImportError : No module named clanglite
Traceback (most recent call last):
  File "/home/main/miniconda/lib/python2.7/site-packages/SCons/Action.py", line 1054, in execute
    result = self.execfunction(target=target, source=rsources, env=env)
  File "/home/main/miniconda/lib/python2.7/site-packages/SCons/site_scons/site_tools/wig.py", line 38, in boost_python_builder
    parser = import_module('autowig.site.parser.' +  AUTOWIG_PARSER)
  File "/home/main/miniconda/lib/python2.7/importlib/__init__.py", line 37, in import_module
    __import__(name)
ImportError: No module named clanglite
scons: building terminated because of errors.
```
Some corrections:

4) Once the headers habe been installed in the system -> have been

5) situed -> situated

6) For the complete procedure refers to the -> refer

7) Once these preliminaries done -> are done

The package has a simple test suite and it is tested by Travis-CI. The fact
that two out three notebooks failed for me in Docker (which should be very
reproducible) suggests that the installation and usage procedure is somewhat
fragile.

8) It would be nice to clarify exactly which Docker tag is supposed to work and
clarify in the documentation the exact steps to make all three notebooks
succeed.
Validity of the findings: The package seems to work, with the caveats in the previous section. The main
conclusion that AutoWIG greatly simplifies the process of wrapping large and
complex C++ libraries seems to be true. In order to increase the adoption of
the package, I provided a note about a license below, that the authors might
optionally consider.

I recommend the article for publication, after the minor changes 1)-8) are
fixed.

License: currently the AutoWIG software is distributed using the CeCILL-C
license. The Python scientific community encourages to use permissive licenses
such as MIT or BSD [1], and most packages are licensed as such, e.g., from the
libraries the authors cited in the paper, the following use permissive
licenses: Python, SciPy, Cython, Boost.Python, SWIG, F2PY, XDress, IPython,
Jupyter, LLVM, Clang, Armadillo, Matplotlib, VTK, Mako, libc++ (STL), Sphinx,
WrapITK, Docker. The libraries that do not use permissive open source licenses:
Sage, Matlab, SWIG, R, Octave, Rcpp, Doxygen. The authors might consider using
a permissive license such as MIT and BSD to get wider adoption of the library
by the scientific Python community. However, this is by no means an obstacle to
publishing this article, and also the authors might not be in a position to
change the license in the first place.

[1] http://nipy.sourceforge.net/software/license/johns_bsd_pitch.html
Additional comments: No additional comments


Instruction: You are one of the reviewers, your task is to write a review for the article. You will be given the title of the article, the number of the round in which the article is located, and your order among the reviewers.

Input:
Title: AUTOWIG: AUTOMATIC GENERATION OF PYTHON BINDINGS FOR C++ LIBRARIES
Review round: 2
Reviewer: 1

Output:
Basic reporting: The paper itself is much improved. I only have one suggestion, which is to use the names of the wrapped libraries in the left column of the added table (table 3), instead of just section numbers.
Experimental design: I am happy to see that AutoWIG now supports Python 3, and am also happy to see the license change to a more standard licence. The examples are now much easier to see and run that they are in a separate repo. The fact that the generated files are untracked, along with the git status outputs in the notebooks, makes it clearer which files are generated.
Validity of the findings: Unfortunately, I was not able to get the example notebooks to run on my Mac. Each notebook failed with errors like

  ImportError: dlopen(/Users/aaronmeurer/anaconda3/envs/FP17/lib/python3.6/site-packages/clanglite/__clanglite.so, 2): Library not loaded: @rpath/libboost_python.dylib
    Referenced from: /Users/aaronmeurer/anaconda3/envs/FP17/lib/python3.6/site-packages/clanglite/__clanglite.so
    Reason: image not found

This happened even if I skipped the conda build steps and used the package from conda-forge. It seems the libboost conda package used by the examples does not actually contain libboost_python.dylib. I do not have enough experience with boost-python to offer a solution, although I did notice that the boost conda package (from conda-forge) has libboost_python3.dylib.

Incidentally, it may be a good idea to create an environment.yml that locks in specific package versions known to work with each platform (conda env export -n fp17) for the purposes of reproducing the paper.

In the Python 3 docker container, both for the 1.0 and "latest" tagged containers, dependent.ipynb failed when building python-stat-tool with

    In file included from build/src/py/wrapper/_stat_tool.h:6:0:
    /home/main/miniconda/conda-bld/python-stat_tool_1509148131720/_b_env_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placeh/include/stat_tool/reestimation.h: At global scope:
    /home/main/miniconda/conda-bld/python-stat_tool_1509148131720/_b_env_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placeh/include/stat_tool/reestimation.h:177:3: error: 'STAT_TOOL_API' does not name a type
       STAT_TOOL_API double interval_bisection(Reestimation<double> *distribution_re
       ^
    scons: *** [build/src/py/wrapper/_stat_tool.h.gch] Error 1

The other three notebooks worked in Docker under both tags.

It's also worth pointing out that the commands in the notebooks that use ! do not cause the notebook cell with the command to fail when they exit nonzero, meaning errors are typically seen in later cells.

I do not have a Windows machine, so I was unable to test that platform. However, I see that the notebooks are built on AppVeyor, but I was not able to find the builds.

The Binder link failed with

    Step 27/28 : RUN conda env update -n root -f "environment.yml" && conda clean -t
    ipsy
     ---> Running in ad47c3a7b88b
    Fetching package metadata .............
    Solving package specifications:

    NoPackagesFoundError: Dependencies missing in current linux-64 channels:
      - statiskit-toolchain -> libboost
      - statiskit-toolchain -> python-clanglite -> libclanglite
Additional comments: No additional comments


Instruction: You are one of the reviewers, your task is to write a review for the article. You will be given the title of the article, the number of the round in which the article is located, and your order among the reviewers.

Input:
Title: AUTOWIG: AUTOMATIC GENERATION OF PYTHON BINDINGS FOR C++ LIBRARIES
Review round: 3
Reviewer: 1

Output:
Basic reporting: No comment
Experimental design: No comment
Validity of the findings: I was able to reproduce the examples on Linux with the provided Docker containers and macOS (I unfortunately do not have access to Windows), for Python 2 and 3 (except for the StructureAnalysis example on Python 3, as noted in the response and paper). A few of the conda recipes required minor modifications to work in macOS. I have made pull requests to the STL, ClangLite, and StructureAnalysis repos to fix these issues. These require no changes to the paper itself.
Additional comments: No additional comments


Instruction: You are one of the reviewers, your task is to write a review for the article. You will be given the title of the article, the number of the round in which the article is located, and your order among the reviewers.

Input:
Title: INDEXING LABELED SEQUENCES
Review round: 1
Reviewer: 1

Output:
Basic reporting: This is an easy to read article targeted to wider audience. The article illustrates how easy it is to design a tailored solution, exploiting modern compressed text indexing, to combine labels and DNA sequence into a common data structure that allows to make useful queries on the content. All the components are widely known, but this combination has not been previously reported; there is, though, an even more broad generalization of adding an XML tree on top of a sequence [1], which encompasses linear labelling as a special case. That article is not targeted to a general audience, so reporting this special case with a tailored (and different) solution is completely desirable.

English should be improved. I spotted some very easy typos detectable by a spell checker, and with high probability there are more (page 1, line 43: let call -->let us call; page 6, line 137: anwser --> answer).

[1] Diego Arroyuelo, Francisco Claude, Sebastian Maneth, Veli Mäkinen, Gonzalo Navarro, Kim Nguyen, Jouni Sirén, Niko Välimäki. Fast in-memory XPath search using compressed indexes. Softw., Pract. Exper. 45(3): 399-434 (2015)
Experimental design: As this is a new problem, there is no good baseline to compare. Comparison to naive solution is fine.
Validity of the findings: Authors show a proof of concept with a biological question motivating the study.
Additional comments: No additional comments


Instruction: You are one of the reviewers, your task is to write a review for the article. You will be given the title of the article, the number of the round in which the article is located, and your order among the reviewers.

Input:
Title: INDEXING LABELED SEQUENCES
Review round: 1
Reviewer: 2

Output:
Basic reporting: The authors essentially consider the problem of indexing a pair of strings $T \in \Sigma*$ and $L \in {1, \ldots, \ell}$ such that, given a pattern $P$ and a integer $x$ between 1 and $\ell$, they can quickly count and/or return all the positions $j$ such that $T [j..j + |P| - 1] = P$ and $L [j] = x$.

Their solution is essentially to store an FM-index for $T$ together with a wavelet tree for the permutation $W$ of $L$ in which $W [j] = L [\SA [j]]$, where $\SA$ is the suffix array of $T$. Given $P$ and $x$, they use the FM-index to find the suffix array interval for $P$, then use the wavelet tree to count and/or find the occurrences of $x$ in that interval in $W$, then possibly use the FM-index's suffix-array sample to find also the corresponding positions in $T$.

The solution itself is correct, but obvious.  The explanation is unnecessarily complicated, some of the bounds cited are not the best known, and several of the references are outdated.  I would suggest the authors significantly tighten up their presentation before publication, although none of the problems are critical.
Experimental design: No comment.
Validity of the findings: The findings are correct, although not profound.
Additional comments: It seems fairly easy to build an $O (n \log n)$-bit index such that, given a pattern $P$, a label $x$ and a position $i$ in $P$, we can reasonably quickly find all the positions where $P$ occurs in $T$ with its $i$th character labelled $x$.  Offhand, however, I don't see how to reduce the space to $O (n \log \Sigma)$.  A solution to that problem might make the article more interesting for researchers familiar with pattern matching.


Instruction: You are one of the reviewers, your task is to write a review for the article. You will be given the title of the article, the number of the round in which the article is located, and your order among the reviewers.

Input:
Title: INDEXING LABELED SEQUENCES
Review round: 1
Reviewer: 3

Output:
Basic reporting: This paper presents two indexes for labels texts. Each position of a text T of length n is marked with exactly one (possible empty) label.
The label string A is therefore also of length n.

The first index uses an FM-index over T and a WT over the run-length compressed version of A. The WT stores the sequence of run heads, while bit vector B_A marks the first position of the runs in T.
Since a search in the FM-index results in a SA range, access to A is not cheap since SA positions have to be translated into text positions. The cost of this translation is proportional to the SA-sampling s_SA.

In a second version of the index, A is also transformed in SA-order. Run-length compression is again applied.

A set of operations is defined and the authors show how to answer these operations. The algorithms are compositions of already known results.36: on an alphabet -> over an alphabet
Experimental design: The experimental section compares an implementation of the two indexes to a naive baseline. The implementation is based on the SDSL library but important details, i.e. which SA sampling strategy was used, are missing in the current version of the article. It is also unclear why the author opted for rrr_vector to represent the bit vectors. It is expected that sd_vector is superior to rrr_vector for long labels.

I suggest to fix the last two issues and accept the paper.
Validity of the findings: The index is a composition of already know techniques and its practical implementation and the experiments are worth a publication. The benchmark is available and the experiments seem to be sound.
Additional comments: Details:
40: The figure 1 -> Figure 1
58: there is a better construction of the WT (Munro, Nekrich, Vitter TCS 2016); ok mentioned in 146
71: The usual sampling is log^{1+\epsilon} n in theory. Please consider also practice: Gog and Navarro (SEA 2014) present a practical way to sample SA and ISA at the same time. Ferragina, Siren, and Venturini (ESA 2011) experimented with distribution-aware sampling.
Figure 2: These TL -and .. ->  TL- and
106: figure 3 -> Figure 3, also 112
166: Is is not necessary to keep the ordering. This is not possible with Huffman but with Hu-Tucker codes.
221: The Figure -> Figure


Instruction: You are one of the reviewers, your task is to write a review for the article. You will be given the title of the article, the number of the round in which the article is located, and your order among the reviewers.

Input:
Title: JOURNAL OF OPEN SOURCE SOFTWARE (JOSS): DESIGN AND FIRST-YEAR REVIEW
Review round: 1
Reviewer: 1

Output:
Basic reporting: no comment.
Experimental design: no comment.
Validity of the findings: no comment.
Additional comments: It might be nice to include some information on how reviewers for JOSS submissions are identified and/or selected. It seems quite an open review system and I wonder if the editors have protocols in place to deal with multiple unsolicited reviewers jumping onto a submission's review Github issue, potentially with conflicting opinions or biases in their experiences.

Given the JOSS review happens entirely via a Github issue and presumably most reviewers are often working with their own projects on Github, do the JOSS editors make use of any special customisations or additions to try and help JOSS review comments stand out from the general "noise" of Github notifications a reviewer is likely exposed to? Or any other thoughts from the editors to try and help make sure JOSS review notifications are not missed, as briefly mentioned as a reason for some delayed reviews.