Instruction: You are one of the reviewers, your task is to write a review for the article. You will be given the title of the article, the number of the round in which the article is located, and your order among the reviewers.

Input:
Title: MULTIPLE COMPARATIVE METAGENOMICS USING MULTISET K-MER COUNTING
Review round: 1
Reviewer: 2

Output:
Basic reporting: In this manuscript, the authors reported the development of a method to compare metagenomic datasets based on k-mer counting. Not like some other tools, this tool - Simka can not only calculate the Bray-Curtis similarity, but also many other similarity metrics, which is nice. In this method, the k-mers abundance profiles across the metagenomic datasets are calculated. However taking advantage of the additive nature of computing some similarity metrics, the k-mers abundance profiles do not need to be stored and so is the huge k-mer count matrix, which reduces the disk usage. The authors demonstrated the benchmarking of Simka compared to other tools and compared the similarity measurement computed with Simka to that computed using other methods like sequences alignment and taxonomic profiling. Generally the manuscript was well written, with comprehensive description of the methods, data and pipeline for reproducibility. The software package repository is well organized on Github and it has good and clear documentation, which is very nice.  There are some comments below about this manuscript though.
Experimental design: Main Comments

1. One of the most challenging problems in using k-mer counting to compare metagenomics datasets is how to deal with the k-mers from sequencing errors. As the authors mentioned in line 196, many of the k-mers with very low abundance come from sequencing errors. The solution of this method is filtering out those k-mers with abundance as 1, with those “solid k-mers” left. This works fine with metagenomics data set with higher coverage, as shown in the manuscript, with HMP samples as the testing dataset. It will be interesting to see how this method performs for other metagenomic datasets with lower coverage or higher diversity, like some environmental data sets. The IMG/M datasets used in COMMET  paper and the Global Ocean Sampling datasets used in Comareads and Mash are two good candidates since in this manuscript, the authors compared the performance of Simka with COMMET and Mash. Also in line 475, the authors mentioned “Simka is able to capture such subtle signal raises hope of drawing new interesting biological insights from the data, in particular for those metagenomics project lacking good references (soil, seawater for instance).” and in line 528, “However, species composition based approaches are not feasible for large read sets from complex ecosystems (soil, seawater) due to the lack of good references and/or mapping scaling limitations. Moreover, our proposal has the advantage of being a de novo approach, unbiased by reference banks inconsistency and incompleteness.”  It will be great if there are experiment results on those soil, seawater samples that can support these points.
Validity of the findings: 2. In various parts in this manuscript, the authors mentioned that the solid k-mers filtering out does not affect results (line 376, 442, 489). This may be due to the high coverage of the HMP data sets. In the discussion in line 369-380, the authors mentioned that a small proportion (15%) of k-mers account for 95% of all base pairs of the whole dataset, which demonstrates that the HMP datasets have relatively higher sequencing coverage and most of the low abundance k-mers filtered out are probably sequencing errors. This may explain why the Simka results are robust with filtering (line 441) Just claiming that filtering out low abundance k-mers does not affect similarity measurement may not be accurate, at least before we see how this works for other environmental data sets with lower coverage recommended above. It will be nice if the authors can explain this more clearly.

3. Similarly, in line 490-493, the authors mentioned the filter can be disabled for samples with low coverage or where the rare species have more impact.  But in this situation, how to deal with those large amount of erroneous k-mers? How will this affect the performance?  In line 493-495, the authors claimed Simka is able to scale without solid k-mers filter. This may be true for the HMP data shown in the manuscript, but we still need to wait to see how it works for the low coverage data sets.
Additional comments: 1. In Abstract- Methods, “Simka scales-up today metagenomic projects thanks to a new parallel k-mer counting strategy on multiple datasets.”  Should  “today” be “today’s”?
2. Line 3-6, “In large scale  metagenomics studies such as Tara Oceans (Karsenti et al., 2011) or Human  Microbiome Project (HMP) (Human Microbiome Project Consortium, 2012a)  most of the sequenced data comes from unknown organisms and their short  reads assembly remains an inaccessible task”. But in line 330, “One advantage of this dataset(HMP) is that it has been extensively studied, in particular the microbial communities are relatively well represented in reference databases” The descriptions about HMP seem like a contradiction here.
3. Table2, “2X16G paired reads”. it may be better to be just “2 X 16 billion paired reads”. I am not quite sure if “G” can be used like this.
4. Line 433, “On the other  hand, Mash distances correlate badly with taxonomic ones (r = 0.51, see the comparison protocol in Article S1). “ It will be nice to cite Figure S3 here.
5. Line133 “For example, experiments on the HMP (Human Microbiome Project Consortium, 134 2012a) datasets (690 datasets containing on average 45 millions of reads each) require a storage space of 630TB for the matrix KC. “ How does the “630TB” calculated? For this method,  the k-mer counting matrix and k-mer frequency vectors do not need to be stored. But the frequencies of all the k-mers in each partitions across difference data sets (red squares in Figure 2) are still stored on the disk after the sorting count stage, right? If so, how different is it compared to storing the k-mer counting matrix?
6. What is the role of the GATB library in Simka? If the GATB does the actual sorting count, then the paragraph in this manuscript about sorting count may be condensed.  Also the description about the work of Chao et al. (2006) can be more precise too.
7. The software package repository is well organized and with good documentation. Just while I was trying to run the  example test with “./bin/simka -in example/simka_input.txt -out results -out-tmp temp_output”, it failed with “Illegal instruction: 4”. I was using simka-v1.3.0-bin-Darwin.tar.gz and on mac OS 10.11. It may be the problem on my side. But this may be good for the authors to know.


Instruction: You are one of the reviewers, your task is to write a review for the article. You will be given the title of the article, the number of the round in which the article is located, and your order among the reviewers.

Input:
Title: PROBABILISTIC BIOMECHANICAL FINITE ELEMENT SIMULATIONS: WHOLE-MODEL CLASSICAL HYPOTHESIS TESTING BASED ON UPCROSSING GEOMETRY
Review round: 1
Reviewer: 1

Output:
Basic reporting: No comments.
Experimental design: No comments.
Validity of the findings: No comments.
Additional comments: In this manuscript, the authors describe a strategy to understand the statistical implications of model input variations/uncertainty on model outputs obtained through the whole geometry. By calculating statistical parameters throughout the whole model domain, representing them as a field variable and proposing a non-parametric analysis to identify geometric regions that may have significance, the authors provide the possibility to move beyond the abridged analysis of simulation results based on scalar outcomes, e.g. peak von Mises stresses. The manuscript is well written, the concepts are conveyed with good examples, and the limitations of the work were discussed in a detailed manner. This reviewer is in favor of publication of this study. Yet, a few issues are noted below:

- The Mooney-Rivlin formulation noted in the manuscript, Eqn. 7, does not seem to match the one used in FEBio, particularly when volumetric component is concerned. The authors may want to check FEBio theory manual to confirm.

- It is not clear how non-parametric permutation-based probabilities are calculated for model outputs, i.e., using "sign permutation" directly on model outputs OR conducting "sign permutation" on model inputs and then conducting finite element analysis. Clarification of this is important as it dictates the required number of potentially costly simulations.

- In section 5.4, the authors note an important limitation - the methodology being suitable to designs involving only a small number of unknown parameters. While this is understandable, it may prevent conducting useful studies. It will be useful to know more about at what level such analysis become not feasible. For example, is it possible to use Model A, this time have a failure threshold parameter (variable from model to model and from element to element), where von Mises stress can be compared against to see where in the geometry failure will likely happen. It may be exciting to provide such a study as an example with two model inputs with uncertainties.

- The first author has done similar work on temporal analysis of 1D signals. How does this study relates to the author's previous work? Is it an extension to field problems, yet still 1D in terms on model input?

- It will be beneficial to disseminate the Python scripts, at least for Model A, for the readers to be able to reproduce the results and play with the capabilities of the described method.

- The authors provide a useful comparison in the last paragraph of section 5.2. Nonetheless, this paragraph may need to strengthened with examples to guide the reader about the situations where local extremum approach may be more appropriate than the model-wide approach and vice versa. It is easy to be a proponent of local extremum approach simply because bad things likely happen at locations of highest stresses, e.g. failure. The example on local extremum on mesh sensitivity is appreciated. Yet, this reviewer wonders at what type of physiological situations knowledge about non-local extremum model regions become useful. For the sake of argument (and realizing that Model C was not necessarily provided for physiological investigations), does it really matter to know that strains in ligament attachment sites are highly sensitive to internal/external rotation when they are not necessarily deforming that much to begin with?

- Please correct the phrase ".. there is a relatively large uncertainty amount of uncertainty .."


Instruction: You are one of the reviewers, your task is to write a review for the article. You will be given the title of the article, the number of the round in which the article is located, and your order among the reviewers.

Input:
Title: PROBABILISTIC BIOMECHANICAL FINITE ELEMENT SIMULATIONS: WHOLE-MODEL CLASSICAL HYPOTHESIS TESTING BASED ON UPCROSSING GEOMETRY
Review round: 1
Reviewer: 2

Output:
Basic reporting: The paper is well written and complies with the PeerJ policies and formats. However, it is hard to put it in perspective with the current practices in probabilistic finite element methods, and the practical advantages and implications of the proposed approach. Please consider the following questions/comments to further improve your manuscript:

1. Uncertainty quantification in computational methods is nowadays a mature field of research and there exist many competing methods (e.g., polynomial chaos, ANOVA, non-parametric surrogates based on Gaussian processes, etc.) for analyzing continuum statistics and propagating probability densities through nonlinear systems using FE simulations, experiments, etc. Such techniques are never mentioned in the manuscript, and it is not clear how the proposed method connects and compares against those well-established approaches. In particular, the methodology based on t-statistics outlined in Sec. 2.2 seems quite simplistic, and the conclusions drawn in lines 134-144 and section 5.1 are quite obvious. Please, provide a detailed discussion on the motivation and merits of the proposed approach as compared with the current-state-of-the-art in uncertainty quantification.

2. p.3, l.93-94: Is it this the noise that causes the asymmetry observed in the t-statistic shown in Fig.3b? What is the purpose of adding this small random signal?

3. p.4, l. 113: The abbreviation “SD” is introduced without explanation. Please spell out every abbreviation the first time it is used in the text.

4. Please provide sufficient information so that the reader can reproduce every numerical example presented. For example, provide a brief description of which equations are solve using FE? What are the boundary/initial conditions, etc?

5. p.7, l.196-197: how do you select the nodes? does this matter a lot? do you use quadrature to compute the upcrossing integral? if yes, is the selection of the nodes and the quadrature related?

6. Figure 6: Figure captions should should be self contained. Please briefly explain what we see in a,b,c,d.

7. p.14, l.386-388: Please provide a more detailed comment on the difficulties introduced by high- dimensional parameter spaces. How does the computational cost and feasibility of the proposed approach scale in such cases?
Experimental design: The submission describes original research and is in accordance with the scope of the journal. Regarding result reproducibility please see item 4.) in the Basic Reporting section.
Validity of the findings: The submission meets the scientific standards of the journal. One minor concern is that  there seems to be no connection with the current literature on uncertainty quantification in computational science. Moreover, the technical approach is rather simplistic and the conclusions drawn may seem obvious to a reader acclimated with probabilistic finite element simulations.
Additional comments: No additional comments


Instruction: You are one of the reviewers, your task is to write a review for the article. You will be given the title of the article, the number of the round in which the article is located, and your order among the reviewers.

Input:
Title: COMPETITION AND COOPERATION WITH VIRTUAL PLAYERS IN AN EXERGAME
Review round: 1
Reviewer: 1

Output:
Basic reporting: This paper is generally well-written with minor typos (e.g., Line 94 -- "maybe" should be "may be", etc.). Due to the experimental nature of the study in a controlled laboratory setting, the Introduction section of the paper should be rewritten to include a "Research Questions" statement and the associated "Hypotheses" substantiated by prior research. The current literature review section should be separated into a standalone Related Work  section. Also, it'd be great for the authors to state the primary contributions upfront at the end of the Introduction section to better guide the readers
Experimental design: The write up for the experimental design is generally clear but there exists a number of discrepancies in the reporting of study 1 and study 2 that can be improved. For example, in study 1, the participant assignments were "randomized" (line 121) whereas study 2 follows a "Latin Square" (line 317). Study 1 contains 3 conditions (e.g., control, ghost, and trainer; line 161-166) but study 2 contains also 3 conditions but is mentioned as "2" conditions (line 250-254) which is inconsistent with Table 3's caption. Study 2 did not measure RPE.

The paper does not provide a correlation table for study 1, but provides a correlation table for study 2 (though the addition of this table serves very little purpose due to the general lack of significant correlations). The authors could've done a dimension reduction to reduce SOQ, especially since none of the covariates correlate with any of the DVs. If anything, the paper should provide correlations of covariates and *all* DVs (also including calories, motivation, and enjoyment) to make the presentation more meaningful.

Significant relationships should be denoted in Table 1 and Table 2 (e.g., using superscripts a, b, c given a > b > c).
Validity of the findings: The paper contains several unsubstantiated claims. For example, statements in Line 232-233, 402-403, and 411-414 need to be justified with statistical analysis--need to provide N, posthoc comparisons, and clustering results (e.g., k-mean). Without stats, the claims are completely unsubstantiated.

The most significant issues with this paper is that the argument of competitive or cooperative is mischaracterized given the way the experimental conditions are implemented. Technically, paper experiments with 4 conditions--control (twice), ghost, basic trainer (inhuman agility), trainer-gamified (simulating more naturalistic AI-bot, which the authors classify as "competitive"), and trainer-unbeatable (supposed to simulate "cooperative" condition but fails to do so; see my comments later). By definition, "cooperation" is a situation in which the players work together toward a common goal, but the "cooperative" condition that is implemented in the experiment is one in which the AI-bot *always* wins without any cooperative element (see Figure 4 and line 398-401). This can also explain why enjoyment/motivation of competitive > control & cooperative (see Table 2). In the "cooperative" condition, the players always "lose" by definition. The way in which cooperative condition is implemented in this work is better described as a "tutorial" or an "assisted/cheat mode" in which optimal path is hinted. Again, there's no cooperative element in this implementation, which breaks the framing of competitive vs. cooperative framing of the paper.

The system should implement a feedback mechanism (e.g., with a distance or progress bar visual indicator indicating the gap in distance or audio feedback) so that the players can still know the AI's progress even if the AI falls behind (line 229).

The 2 studies can be treated as 1 and the analysis can be done across the 4 conditions stated above. It appears that the ghost condition may outperform any of the AI-bot conditions (comparing with self or with ghost conditions of others in similar profile; see Quantified Self literature). This may be a stronger argument that the current framing of the paper.
Additional comments: Future work needs to account for novel effect and halo effect that could be present in the described studies.

This paper requires *very* major revision due to the inherent issues in framing and the way in which the procedures are described and the statistical tests are analyzed/presented.


Instruction: You are one of the reviewers, your task is to write a review for the article. You will be given the title of the article, the number of the round in which the article is located, and your order among the reviewers.

Input:
Title: COMPETITION AND COOPERATION WITH VIRTUAL PLAYERS IN AN EXERGAME
Review round: 1
Reviewer: 2

Output:
Basic reporting: This article is weel written. I just found one typo :

In the abstract : ... than playong on one's own with (a) the virtual trainer (delete a or the)

And an tiny error : line 110, authors mention three condition while there are only two.
Experimental design: The experimental design is well described. Results are clearly presented
Validity of the findings: Some choices could be explain :
Are 5 minutes enough to separate two sessions. I'm not expert in sport but it seems that this duration is short and even the control condition can decrease the fitness and then the performances of users. In the same line, even if different orders are proposed to different users,  it could be interesting to test a possible order effect on the data.

Why do you evaluate subjective feeling with only one question. Generally, a questionnary includes some redundant questions and an evaluation of the coherency of the different answers to these question is evaluated.

I would like to know which condition is used to record the ghost (is the user already with a trainer or not ?). This point could modify the behavior of the ghost and then, the results. Whatever the condition is, it is important to mention it and also to discuss the impact of the recording of the ghost on the results.

To finish, it is always more easy to compare conditions. However, independant conditions measurment can be suprising. It could be interessesting that the authors give their opinion on this point. What were the results if they had test only one condition without comparaison ?
Additional comments: Interesting work. I hope that you can take into account some of my propositions.


Instruction: You are one of the reviewers, your task is to write a review for the article. You will be given the title of the article, the number of the round in which the article is located, and your order among the reviewers.

Input:
Title: COMPETITION AND COOPERATION WITH VIRTUAL PLAYERS IN AN EXERGAME
Review round: 1
Reviewer: 3

Output:
Basic reporting: This is a promising paper reporting two cross-sectional studies investigated the effects of competition and cooperation on exercise performance in an immersive Virtual Reality (VR) cycle exergame. The results, as claimed by the authors, demonstrate that a competitive experience in exergaming is an effective tool to elicit higher levels of exercise from the player, and can be achieved through virtual substitutes (e.g., the ghost or virtual trainer) for another human player.

The introduction and motivation to the study can be better argued by reviewing more related studies and explaining the gaps in current research. It is unclear why the 2 studies need to manipulate the social factors of competition and cooperation, and encourage the player to maintain an appropriate level of exercise. Why are the study conditions considered in the 2 studies?

The paper is generally presented in an easy-to-read manner. The authors may like to check out in-text citation reporting format. Attention should also be paid to the use of  tenses when reporting studies conducted and review of literature, for example.

Attached are some comments made on the manuscript.
Experimental design: Broadly, I think the study design can be explained in more details:
- Why is the exergame selected?
- Why are the 3 conditions tested? What hypotheses are formulated?
- Explain the sampling profiles. Two studies had more males. Could the study findings be biased?
- Why are the measurement items selected? In the case of the "enjoyment" and "motivation" constructs, it is a one-item measurement. Is this sufficient?
Validity of the findings: As highlighted above, the experimental designs may lead to biases in the findings.

It would have been better to combine the 2 studies into a single study and the same subjects be tested under the conditions, so the findings can be more accurately interpreted.
Additional comments: This is an interesting paper and I think the authors are commended on embarking the 2 studies. If it could be carried out again, it is advised that the authors think about the hypotheses they would like to test. The recruitment of the subjects should also be more representative of the population group they intend to study and indicate the reasons for selecting this group to study.


Instruction: You are one of the reviewers, your task is to write a review for the article. You will be given the title of the article, the number of the round in which the article is located, and your order among the reviewers.

Input:
Title: COMPETITION AND COOPERATION WITH VIRTUAL PLAYERS IN AN EXERGAME
Review round: 2
Reviewer: 1

Output:
Basic reporting: See below.
Experimental design: See below.
Validity of the findings: See below.
Additional comments: I'm satisfied with the revised manuscript. I do wish that the authors would attempt to experiment with different types of competitive and cooperative elements in future studies, but that's beyond the scope of this paper. In its current form, I do think that the paper provides novel contribution to the exergame literature.

One minor suggestion: when reporting significant differences using superscripts in Table 1 and Table 2, superscripts should be used when a relationship a > b > c exists. In other words, superscripts should be omitted when no difference is present. The convention of using superscripts to denote a > b is similar to using asterisks to denote statistically significant comparisons. If there's no difference and a superscript "a" is denoted across all means, then it makes it extremely difficult to read.


Instruction: You are one of the reviewers, your task is to write a review for the article. You will be given the title of the article, the number of the round in which the article is located, and your order among the reviewers.

Input:
Title: PREDICTING JUDICIAL DECISIONS OF THE EUROPEAN COURT OF HUMAN RIGHTS: A NATURAL LANGUAGE PROCESSING PERSPECTIVE
Review round: 1
Reviewer: 1

Output:
Basic reporting: No comments.
Experimental design: This review is prepared by a referee without training in quantitative methods, in computer science, or in natural language processing, and as such should be read with an appropriately-sized grain of salt.
The description, under 'Data', of how cases were selected needs clarification in several ways (lines 165-182).
First, when it is said that Article 3 'Prohibits torture', are we to understand that the study does not cover the other prohibitions contained in Article 3 (such as the prohibition on inhuman treatment)? Precision is important in legal writing and it is important here.
Second, the number of cases seems much lower than what one would expect, and much lower than what a rudimentary search of the HUDOC database generates. For example, a basic HUDOC search of Article 6, in English, generates over 10,000 cases. Of those, HUDOC indicates there at least 8,000 violation cases and at least 900 non-violation cases. If the methodology at lines 172-177 is replicable, one would expect that this study would have included at least 1,800 cases in its study of Article 6 (being all the cases in the smaller class plus a randomly selected equal number of cases from the larger class). And yet here the number studied is just 80. It is hard to discern the reasons for the discrepancy between the draft article and the results of a basic HUDOC search.
Third, the reasons for choosing articles 3, 6, and 8 could be substantiated a bit more. Surely *all* of the ECHR rights may be regarded as "important human rights that correspond to a variety of interests" (lines 167-168)? Why focus on these three?
Validity of the findings: This review is prepared by a referee without training in quantitative methods, in computer science, or in natural language processing, and as such should be read with an appropriately-sized grain of salt.
The article claims that "there is a strong correlation between the actual facts of a case and the decisions made by judges" (lines 264-265). I have serious concerns about whether or not this conclusion is substantiated by the data which preceded the Discussion. First, it is unclear what the authors mean by "the *actual* facts" (line 265). Are the "actual" facts somehow different from "the facts"? Second,  it is not clear what the authors mean when they say "information available to the judges before they make any comments or decisions" (line 180). Are the authors implying that the judgments contain all the information available to the judges before they made their decision? If so, this would seem to be a misunderstanding of how courts work (surely, at the very least, one would want to look at the full written arguments of the parties, rather than simply the summaries of those written arguments that are contained in judgments?). Third, it seems to me that this study proves, at best, that there is a correlation between the facts *as described in the judgement* and the result of a case. There is a difference between "the actual facts of a case" and "the facts as they are described in the judgment of case". The article does not acknowledge this difference at all. This is a problem. I'm afraid that the authors seem to be under the impression that the facts section of a judgment is an objective scientifically-established recitation of the facts. Unless the authors are aware of ECtHR practice that I am unaware of, this seems dangerously naive. On my understanding, the judgments of the ECtHR are prepared by the judges, their assistants, and the Court Registry. In any court anywhere around the world, including the ECtHR, it would not be unusual in the slightest for the judges, the assistants, or the registry, to frame the facts in light of their full understanding of the case (which would include their view on whether or not there is a violation). Facts sections of judgments are not peer-reviewed scientific papers. They are subjective summaries of the facts, including what the authors think is relevant and what they think is irrelevant. If a judge/judicial assistant/registrar is of the preliminary view that a violation is likely, it would not be at all unusual for them to frame the facts differently than how those same facts would be framed if they were of the view that a violation was unlikely.  This raises a problem: the article involves the authors taking the facts section of a delivered judgment, and then predicting whether or not that same judgment will result in a violation or not. This may be useful, but it does not seem to be the same as "predicting judicial decisions...using only the textual information available to the judges before they make any comments or decisions about a specific case..." (lines 312-313). The model does not seem to provide any capability for ex ante prediction -- i.e. it does not allow the result of a judicial decision to  be predicted until the facts section of the judgment can be analysed (and the facts section of the judgment cannot be analysed until the judgment is handed down). Surely this limits its utility?
Perhaps I misunderstand the mathematical value of the study; perhaps I misunderstand the internal workings of the European Court. But even if I am wrong on the maths or on the workings of the Court, the article needs to be considerably clearer about what it is predicting and about the nature of how judgments are written and prepared. Without that, it is hard to attach too much significance to its findings, I'm afraid.
Fourth, the authors claim that their study amounts to support for legal realism over legal formalism (line 322). This may be so, but a much more sophisticated account (than that at lines 28-37) of the debate about realism and formalism would be needed to draw much of a conclusion here.
Additional comments: No additional comments


Instruction: You are one of the reviewers, your task is to write a review for the article. You will be given the title of the article, the number of the round in which the article is located, and your order among the reviewers.

Input:
Title: PREDICTING JUDICIAL DECISIONS OF THE EUROPEAN COURT OF HUMAN RIGHTS: A NATURAL LANGUAGE PROCESSING PERSPECTIVE
Review round: 1
Reviewer: 2

Output:
Basic reporting: This paper is about applying the state-of-the-art natural language processing analysis and a machine learning algorithm to build a binary classifier to predict judicial decisions.

The paper is clearly written and easy to understand. The Authors have followed the appropriate document structure and the submission is self-contained.
The authors had made the data set annotation public-ally available, and the content of the cases can be download from the European Court of Human Rights (EctHR) website.
Experimental design: The research question is clearly defined: it is possible to use text processing and machine learning to predict whether, given a case, there has been a violation of an article in the convention of human rights. The research question is certainly relevant, and the results are interesting for the natural language processing and machine learning community, but it’s unclear how these findings are useful for the law and human rights community, since nothing is mentioned in the paper with regards to this question, for example, 'it would be useful to apply this kind of classifier as a tagging tool for highlighting cases in which violation of human rights are likely to be true? perhaps as a prioritizing or filtering means?'
This is more a curiosity than a negative comment.

From the natural language processing and machine learning point of view, the methodology is  correct. The authors use well known features as bag-of-words and topic models as well as an appropriate classifier (SVM).
Nevertheless, there are a few weak points that are not explained or addressed in the paper.
- Topic models: in spectral clustering, the number of topics is input parameter, but nothing is mentioned about how the value of this parameter, in this case 30 topics, was chosen.
- I was expecting to see experiments using both set of features, bow and topics, but results are only reported for experiments using one set of features at a time. Why are there no experiments that combine both features sets? If the performance was lower than when using the individual features sets, the outcome is still useful for the community and should be reported.
Validity of the findings: Accuracy is reported as the evaluation metric used to measure performance, but nothing is said about how accuracy is calculated, the formula or an explanation would be helpful. I'm assuming accuracy should be understood to mean: the proportion of true outcomes (true positives and true negatives) among the total number of cases. Please, clarify this.

In the discussion section, the authors gave examples how topics aligned with the theme of some of the cases, but it’s hard to understand if those examples are from the dataset used or from another dataset. I figured it out by exploring the dataset itself. A line clarifying this would be helpful.

Comment something about the cases that were wrongly classify, for example, is there any commonality between the wrongly classified instances? What does it mean for the law community to have more than 20% of its cases wrongly classified?
Additional comments: The topic of the paper is very interesting and the paper is easy to follow, but I would like to read about how this results can be use by the law community.


Instruction: You are one of the reviewers, your task is to write a review for the article. You will be given the title of the article, the number of the round in which the article is located, and your order among the reviewers.

Input:
Title: PREDICTING JUDICIAL DECISIONS OF THE EUROPEAN COURT OF HUMAN RIGHTS: A NATURAL LANGUAGE PROCESSING PERSPECTIVE
Review round: 2
Reviewer: 1

Output:
Basic reporting: No comments.
Experimental design: I have been asked to re-consider a revised draft article entitled ‘Predicting Judicial Decisions of the European Court of Human Rights: A Natural Language Processing Perspective’ for ‘PeerJ CS’.

As with the earlier review, this review is prepared by a referee without training in quantitative methods, in computer science, or in natural language processing, and as such should be read with an appropriately-sized grain of salt.

The authors have responded to feedback thoughtfully, which must be commended. But I am afraid that, subject to the caveats above, the article’s legal analysis remains problematic. I don't wish to be unkind or unfair with any of these comments: I look forward to a day when this sort of model can succeed. But this paper needs further work.

(all line references are to the track-changes pdf version)

At lines 88-94, the authors suggest the model can ‘be used to rapidly identify cases’, perhaps in ‘submitted cases’. It remains fundamentally unclear to me what text the authors imagine the model will be operating on to make ex ante predictions, given that it works solely on text in completed judgments. Is it not at least worth *identifying* what the text is that the model would operate on in order to make its ‘rapid’ ex ante predictions? Otherwise, the risk is that the argument will appear to be limited to this: ‘Our model allows us to analyse how the facts are summarized in published cases and predict how, later in that very same published case, the case will be resolved’. This may be interesting in itself, but I’m not sure it’s what the authors want the model to do.

The new section at lines 140-166 seems problematic, unless I have misunderstood key aspects of it.
*First, the authors say that the ECtHR has very limited fact-finding powers (true). But the authors then move on, without citing any authority, to say that this means that ‘in the vast majority of cases, [the ECtHR] will defer…to the judgments of domestic courts that have already heard and dismissed the applicants’ complaint’. This is problematic in various ways (not least that it implies that the domestic courts hear and dismiss complaints on the same legal questions that the ECtHR does, which seems to suggest the ECtHR is an appellate court – more on this below). More problematically, the authors’ logic is unclear: why wouldn’t the ECtHR defer to the summary of the facts prepared by (e.g.) the government lawyers? Moreover, even if it were true that the Court defers in this manner ‘in the vast majority of cases’, surely it should not be difficult to find a range of law journal articles and analysis supporting this legal or procedural proposition? And finally, this logic would suggest that any predictive model should look principally or exclusively at the domestic courts' factual summaries, would it not?
*Second, the authors say that ‘the Court cannot openly acknowledge any kind of bias on its part’ and therefore ‘on their face, summaries of facts…have to be at least framed in as neutral…’. The significance of this point is unexplained. Are the authors arguing that the Court does, in reality, prepare neutral summaries? Or that it may well be biased but that it must hide that bias? How does this help the argument about making ex ante predictions of any sort? Moreover, the authors do not seem to appreciate here that outright bias is only one problem: the bigger problem for their argument is the possibility of perfectly rational differential emphasis by the judges/registry/etc of facts that they know will be significant *because they are also involved in reaching the legal conclusions*. Unless I have misunderstood the ECtHR's procedure, in which case I apologize.
*Third, the absence of disputes before the ECtHR about the facts does not mean that there cannot be different facts emphasized or prioritized by the judges/registry in light of the analysis that they most likely know will follow.
*Fourth, the authors say ‘the “Circumstances” subsection is the closest (even if sometimes crude) proxy we have to a reliable textual representative of the factual background of a case’. Perhaps this is so, but one may wonder about what ‘reliable’ is worth here without a clearer sense of the model's utility. Surely the facts as summarized in government and/or applicant arguments might be worth a look if the goal is to assist the Court/lawyers in making ex ante predictions about how cases will be decided, even if they are not 'reliable' in the way a peer-reviewed paper might be reliable?

At several points (line 325, line 340), the ECtHR seems to be referred to as an appellate court. It is not an appellate court. This means, (1) the range of orders and remedies available to the ECtHR are not those of an appellate court, with consequences for its analysis; (2) the ECtHR will frequently be applying different *legal* tests to those applied by the domestic appellate courts (eg, ‘was there a violation of Art5 or Art6?’ vs ‘was the conviction unsafe?’), and (3) therefore different *facts* or different emphasis on those facts may be of interest to the ECtHR than those that were of interest to the domestic court.
Validity of the findings: See above.
Additional comments: Please let me add: this is really creative and interesting work and I hope that it will be able to be developed to a point where it is of utility to scholars and lawyers in all disciplines. Perhaps I misunderstand the mathematical value of the study; perhaps I misunderstand the internal workings of the ECtHR; but at present I am afraid I simply do not think the legal analysis is sufficient.


Instruction: You are one of the reviewers, your task is to write a review for the article. You will be given the title of the article, the number of the round in which the article is located, and your order among the reviewers.

Input:
Title: PREDICTING JUDICIAL DECISIONS OF THE EUROPEAN COURT OF HUMAN RIGHTS: A NATURAL LANGUAGE PROCESSING PERSPECTIVE
Review round: 2
Reviewer: 2

Output:
Basic reporting: The authors of the papers have answered the  reviewers comments in a satisfactory manner.
Firstly, and more important, relevant concerns from reviewer 1, who have a strong background on Law were addressed, especially in nuancing the conclusion about the correlation between the Facts and the Court decision and using a more precise legal writing along the paper.
Also, clarification about the Facts section and its subsection ‘Circumstances of the case’ make the paper more convincingly and easy to follow.
From the technical part, clarification on the evaluation metric used and further experimentation with combined features sets were carry out,  making the paper more substantial and methodologically stronger, and improving the overall accuracy of the predictor.

The authors commented during the review process that there are some barriers for accessing the data, from the EctHR portal. Besides, the authors mentioned that accessing cases from domestic courts is not straightforward either.
I would like to see a comment regarding data access issues, perhaps a sentence or two in the conclusion section. Are data access issues stopping or slowing down emerging research as the one presented in this paper? Should cases in the  EctHR and domestic cases be easily and freely available? If that it the case, is the EctHR making progress in open their repositories for public good?

Looking forward to see this paper published in the journal.
Experimental design: No-comments
Validity of the findings: No-comments
Additional comments: No-comments


Instruction: You are one of the reviewers, your task is to write a review for the article. You will be given the title of the article, the number of the round in which the article is located, and your order among the reviewers.

Input:
Title: PREDICTING JUDICIAL DECISIONS OF THE EUROPEAN COURT OF HUMAN RIGHTS: A NATURAL LANGUAGE PROCESSING PERSPECTIVE
Review round: 3
Reviewer: 1

Output:
Basic reporting: Fine.
Experimental design: Fine, subject to the below.
Validity of the findings: Fine, subject to the below.
Additional comments: The caveats in the rebuttal and the changes made to the paper are interesting. The relevant limitations that are acknowledged -- those that make the paper reliant on crude proxies -- mean that the paper's arguments would struggle to be satisfactory in a legal paper or in legal argument. I understand that the methods adopted in this area may be different.

Four specific revisions:
(1) The the rule on exhaustion of domestic remedies seems to be  poorly articulated, or perhaps misunderstood, at line 152. In particular, its application at the domestic level seems back-to-front, at least as it is explained here. It should be revised.
(2) If there is literature to support the assumption made in lines 108-111, that literature should be noted. Or is it the literature in lines 134-173?
(3) The authors have explained eloquently the limitations on their methodology due to unavailable data. It is therefore jarring to see an expression of belief at line 116-117 ("we believe there is") that, surely, is unsupported due to those very limitations? Is belief the basis for scientific argument here?
(4) Line 416 says "Large repositories like HUDOC should be easily and freely accessible." This could be more precise: is HUDOC not easily and freely accessible? Isn't the authors' real concern with the fact that HUDOC is a "case law database" and not a "database of case law and other things"?

As I have said before, this is really creative and interesting work and I hope that it will be able to be developed to a point where it is of utility to scholars in all disciplines and to lawyers.


Instruction: You are one of the reviewers, your task is to write a review for the article. You will be given the title of the article, the number of the round in which the article is located, and your order among the reviewers.

Input:
Title: PREDICTING JUDICIAL DECISIONS OF THE EUROPEAN COURT OF HUMAN RIGHTS: A NATURAL LANGUAGE PROCESSING PERSPECTIVE
Review round: 4
Reviewer: 1

Output:
Basic reporting: Fine.
Experimental design: Fine.
Validity of the findings: Fine.
Additional comments: Fine.


Instruction: You are one of the reviewers, your task is to write a review for the article. You will be given the title of the article, the number of the round in which the article is located, and your order among the reviewers.

Input:
Title: COLLABORO: A COLLABORATIVE (META) MODELING TOOL
Review round: 1
Reviewer: 1

Output:
Basic reporting: The paper is well written, the literature is relevant. Some references to existing tools can be added and some improvements can be done to make the paper clearer (See general comments below). Figure 13 is too small. The tool was made available on Github, but I did not find the link to the update site zip file, nor how to install the tool from the sources without error on recent Eclipse versions.
Experimental design: The research question is relevant and well defined. It fits the scope of the journal.
Validity of the findings: The conclusion is appropriately connected to the research question. It is unclear if some points of the conclusion fall under already achieved work or perspectives (See general comments below).
Additional comments: This paper presents Collaboro, both a Domain Specific Modeling Language (DSML) and a tool for the collaborative definition of DSMLs by a community composed of developers and end-users. This paper states the importance of involving end-users in the development of DSMLs to receive their feedbacks early in the development process. Therefore, the proposed DSML introduces concepts for representing the collaboration by means of participations, votes and changes operated to the DSML definition. Changes can occur on either the abstract syntax (the metamodel) or the concrete syntax (the notation). To do so, both are represented using the Ecore language. The concrete syntax is represented by a notation metamodel defined by the authors. It allows for the definition of graphical, textual, and hybrid notations, which has not been addressed in the literature before, according to the authors. The authors also argue that the Collaboro DSML can also be applied for collaborative modeling (not only for collaborative DSML modeling), modulo some changes in the DSML metamodel definition. Finally, the paper also introduces a metric-based recommender which is used to evaluate the quality of the DSML regarding a set of applied metrics. This contribution is mostly presented for ensuring the quality of the notation, based on the Moody's nine principles.

The paper is well written, the structure is clear, and the running example helps understand the paper. While I found the paper a bit too similar with the paper published at the CAiSE conference in 2013, the authors clearly state at the end of the introduction the added contribution (metric-based recommender, web frontend, ...).

While I recommend this paper for being accepted, the following suggestions can help the authors improve the paper:

- In the introduction, I was confused about what Collaboro really is. The author should clearly state at the beginning that the name "Collaboro" designates both the DSML and the tool to support the collaborative DSML definition task.

- The paper proposes a renderer engine to render on-the-fly model examples that are used to enable collaborations between end-users and developers. While the authors claim that it is possible to render graphical models using SVG, it is unclear how it is done, the running example only showing the rendering of textual examples. In particular, it is not clear if specific automatic layout engines are used, nor if it is possible to render edges to connect shapes. Specially, the excerpt of the notation metamodel shown in Figure 4 does not contain a "connector" concept to connect two graphical elements. Please provide more details about the rendering on-the-fly of graphical models using the notation metamodel.

- The authors justify the proposition of a new notation metamodel as no generic metamodel covering both graphical and textual syntaxes has been proposed yet. While this claim may be true, I am not convinced about the need of re-creating a notation metamodel from scratch. Specially, some graphical notation metamodels already exist -- such as the Pictogram Metamodel of Graphiti (https://eclipse.org/graphiti/images/pictograms.pdf) -- and are more complete, extensible and also based on EMF / Ecore. I understand that the purpose of the notation metamodel coupled to the renderer engine is not to provide a full modeling environment, as stated by the authors. Nonetheless, some concepts, such as anchors and connectors are as important as the other proposed graphical concepts and should also be discussed during the collaboration phase between end-users and developers. Let's imagine for example a case study where the position of a port (N, W, E, S) conveys important information about the nature of that port. Such concepts seem to be missing from the proposed metamodel (or at least from the excerpt in Figure 4). It could have been possible to extend existing notation metamodels to add the missing notation concepts instead of re-creating an entire notation metamodel from scratch. Please strengthen the discussion about the choice that has been made regarding existing notation metamodels.

- On Figure 4, only the textual part of the hybrid notation seems to be connected to the abstract syntax by means of EReferences and EAttributes. Could it be possible to connect all NotationElements to the abstract syntax ? Figure 4 suggests only textual element can refer to the abstract syntax while Figure 6 suggests that all notation elements can be connected to the abstract syntax (through the "maps" relationship). Would it be possible to elaborate that ? It is very common that a value of a graphical attribute is not simply defined by a primitive value such as an Integer or a String, but instead is computed regarding the properties of the model element. For example, let us consider a traffic light where the value of the "fillColor" attribute of each circle (green, orange, red) is computed regarding the state of the traffic light.

- I am not sure to understand how and where a model which conforms with the notation metamodel is created. Does Collaboro provide a specific wizard, e.g. an EMF tree-like plugin or a more advanced wizard to model and render the notation ? Is the rendering re-generated each time a change to the notation model occurs ? How can Collaboro be compared to existing tools such as Eclipse Sirius, that allows for the re-rendering of a diagram every time a change to the notation occurs ?

- The second paragraph of the conclusion section is really interesting, but I am not sure whether it is stated as a future work or if the automatic creation of Xtext or GMF configuration files has already been achieved. Whether it is already implemented or not, it could be a good idea to insert this discussion in the section "Making DSML development Collaborative" page 4.

Some minor comments:

- Figure 13 is too small ;
- Sections are unnumbered so several references to these sections are missing (e.g. at the end of the first paragraph of subsection "Renderer", page 7) ;
- The "GPL" acronym appears in the introduction and only defined page 11, while general-purpose modeling language was written out in full before the first use of GPL. Please use between GPL / GPML, DSL / DSML and define the acronyms before their first use.


Instruction: You are one of the reviewers, your task is to write a review for the article. You will be given the title of the article, the number of the round in which the article is located, and your order among the reviewers.

Input:
Title: COLLABORO: A COLLABORATIVE (META) MODELING TOOL
Review round: 1
Reviewer: 2

Output:
Basic reporting: Overall, the paper is nicely written and the used language is appropriate (sometimes it
has a little bit the flavor of spoken language) and clear.  The introduction gives a good
overview on the context and adequate references to related work. The background
section called „Collaborative (Meta)Modelling“ gives a good overview on the setup of
the research. A running example of a baggage claim system on an airport is considered.
Then an imaginary collaboration scenario is discussed followed by a description of the
tool followed by some application scenarios.

The paper has huge overlaps with the paper „Enabling the Collaborative Definition of DSMLs. International Conference on Advanced Information Systems Engineering, Jun 2013, Valence, Spain. 2013“ by the same authors. This is understandable to a certain extend, because this
work aims to be an extension. However, is it really necessary to copy the abstract and
just append a few more words? Further, the lessons learned of this paper basically refer to the
lessons learned in the other paper (plus the insight that a web-based front end would be
useful). Also several figures are either reused or are very similar. The extensions in this work
cover mainly features like the support for graphical DSMLs, a web-based front-end
and  a metric-based recommender. Especially the later is a very interesting extension
and offers a lot of potential - here I would like to see more on how the recommender
actually works - the paper mainly focuses mainly on the description of the several metrics
without giving a real comparison.

Some detailed comments:

It would better connect the first and second sentence of the abstract, e.g, Software
development processes are collaborative in nature, in particular the input of the
end-user is indispensable.

In general, most references need better integration into the text, e.g., Page 2, lines 49, 78, 93
Page 1, line 44: abbreviation GPL has not been introduced
Page 3: semantics … has been considered as part of future work -> will be considered

Figure 1: It is not entirely clear how example of abstract syntax relates to the example of
concrete example. For me the concrete syntax example looks like an instance of the model
shown in (a).

Some more details of the recommender system would be of interesting. For me it does not
come out clearly how much impact it really has on the development process. For the
recommender it should be feasible to setup interesting experiments accessing its quality
in an objective and reproducible manner.
Experimental design: The paper describes a tool and its implementation but not really an evaluation under an
experimental setup. If the paper would have such an evaluation (maybe in an educational
setting if no real-world project is at hand), the contribution would be much stronger.
Validity of the findings: Not empirically proven, but intuitively valid.
Additional comments: Basically, this kind of work is very important - collaboration is big challenge in modeling
and probably also in the development of DSLs. My main concern is that (1) it is too close to the
CAISE paper - here the additional scientific contributions have to be more explicit and (2)
there is no rigorous and reproducible evaluation of the approach. I know this is difficult
to evaluate if people are involved, but this is part of the scientific challenge. Intuitively, I
agree with most claims and observations made in this paper, but it would be important


Instruction: You are one of the reviewers, your task is to write a review for the article. You will be given the title of the article, the number of the round in which the article is located, and your order among the reviewers.

Input:
Title: COLLABORO: A COLLABORATIVE (META) MODELING TOOL
Review round: 2
Reviewer: 1

Output:
Basic reporting: The paper is well written, the literature is relevant. Some references to existing tools have been added and the some improvements have been done to make the paper clearer since the first revision. I could have access the update site and could have installed the tool on Eclipse Mars. However, I could not have replicated the running examples nor find the examples relevant w.r.t. how rendering on the fly notation models. A tutorial would be interesting to provide.

All-in one, I think the paper is interesting and fits the PeeJ Computer science journal, modulo some updated comments (see General Comments for the Author).
Experimental design: The research question is relevant and well defined. It fits the scope of the journal.
Validity of the findings: The conclusion is appropriately connected to the research question. The authors clarified how the different points of the conclusion fall under already achieved work or perspectives.
Additional comments: This paper presents Collaboro, both a Domain Specific Modeling Language (DSML) and a tool for the collaborative definition of DSMLs by a community composed of developers and end-users. This paper states the importance of involving end-users in the development of DSMLs to receive their feedbacks early in the development process. Therefore, the proposed DSML introduces concepts for representing the collaboration by means of participations, votes and changes operated to the DSML definition. Changes can occur on either the abstract syntax (the metamodel) or the concrete syntax (the notation). To do so, both are represented using the Ecore language. The concrete syntax is represented by a notation metamodel defined by the authors. It allows for the definition of graphical, textual, and hybrid notations, which has not been addressed in the literature before, according to the authors. The authors also argue that the Collaboro DSML can also be applied for collaborative modeling (not only for collaborative DSML modeling), modulo some changes in the DSML metamodel definition. Finally, the paper also introduces a metric-based recommender which is used to evaluate the quality of the DSML regarding a set of applied metrics. This contribution is mostly presented for ensuring the quality of the notation, based on the Moody's nine principles.

*** UPDATE ***

The authors clarified most of the points of the initial review. I agree with the answer and the changes made on the paper about the notation metamodel (Answer R1.5). I still find the answers discussion on the rendering of graphical notations half-convincing (Answers R1.3 and R1.6). It is something I was unable to test in the tool, and no example in the Git repository seems to address the creation of graphical notations.

While I think that the paper can be accepted for publication in the PeerJ Computer Science journal, I really think that the authors could benefit from illustrating the paper with examples of both the textual and the graphical notations. As is, some claims about the rendering on-the-fly, the presence of auto-layout engines and so on cannot be validated or tested over the tool. I would find appropriate for the authors to add an illustration (maybe alongside Figure 12b) coming from their tool to represent what a graphical notation example of the Baggage Claim case study would be. It is maybe what is represented in Figure 13, but the pictures in the two upper areas (Abstract Syntax and Concrete Syntax Examples) are still very small and unreadable (even after zooming on the computer).

Apart from that, I think that the fourth feature claimed in the introduction (support for collaborative modeling) is a bit of over statement as, as mentioned in the paragraph below, it is an adaptation envisioned by the author (which has consequently not been implemented yet, so it cannot be considered as a feature). I would suggest the authors to remove it.


Instruction: You are one of the reviewers, your task is to write a review for the article. You will be given the title of the article, the number of the round in which the article is located, and your order among the reviewers.

Input:
Title: MACHINE LEARNING CAN DIFFERENTIATE VENOM TOXINS FROM OTHER PROTEINS HAVING NON-TOXIC PHYSIOLOGICAL FUNCTIONS
Review round: 1
Reviewer: 1

Output:
Basic reporting: The authors present here a meta-approach that combines 9 different machine learning tools for the classification of toxic proteins from their non-toxic counterparts.
The text is well written and the rationale behind the method is expressed in a clear and concise manner.
Experimental design: Results:
In line 111, the authors state that "any blastp hits with duplicated FASTA sequences in the Uniprot/SwissProt-ToxProt database were discarded". Do the authors allow for any variability or do they refer to identical (100%) sequences here? Similarly, the authors should indicate how they handle internal homology in the training sets. The manuscript lacks a detailed description of the creation of the k-folds. My concern is that the authors might be incurring in overtraining if similar (i.e. non-identical globally but closely identical in the tox-bits) signals stemming from closely related proteins are considered in the training and the evaluation sets. And of course, it hinders reproducibility.

The threshold for the selection of valid results is decided after the training, and therefore, possibly introducing overfitting. The authors correctly incorporate a leave-one-out approach in the 25-fold crossvalidation and separate the dataset into training and evaluation. That would be a valid approach if there was no parameter selection in the assessment. As the threshold (i.e <3 non-toxic, 4-5 pot. toxic, 6> toxic) is used afterwards, the distribution of the cross-validation should include three sets: training, test and evaluation. A similar reasoning applies to the selection of the 9 internal methods for classification, as feature selection occurred after the training.

Although the authors correctly include commonly used statistics for the evaluation of machine learning approaches in the Supplementary Note 1B, the numbers for these are not provided and the results are discussed mainly in terms of accuracy. In the problem at hand it is interesting to now the accuracy of prediction on the positive set but also the ability to recall sequences.

Similarly, Figure 1 should be reconsidered. Probably providing together specificity and sensitivity (i.e. a ROC curve). The legend could still be used to display the performance for each cut-off valies and the values for the naiveBLAST and oneBLAST methods. From the observation of the figure, it is not clear to me how the performance was calculated in the manuscript. Again, the all statistics measuring performance are needed. In particular, those focusing on the positive (i.e. toxic proteins) part: precision, recall, f-score and MCC to cite some examples. From the observation of the figure, it seems that the authors present a method very good at predicting the negative class (non-toxic, majority class) but with low recall in the positive class (toxic).

In general, comparisons in the manuscript should be done with respect to precision (accuracy on the positive class) rather than overall accuracy as the prediction of non-toxic proteins is not relevant in this case and constitutes a big majority of the training set.

The same applies to Table 1. Please, report all statistics for all the methods to facilitate comparison. Or even include them in Figure 1 for direct comparison.

In Table 3, please, provide the individual scores.
Validity of the findings: In the paragraph starting in line 246 the authors describe other approaches that, although limited in scope, address a similar problem. However, the performance of these specific method is not critically compared to the performance of the ToxClassifier presented here. The readership of the manuscript would definitely benefit from a detailed benchmark with these in addition to the other sequence based generic approaches.
Additional comments: The Methods section needs improvement to reach publication standards. I would suggest that the information in the Supplementary Notes is expanded and included into the Methods section. In particular, the sections referring to the different annotation systems and the statistics used for the evaluation of the methods constitute a core part of the interpretation of the results and should be described carefully in the methods sections. I mentioned above some problems with respect to the generation of the training and evaluation sets in the k-fold crossvalidation approach used. This should constitute a section in the Methods as well.

In line 49, the sentence "Thus, remote .... approach alone" might need rewording. The meaning is not clear to me in its current form.


Instruction: You are one of the reviewers, your task is to write a review for the article. You will be given the title of the article, the number of the round in which the article is located, and your order among the reviewers.

Input:
Title: MACHINE LEARNING CAN DIFFERENTIATE VENOM TOXINS FROM OTHER PROTEINS HAVING NON-TOXIC PHYSIOLOGICAL FUNCTIONS
Review round: 1
Reviewer: 2

Output:
Basic reporting: The paper is well written and the aims are clearly stated. The introduction and background provided are adequate. Figures are relevant and the explanation that is included on them is enough to understand their content.

The literature included is relevant and up-to-date.
Experimental design: The paper presents the results of the use of standard machine learning classification methods to distinguish between toxic and non-toxic peptides and proteins.

The experiments are well designed and the databases are constructed in a principled way. However, there is a lack of comparison with other relevant programs aimed at the same goal which are described in the paper but not used as a baseline to evaluate the validity of the proposed approach.

From the computational point of view not sufficient information is given about how the models were trained or how the hyper-parameters of the models where chosen. This would make the reproduction to the experiments troublesome.
Validity of the findings: Although the experiments and results are valuable, I think the novelty of the paper is weak. The application of standard machine learning methods to toxic/non toxic protein classification is interesting but it is not a clear innovation over other existing programs that use other similar techniques.

I think that the author should justify the contribution of their paper to the field in addition to the seemingly good results achieved.
Additional comments: No additional comments


Instruction: You are one of the reviewers, your task is to write a review for the article. You will be given the title of the article, the number of the round in which the article is located, and your order among the reviewers.

Input:
Title: MACHINE LEARNING CAN DIFFERENTIATE VENOM TOXINS FROM OTHER PROTEINS HAVING NON-TOXIC PHYSIOLOGICAL FUNCTIONS
Review round: 2
Reviewer: 1

Output:
Basic reporting: The authors have carefully addressed my concerns to the previous version of the manuscript. Therefore, I have no more objections to its publication.
Experimental design: No Comments
Validity of the findings: No Comments
Additional comments: No Comments


Instruction: You are one of the reviewers, your task is to write a review for the article. You will be given the title of the article, the number of the round in which the article is located, and your order among the reviewers.

Input:
Title: LARGE-SCALE COMPARATIVE VISUALISATION OF SETS OF MULTIDIMENSIONAL DATA
Review round: 1
Reviewer: 1

Output:
Basic reporting: The article is clear and has a correct structure. Motivation is duly substantiated and related work is extensive and updated.
Experimental design: Section 3 is somewhat generic in terms of the design description. Section  4, and more specifically 4.3, provides much space to the description of the control software implementation, but only describes a few examples.
It could be explained in this section  the appropriate set of tools and  functionalities for the analysis of the two proposed problems: Magnetic Resonance Imaging (MRI) data from IMAGE-HD and large-scale systematic morphological classification of the kinematic structures of galaxies.
Regard this second problem  is cited in the article but later nothing is explained about him.
Validity of the findings: The developed system, ENCUBE,  has some excellent features such as:
It allows comparative visualization and analysis of large amounts of data, actions aplied to data cubes in parallel, large display visualization area, very high processing power, collaborative workspace, stereoscopic capability and workflow serialization.
Additional comments: It is a good tool for scientific visualization.


Instruction: You are one of the reviewers, your task is to write a review for the article. You will be given the title of the article, the number of the round in which the article is located, and your order among the reviewers.

Input:
Title: LARGE-SCALE COMPARATIVE VISUALISATION OF SETS OF MULTIDIMENSIONAL DATA
Review round: 1
Reviewer: 2

Output:
Basic reporting: No Comments.
Experimental design: The submission must describe original primary research within the Scope of the journal. -> Ok
    The submission should clearly define the research question, which must be relevant and meaningful. The knowledge gap being investigated should be identified, and statements should be made as to how the study contributes to filling that gap. -> There is no research question.
    The investigation must have been conducted rigorously and to a high technical standard. -> There is no experiment.
    Methods should be described with sufficient information to be reproducible by another investigator. -> Ok
    The research must have been conducted in conformity with the prevailing ethical standards in the field. -> Ok
Validity of the findings: The data should be robust, statistically sound, and controlled. -> There is no data gathering.
    The data on which the conclusions are based must be provided or made available in an acceptable discipline-specific repository. -> All conclusions are speculative.
    The conclusions should be appropriately stated, should be connected to the original question investigated, and should be limited to those supported by the results. -> This is not ok in the writing.
    Speculation is welcomed, but should be identified as such. -> It is not identified as speculative.
    Decisions are not made based on any subjective determination of impact, degree of advance, novelty, being of interest to only a niche audience, etc. Replication experiments are encouraged (provided the rationale for the replication, and how it adds value to the literature, is clearly described); however, we do not allow the ‘pointless’ repetition of well known, widely accepted results. -> The conclusions are mostly subjective.
    Negative / inconclusive results are acceptable. -> There are no experiments.
Additional comments: The article describes encube, a data visualization deviced aimed at the exploration of large data.

There is no doubt that the authors have put great effort into this work, and that it seems to present relevant features. However, the text has some serious flaws. Most of them seem to be linked to the excessive description of the system, which makes the paper miss points that are relevant in scientific literature: evaluation and the detection of the contributions of the work using objective data.

These are my comments:

“For a comprehensive review of the variety of standard techniques see, for
example, Akenine-M ̈oller et al. (2008), Toriwaki and Yoshida (2009) and Szeliski (2010).”
> It would be very useful for the reader to have some (two? Three?) figures with examples of standard techniques and how they work. Also, why are there three review papers on the subject in three consecutive years? It can be the case that there was a lot of innovation in the field in these years. Could the authors state what is the contribution of each of these reviews?

“Structured three dimensional (3D) images or data cubes are ubiquitous in scientific research.”
> I disagree with that. There are many fields in science that are oblivious about data cubes. Again, some figures and a deeper discussion would greatly increase the appeal of this paper.

> Along “Related Work”: again, I miss some figures that could highlight the differences between visualization systems proposals and the preceeding ones. It would probably be a good idea to only select the two or three that provided inspiration for encube.

In Section 5.3
> Without an user study, all of these discussions are based on anedoctal data or speculation. This article needs an user study. My next note also regards this question.

> Although the authors state that there are three important questions (“1) how to integrate comparative visualisation and analysis into a unified system; 2) how to document the discovery process; and 3) how to enable scientists to continue the research process once back at their desktop.”). However:
> 1) I am not convinced that this was discussed along the article. Again, the reader needs at least some comparative figures displaying how encube is different from previous systems. Also, this only accounts for the proposal of the system: proving that this point was properly addressed requires an user study.
> 2) Again, an user study is necessary to validate the proposed system. How did users interact with the discovery process history?
> 3) By research, I am guessing the authors mean “discovery” or “exploration”? The server-based service is a good solution for this. The user study, however, should highlight whether the server-based approach actually improved the data exploration experience.

I believe that the core of the work (the proposed system) is of good quality, and that the authors should calmly address the issues above prior to publishing.


Instruction: You are one of the reviewers, your task is to write a review for the article. You will be given the title of the article, the number of the round in which the article is located, and your order among the reviewers.

Input:
Title: TCP ADAPTATION WITH NETWORK CODING AND OPPORTUNISTIC DATA FORWARDING IN MULTI-HOP WIRELESS NETWORKS
Review round: 1
Reviewer: 1

Output:
Basic reporting: This paper contains a lot of typos or grammatical errors to correct.
In line 54, showes -> shows.
In line 55, opportunisty -> opportunity.
In line 128, in a same batch -> in the same batch.
In line 158, a crucial importance -> of crucial importance.
In Figure 1, fowarding -> forwarding.
In line 236, initially -> initial.
In line 268, it -> its.
In line 306, each links -> each link
In line 307, ploted -> plotted.
In line 313, throughput -> throughputs.
In Figure 8, througput -> throughput.
Besides, there are articles missing in some cases. Please correct them.

The introduction of background knowledge and related work is thorough. Though it is a choice of style, I think that many citations in the first three paragraphs of the introduction can be moved to related work.

Fonts of numbers in Figure 5 look very small, and colors of TCP and TCPFender are hard to differentiate in Figure 6, 7, and 8, when printed in the greyscale. Also, the legend is missing in Figure 7.

The code and raw data relevant to the performance evaluation are not available.
Experimental design: In this paper, the authors propose a solution that incorporate opportunistic forwarding and network coding with the loss-based TCP congestion control algorithm. The studied question is defined clearly and interesting. The key idea of this solution is an adaption layer working between TCP and the network layer, which can encode and send multiple batches of data simultaneously at the source, and provide early congestion feedback at the destination.
Validity of the findings: In this paper, all nodes including the source, forwarders, and the destination need to store packets in multiple batches. It is not shown in this paper that how much more memory will be used to store packets from multiple batches, under different network conditions.

In the evaluation, a large file is transmitted and the throughput is the major metric measured. As the elephant flow is typically not sensitive to the delay, it would be interesting to investigate if TCPFender could improve the delay of mice flows that are not sensitive to throughput but delay.

Besides, as stated in the paper, opportunistic forwarding may introduce extra packets which can congest the network and hurt the actual goodput. However, it is unknown how many extra packets in total are transmitted through the network. Moreover, the reviewer suggests evaluating the performance with background traffic.
Additional comments: No additional comments


Instruction: You are one of the reviewers, your task is to write a review for the article. You will be given the title of the article, the number of the round in which the article is located, and your order among the reviewers.

Input:
Title: TCP ADAPTATION WITH NETWORK CODING AND OPPORTUNISTIC DATA FORWARDING IN MULTI-HOP WIRELESS NETWORKS
Review round: 1
Reviewer: 2

Output:
Basic reporting: Some important prior literature is missing. For example, the following paper also extends MORE to allow multiple batches to flow in the network:

[R1] Y. Lin, B. Li, and B. Liang, ``CodeOR: Opportunistic routing in wireless mesh networks with segmented network coding," IEEE International Conference on Network Protocols (ICNP), 2008.

In fact, the work of MORE has been cited over 1000 times. So, I believe that more papers should be discussed in the section of Related Work.
Experimental design: I believe that the concept of `innovative packets' should be explained in more detail, because it is an important concept in the design of TCPFender.
Validity of the findings: It is unclear that how many simulations have been conducted for each experiment. Without such information, it is hard to assess the statistical significance of the conclusions.
Additional comments: This paper proposes a new TCP scheme that supports opportunistic routing and network coding. The proposed scheme extends MORE in the following ways. First, the scheme applies accumulative coding so that the source can perform random linear network coding before it collects all the packets in a batch. Second, the scheme allows multiple batches to flow in the network. With these two extensions, the paper proposes a variant of TCP-Reno on top of MORE, which is called TCPFender.

However, I don’t think that the paper can be accepted in its current form. My concern is mostly about the novelty of the paper. First of all, the idea of `accumulative coding’ is well known in the literature of network coding. So, its application to multi-hop wireless networks is quite straightforward. Second, the idea of allowing for multiple batches in MORE has already been studied in several papers including [R1] mentioned above. Therefore, the proposed two extensions of MORE in the paper do not have sufficient novelty.

Another concern of mine is about the simulation part of the paper. In particular, the paper only compares its proposed scheme with MORE, but not with some extensions of MORE (such as CodeOR proposed in [R1]). Since the work of MORE receives more than 1000 citations, there must be plenty of good extensions in the literature. Some exemplary ones should be compared with the proposed scheme in this paper.

Therefore, I would like to suggest the authors to conduct a comprehensive literature survey on the extensions of MORE and then revise the paper accordingly.


Instruction: You are one of the reviewers, your task is to write a review for the article. You will be given the title of the article, the number of the round in which the article is located, and your order among the reviewers.

Input:
Title: TCP ADAPTATION WITH NETWORK CODING AND OPPORTUNISTIC DATA FORWARDING IN MULTI-HOP WIRELESS NETWORKS
Review round: 2
Reviewer: 1

Output:
Basic reporting: No comments.
Experimental design: No comments.
Validity of the findings: No comments.
Additional comments: The paper has been improved significantly and should be accepted except the following minor issues:

1.  The authors' response to my concern on the file size in the evaluation is
not fully convincing. The authors argue that the decoding delay is not overly
significant and can be comparable with the file-wise delay. This may still mean that
the additional latency to transmit a small file can be larger than TCP without
NC. A more explicit and clearer explanation is needed in the paper.

2. The font size in Fig. 5/6/7/8/11 is still too small, and
much smaller than the font size in other figures. Please update and make them
consistent. Besides, colors of TCP+NC and TCPFender in Fig. 6/7/8 still look similar in
greyscale.

3. Please correct the following typos:
line 25: became -> become
line 294: an -> a


Instruction: You are one of the reviewers, your task is to write a review for the article. You will be given the title of the article, the number of the round in which the article is located, and your order among the reviewers.

Input:
Title: TCP ADAPTATION WITH NETWORK CODING AND OPPORTUNISTIC DATA FORWARDING IN MULTI-HOP WIRELESS NETWORKS
Review round: 2
Reviewer: 2

Output:
Basic reporting: No Comments
Experimental design: No Comments
Validity of the findings: No Comments
Additional comments: I'm glad to see that some of my previous comments have been carefully addressed. For example, additional related works have been cited and discussed, and the important concept of `innovative packets' has been explained in detail.

To justify the novelty of the paper, the authors stated (in their response) that the main contribution of the paper is to ``design an adaptation layer over the network layer to support TCP-Reno without any change to TCP-Reno itself" instead of the two extensions of MORE (which have been proposed before). In addition, the authors claimed that their paper is the ``first paper that is designed to support TCP Reno by opportunistic data forwarding and network coding" in wireless multi-hop networks. This is a strong claim! However, this claim doesn't seem to be convincing. For example, the work of ComboCoding (by Chien-Chia Chen, Mario Gerla, and others) proposed a network-coding scheme that is ``implemented in the network layer" and is ``transparent to TCP." In particular, their scheme supports a variety of TCP protocols, including TCP-Reno and TCP-NewReno.

Therefore, I'd like to suggest the authors to compare their work with ComboCoding and other similar works.


Instruction: You are one of the reviewers, your task is to write a review for the article. You will be given the title of the article, the number of the round in which the article is located, and your order among the reviewers.

Input:
Title: OSOME: THE IUNI OBSERVATORY ON SOCIAL MEDIA
Review round: 1
Reviewer: 1

Output:
Basic reporting: This is a report on the creation of the Observatory on Social Media, an infrastructure a) retrieving and storing Twitter data (tweets), and b) offering free analytical services build on these data. The long term goal is to extend this service to a diversity of social media data (not just Twitter) in order to serve the needs of the public and social, behavioral and economic (SBE) research.
Experimental design: There is no experimental design.
Validity of the findings: As far as I am aware, this is the first time that such an infrastructure is made available to the public, through services accessible through human and programmatic interfaces. This is a significant achievement, given the size of the dataset (70B tweets, with collection ongoing) and given that a number of services over this data are provided in free access. This is a real and unique benefit offered to the public.
Additional comments: A couple of questions:
-	I could not find policy guidelines on the usage of tweets / twitter users objects returned by the tools. Twitter mentions restrictions of 50,000 objects made available per day (https://dev.twitter.com/overview/terms/agreement-and-policy , section 6 b). The paper does not explain where OSoMe stands in regard to this?

-	What is the relationship between OSoMe and Truthy? I understand that truthy is an ancestor to OSoMe, yet it seems to be more than that as the “botornot” service redirects to a truthy url. The footnote 1 in the paper does not explain the relationship clearly.

-	Development plans. The description of the API says it offers access to “timelines, active users, counts, or tweet ids.” Yet the API doc indicates less than that, which suggests there is plan to expand the API in the future. The paper could share this roadmap.


Instruction: You are one of the reviewers, your task is to write a review for the article. You will be given the title of the article, the number of the round in which the article is located, and your order among the reviewers.

Input:
Title: OSOME: THE IUNI OBSERVATORY ON SOCIAL MEDIA
Review round: 1
Reviewer: 2

Output:
Basic reporting: -	The article is well-written and clearly understandable. Figures are fitting the text.
-	The motivation of why such an observatory is useful and how and by whom it can be used is clear and I fully agree that an easily accessible and searchable (by API or visual interfaces) Twitter data set that is constantly updated can be very valuable for CS and the “SBE” sciences – and that such a service is currently not available in this form. However, novelty is not a deciding PeerJ criterion.
-	Regarding the PeerJ criteria, there are no clear research questions or hypotheses stated, rather, the software architecture and integrated Tools of OSoMe are described descriptively.
-	While the website lists some related software code, none is referenced in the paper, particularly not the source code for the actual data collection system. There is no data being analyzed in the paper.
Experimental design: - there is no experimental evaluation conducted
Validity of the findings: - no findings are reported (see lacking experiments)
Additional comments: ##Fit to Aim & Scope of PeerJ##

-	For me, this is not an “original primary research” paper as necessitated by the editorial criteria of PeerJ but what is sometimes called a “system paper”. There are no clear research questions or hypotheses stated, rather, the software architecture and integrated Tools of OSoMe are merely described descriptively. Most of the end-user tools have been previously proposed. There is not clear research question and no evaluation is carried out of any of the components of the system. I don’t see the research contribution of this paper; it’s the description of a platform. Even for a system paper, I would have expected some statistics on users, performance (e.g. how performant is the API, how many clients+requests does it handle), etc.
-	The backend seems to be well-designed and employs some state of the art technology but also doesn’t really strike me as a new (research) contribution.

##Representativeness of the offered data##

-	Given that research has pointed out (and is acknowledged by the authors) that collecting from Twitter’s APIs can introduce notable biases in what data is collected, I would have at least expected a discussion of this bias, coming from the Streaming API they use, as well the possible evolution of that bias over time, with changes implemented to Twitter’s technical infrastructure (the reported ratio change of geo-tagged Tweets might actually be an indicator for this, for instance).

-	It’s also not clear what the authors refer to when talking about that “the data has been collected from a random 10% stream sample of public Twitter posts”: I might be mistaken, but is the streaming API not only giving 1% of all Tweets? Has this changed over the years? The whole Tweet collection process and what fraction of Tweets is stored in OSoMe is not transparent at all (And, as has been rightfully argued in e.g. Ruths & Pfeffer, 2014, size does *not* automatically heal bias; this holds especially when studying smaller subpopulations with this data).

-	Optimally, readers and the users of OSoMe need to have an idea of what biases were captured with the collected data, as the resulting inferences drawn by users not familiar with the technical background of such a service, especially from the SBE sciences, can lead to grave misinterpretations. Not only is there no insight given into those possible biases or how they might have been addressed during data collection, the issue is basically not discussed at all for OSoMe.

-	On top of the collected data, the tools used to visualize it add another layer of selectively filtering certain parts and structures of the data. There is no evaluation provided in this or the related publications for the tools that actually measures how these filters distort the representativeness of the data.
o	Maps tool: only a very small percentage of tweets actually has geo-tags
o	Network movies: the algorithm behind this tool selects only certain nodes and edges ("most relevant part of the network") which is not evaluated in terms of the meaningfulness of the results it produces.
o	Network graph (static): only selects certain k-cores, for visualization purposes

-	Given that OSoMe is supposed to be a service for end-users that have little to no knowledge of the data collection and subsequent processing, i.e. more or less blindly trust the representativeness of this data at least with respect to the ground population of all Tweets, I find it very troubling that these questions are not tackled.

##Usability and usefulness of end-user tools##

-	Apart from any data issues, no evaluation is visible that shows that users (i) can operate, (ii) understand and appropriately interpret, and (iii) profit from using these tools. E.g., the network movies seems pretty confusing to me and I don't see how an average end-user can make much sense out of them. As the tools are advertised as a major part of this system, user studies are mandatory to see if they actually fulfill their goal.

-	Minor: The loading times were in my own several try-outs quite high for an end-user-centric site and certainly higher then advertised in the paper (e.g. Maps tool)


Instruction: You are one of the reviewers, your task is to write a review for the article. You will be given the title of the article, the number of the round in which the article is located, and your order among the reviewers.

Input:
Title: OSOME: THE IUNI OBSERVATORY ON SOCIAL MEDIA
Review round: 1
Reviewer: 3

Output:
Basic reporting: The paper presents the IUNI Observatory on Social Media, which is an open platform for the analysis of large collections of tweets. This platform aims to facilitate data analysis by providing retrieval, visualisation and analytics tools.

The paper is well written, including a good introduction and well-referenced background. They describe their system architecture and introduce some of the system's tools. While I find this system highly relevant to the research community, I wonder if this is the correct medium for this paper. The paper is presented as a general report of the system rather than a system that has been thoroughly evaluated. At the moment it looks more like a technical report than a research paper. There is no evaluation on any of the presented tools nor as a user-based evaluation or in terms of the system’s time-performance for processing big data.

In that sense there is no hypothesis to be proved in the paper, which makes the paper short as a journal publication.
Experimental design: There is no experimental design to judge for this paper.
Validity of the findings: The paper presents a description of the set of tools provided in their system. However, no evaluation on the performance of these tools was presented.
Additional comments: At the moment the paper is publishable as a technical report but not as a journal paper. I encourage the authors to gather the evaluations done on their system and describe such evaluations within this paper.


Instruction: You are one of the reviewers, your task is to write a review for the article. You will be given the title of the article, the number of the round in which the article is located, and your order among the reviewers.

Input:
Title: OSOME: THE IUNI OBSERVATORY ON SOCIAL MEDIA
Review round: 2
Reviewer: 1

Output:
Basic reporting: The authors provided adequate answers to my questions and remarks.
Experimental design: The authors provided adequate answers to my questions and remarks.
Validity of the findings: The authors provided adequate answers to my questions and remarks.
Additional comments: The authors provided adequate answers to my questions and remarks.


Instruction: You are one of the reviewers, your task is to write a review for the article. You will be given the title of the article, the number of the round in which the article is located, and your order among the reviewers.

Input:
Title: OSOME: THE IUNI OBSERVATORY ON SOCIAL MEDIA
Review round: 2
Reviewer: 2

Output:
Basic reporting: As before; alas article is now an "Application Note" and I suppose most of the guidelines for research papers do not apply in that case. However, I did not had the appropriate guidelines for Application Notes available.

One major PeerJ requirement I'm not sure about if it applies here and which is for the Editor to decide is this:
"All appropriate raw data has been made available in accordance with our Data Sharing policy." and further
"Your reviewers must have your raw data or code to review. Please submit:
As a link to a repository where the data is accessible.
Uploaded as a supplemental file.
Generally however, there are very few circumstances in which we can accept a manuscript without raw data. It is not required if your submission does not deal with raw data or code.
Some examples of invalid reasons for not submitting raw data or code:
- The data is owned by a third-party who have not given permission to publish it within this article. Please add a note to PeerJ staff to not publish the raw data alongside your article.
- The raw data is too large. Please upload the raw data to an online repository (e.g. Figshare, Dryad etc)."
--> If that is a requirement for Application Notes as well, this paper would not qualify as neither the source code of the system nor the data about the performance evaluation is available. (the twitter raw data is of course not possible to include)
Experimental design: -
Validity of the findings: -
Additional comments: Thanks to the authors for addressing most of my concerns and questions.

- As for my remarks regarding research questions, testing etc. required in a research article, these are basically nullified by this now being submitted as an Application Note, which I take to be the same as what is commonly referred to as a Systems Paper.

- Regarding the bias in the data I appreciate the very clear and honest disclaimer that was added in the "Data Source" section. This caveat of the system should also be made clear in any documentation and on the website for prospective (SBE) users.

- The evaluation gives some insight into the performance and future avenues for the architecture. I would've actually liked to see an even deeper discussion of the possible architecture improvements and current bottlenecks, since this is now a system paper/ app. note.

- There is no evaluation added on the usability/usefulness of the visualization tools, which is beyond the scope of an application note, as far as I can judge. However, I would appreciate a better outline of *how* the authors intend to evaluate these tools, not just a promise that they will.