Instruction: You are one of the reviewers, your task is to write a review for the article. You will be given the title of the article, the number of the round in which the article is located, and your order among the reviewers.

Input:
Title: INTEROPERABILITY AND FAIRNESS THROUGH A NOVEL COMBINATION OF WEB TECHNOLOGIES
Review round: 1
Reviewer: 1

Output:
Basic reporting: The article covers all the points required. The article is written in english and I guess that has been submitted using the available templates (not sure about that, I currently can't access the templates). The figures have enough resolution and are easy to read and are clear. I consider the submission self-contained and the raw data is within the paper or available online in the URLs provided.
Experimental design: The submission is original. As far as I know this paper has not been submitted elsewhere. The research question/aim is clearly defined and it is relevant and meaningful. Since it is provided a set of methodological steps/patterns they are all discussed with enough rigor and provided with useful examples. The methods are described with sufficient information (in most of the sections; in those that I consider that more explanations are needed, I've pointed out.
Validity of the findings: In this case we are talking about a set of patterns for data discovery, hence, classical validation from statistical point of view or similar according to the scientific classical guideliness are not applicable. However, the work presented is valid and the authors have demonstrated its value using a good explanation and also providing valid examples.
Additional comments: The paper provides a set of patterns for data discovery, accessibility, transformation and integration to be implemented in data repositories (general or specific ones; authors focused their work on life sciences) by means of following FAIR data principles. The overall goal of the paper is easy to understand and it is very interesting. The paper is easy to read and to follow the main ideas.
However, I've some specific comments and suggestions that I think that could be useful for the authors. I also potentially disagree with some of the opinions. Please note that my comments I wrote my comments as I was reading so maybe some comments in fact were solved/clarified later.
Authors make use of relatively new concepts within the semantic web area: RML and TPF. I think that more information in form of examples could be useful for the non-expert users to improve the understanding of the paper as well as these particular concepts.
In line 306 there is a sentence that said: “the lack of a novel API means that the information is accessible to generic Web-crawling agents, and may also be processed if that agent “understands” the vocabularies used”. Well, that it is clear and that is some of the main problems to deal with in the context of automatic retrieval, understanding and interoperability within the semantic web. My question is, is possible to address this problem with not-known vocabularies or remains as an open problem? What are the vocabularies to be used or where can they be retrieved? Just please explain a bit more regarding this.
In line 328 authors claim about the support of a multitude of source data formats (including for example, Excel, CSV, JSON, …) but all of this formats are structured formats (even Excel with its own API). However, is it possible to deal with binary files such as PDF that also need specific APIs to get the data and the result will be given, normally, in unstructured text? In line 336 the authors claim that the approach selected was based on the premise that data, in any format, could be metamodeled as a first step towards interoperability, and honestly I’ve serious doubts that this could be really done always, mainly based on the type of object (data source – file) to be used and/or the internal structure. More information regarding this possible limitation would be helpful to understand your approach.
The idea of the FAIR projectors is interesting itself, but I couldn’t catch the idea that you are proposing. In line 446 authors said: “what is required is a universally-applicable way of retrieving data from any transformation script (or any data source), without inventing a new API. We now describe our suggestion for how to achieve this behaviour, and we refer to such transformation tools as FAIR projectors”.  What exactly does the FAIR projectors tools? How are they able to universally transform the data retrieved from any transformation script or data source to the appropriate triples? I’m sorry but I’m a bit lost over here.
The examples provided regarding the FAIR Accessor from the Container Resource URL are quite nice. My only doubt is regarding when you call to a MetaRecord Resource. I tried to do the GET petition to the Container Resource and I successfully obtained the FAIR Accessor. From the FAIR Accessor I got what I understand/think is a MetaRecord (the first record in fact provided as result from the query provided in the Accessor (http://linkeddata.systems/Accessors/UniProtAccessor/C8UZX9 ). According to Figure 5, the MetaRecord should provide me some information for record C8UZX9, and it is true, this information is provided. But this information is provided after lot of triples that honestly I don’t know what they mean. Some of the triples seems to be related with the type of information obtained (taxon, organism, …) but some others are like this one:
<rdf:Description xmlns:ns1="http://semweb.mmlab.be/ns/rml#" xmlns:ns2="http://www.w3.org/ns/r2rml#" rdf:about="http://datafairport.org/local/MappingsFCCC8188-99F0-11E6-A61C-84165D07C3DD">
	<ns1:logicalSource rdf:resource="http://datafairport.org/local/SourceFCCC8188-99F0-11E6-A61C-84165D07C3DD"/>
	<ns2:predicateObjectMap rdf:resource="http://datafairport.org/local/POMapFCCC8188-99F0-11E6-A61C-84165D07C3DD"/>
	<ns2:subjectMap rdf:resource="http://datafairport.org/local/SubjectMapFCCC8188-99F0-11E6-A61C-84165D07C3DD"/>
</rdf:Description>

<rdf:Description xmlns:ns1="http://www.w3.org/ns/r2rml#" rdf:about="http://datafairport.org/local/ObjectMapFCCC90B0-99F0-11E6-A61C-84165D07C3DD">
	<ns1:parentTriplesMap rdf:resource="http://datafairport.org/local/SubjectMap2FCCC90B0-99F0-11E6-A61C-84165D07C3DD"/>
</rdf:Description>
And the URLs contained over these triples are not, currently, resolvable being difficult to get more information about them.
Ok, now I see that this information is part of the projector. It would be interesting to make some reference to this when the MetaRecord examples are provided because if the reader tries to get the data at the same time that he is reading, probably could get lost. In any case, I suppose that the maps should resolve, and they are not currently doing it.
Regarding the transformation performed by the projector, I’m not sure if I understand completely the aim. As far as I saw in the example, the projector is trying to transform the native RDF semantics from UniProt to EDAM. However, the RML is describing a triple where the subject type is ‘Protein Record’ and the object ‘GO Concept ID’. Where is exactly this transformation of semantics? As far as I understand is returning an upper-class type for both elements of the triple (subject and object) that match with the predicate considered (classifiedWith). Is that the aim? I was wondering to replace with the equivalent elements (if applicable) for those instances in EDAM. What is the reason of performing this transformation? Please clarify. I wonder if in this case it has been just done with this upper class because it is not possible to do it in other way, but in other cases can be achieved.
As a final comment, I would like to introduce some kind of more descriptive workflow figure with all the elements involved. At the end it is difficult to understand the relation of the different elements provided with the previous one, and probably this kind of figure would help to have a better understanding.
Regarding the conclusions, although I consider the initiative very interesting, I think that would be very difficult to achieve the goals covet by the authors. It is necessary the development of more easy and friendly tools that allows the researchers to not waste too much time in thinking how to deploy their data according to FAIR principles (even if journals are requiring it..), but that’s just my opinion.
Typos:
-	Please check Dimou et al. reference (it always appears without the year in the text as well as in the references list).
-	In “Metadata Interoperability – The “FAIR Accessor” and the Linked Data Platform” section please ensure that you define the acronym of Linked Data Platform (you describe the term and then use the acronym, but you don’t relate them). Also, in this first paragraph of the section there is a statement regarding LDP definition from https://www.w3.org/TR/ldp/ that has been written in italic and between quotes but there is no reference. I think that reference should be included.
-	Line 843: double efficient


Instruction: You are one of the reviewers, your task is to write a review for the article. You will be given the title of the article, the number of the round in which the article is located, and your order among the reviewers.

Input:
Title: INTEROPERABILITY AND FAIRNESS THROUGH A NOVEL COMBINATION OF WEB TECHNOLOGIES
Review round: 1
Reviewer: 2

Output:
Basic reporting: "No Comments"
Experimental design: The paper lacks serious discussion about previous work related to the topic. How the authors place their solution compared to previous projects. Was it possible that the FAIR principles would be achieved by a minor tweaking of current systems?
Validity of the findings: "No Comments".
Additional comments: Review of the paper titled “Interoperability and FAIRness through a novel combination of web technologies”

The paper address an important topic regarding the access and retrieval of data from heterogonous resources. The paper uses on the shelf technologies to achieve this task.
I have the following concerns regarding the paper.
-	The paper need to be re-organized.  I had to jump from one section to the other to understand the topic. The authors concern to address the FAIR principles lead to interruption of flow of ideas. Starting with the methodology section without concrete example made it difficult to follow the contribution. A concrete example given in the introduction would have been enough to show the challenge and how the solution would work.
-	I could not find serious discussion about previous work related to the topic. How the authors place their solution compared to previous projects. Was it possible that the FAIR principles would be achieved by a minor tweaking of current systems?
-	The suggested method is not really easy and straightforward to implement. What is the technological requirements at both the server and client side to achieve this? How this can be simplified?
-	What makes this method appealing for non-computer scientists? The use of different confusing technologies contradicts the concept of minimal design. Is there any solution to simplify this?


Instruction: You are one of the reviewers, your task is to write a review for the article. You will be given the title of the article, the number of the round in which the article is located, and your order among the reviewers.

Input:
Title: INTEROPERABILITY AND FAIRNESS THROUGH A NOVEL COMBINATION OF WEB TECHNOLOGIES
Review round: 1
Reviewer: 3

Output:
Basic reporting: The paper is well-written and the structure is clear and appropriate. The authors clearly highlight the relevance of the topic in both the abstract and the introduction, providing the reader with a precise context regarding the application domain.

Focusing on the Introduction, the authors might include more information about existing alternatives and previous attempts to provide interoperability and easy integration of data repositories for bioinformatics or other related disciplines. Actually, it is briefly mentioned in the Discussion (lines 709-717), but I would suggest providing further information in this regard as part of the motivation at the beginning of the paper.

Similarly, the last paragraph of the Introduction (l. 156-162) could be extended in order to focus the scope of the paper. For instance, a brief introduction to the basic foundations of the selected technologies and how them are applied to address the challenge of data integration and availability might be useful in order to provide the reader with a more detailed overview of the proposed solution. I think that both aspects (background and contribution) can strengthen an already good introduction.

In general, the figures provide illustrative examples of the concepts surrounding the different technologies. Only two of them need some changes. On the one hand, Figure 1 is located at the beginning of Section 2 (methods) but it is not explained in the context of that section. It is only referred in Section 3 (results), even though the figure does not contain specific information about the case study. Moreover, DCAT has not been explained at this point, so it can be a bit confusing to the reader. On the other hand, Figure 2 assumes that the reader have strong knowledge on the "Triple Descriptor" representation. In fact, it is a major issue throughout the entire paper, as I mention later (please, see general comments). I would suggest including a better explanation of the role of "subject", "predicate" and "object" (in general) and the meaning of the specific "maps" that appear in Figure 2.
Experimental design: The paper does not provide experimental results as such, but instead the authors present an illustrative example. The case study serves to complement the technological foundations of the proposed system, now explaining the meaning of specific data and metadata fields.

The introduction to the results might be improved if a brief description or external references to UniProt and EDAM are provided. In this sense, the paper assumes that the reader has high expertise on several semantic aspects (protein records, ontologies...), which might hamper its readability. In the same lines, the code example shown in lines 508-524 would be rather difficult to understand by readers without knowledge on query languages. It is a particular issue that clearly needs revision, as the role of SPARQL was not explained in Section 2 (methods).

Were these changes applied, the paper can reach the necessary compromise between domain experts and those readers closer to the computing field, being more self-contained as well. More details about the scope of the case study and the used materials could be provided for the sake of reproducibility.
Validity of the findings: The validity of the findings may be compromised due to the lack of experimental results. In this sense, providing public access to the case study should be considered, if possible. Even though the implementation is not complete, it would be really interesting to let readers "play" with the data and navigate across the different FAIR profiles. It will be also in line with the policy of the journal that promotes sharing data and materials.

As part of the conclusions, the authors might discuss the challenges behind the final implementation of the proposed architecture. It includes dealing with huge volume of data and guaranteeing quality of service, among others. Similarly, current limitations and lines of future work should be specified.
Additional comments: A major concern when reading the paper is that the authors do not explicitly mention the current state of the implementation. Although it looks promising and highly valuable to the research community, it seems that the solution is not fully functional yet. As I see it, the manuscript is a position paper presenting a first approximation to the operational solution. I believe that this is not a problem for acceptance, but the paper should not be ambiguous in this point.

It is worth mentioning that the paper provides a comprehensive justification of the technologies selected to build the system. It really helps to present the general idea behind the system and then go into more specific details. The only weak point of this approach I have found is that concepts like "subject", "predicate" and "object" in RDF/RML were not described somewhere in between. For instance, lines 361-365 mention some restrictions about "objects" and "subjects" in the context of RML maps, but the general meaning of these terms seems to come from RDF, which might be unknown for the reader. Similarly, the acronym [S,P,O] is used later (l. 452) to refer to triple pattern fragments (TPF), but understanding its relation to the aforementioned terms is not straightforward.

I find really interesting the idea of multiple FAIR profiles to describe the same data, so that the system can also integrate different data formats. As I understand it, different data users/providers could create new profiles for the same data resource, so a question here is whether the system can deal with possible inconsistencies between the information provided by different profiles. Similarly, some kind of control regarding who publishes new content and what is being published might be required in the future.

Following with this topic, the collaborative approach opens new possibilities to enrich the system, such as the inclusion of opinions about the quality of the data or a compilation of studies based on them.

Some minor comments:

- There is no publication year for reference (Dimou et al.) (l. 231, and others)

- Please, provide the full name or an external reference for acronyms: FoaF (l.551), SKOS (l.556) and BBMRI (l.883).

- Typos: furrther (l.582), "efficient" appears twice (l. 843).

- Line 634, the reference to Figure 6 seems to be incorrect, should it be Figure 7?

- Mixture of British and American spelling: behaviors (l. 58 and others) vs. behaviours (l. 895, l.908)


Instruction: You are one of the reviewers, your task is to write a review for the article. You will be given the title of the article, the number of the round in which the article is located, and your order among the reviewers.

Input:
Title: INTEROPERABILITY AND FAIRNESS THROUGH A NOVEL COMBINATION OF WEB TECHNOLOGIES
Review round: 2
Reviewer: 1

Output:
Basic reporting: The article covers all the points required. The article is written in english and seems to have been submitted using the available templates .The figures have enough resolution and are easy to read and are clear. I consider the submission self-contained and the raw data is within the paper or available online in the URLs provided.
Experimental design: The submission is original. As far as I know this paper has not been submitted elsewhere. The research question/aim is clearly defined and it is relevant and meaningful. Since it is provided a set of methodological steps/patterns they are all discussed with enough rigor and provided with useful examples.

The methods have been suffciently described in comparison with the previous versión. A replication of this work seems to be not applicable based on the paper aim, but possible with the original data.
Validity of the findings: In this case we are talking about a set of patterns for data discovery, hence, classical validation from statistical point of view or similar according to the scientific classical guideliness are not applicable. However, the work presented is valid and the authors have demonstrated its value using a good explanation and also providing valid examples.
Additional comments: The authors have taken into account my comments and have provided a good explanation to my doubts and inquiries. From my point of view and after also taking a look to the other reviewers from my side the paper it is currently ok and can be published. I really appreciate the efforts made by the authors to signifcantly improve the manuscript.


Instruction: You are one of the reviewers, your task is to write a review for the article. You will be given the title of the article, the number of the round in which the article is located, and your order among the reviewers.

Input:
Title: INTEROPERABILITY AND FAIRNESS THROUGH A NOVEL COMBINATION OF WEB TECHNOLOGIES
Review round: 2
Reviewer: 2

Output:
Basic reporting: This version of the paper is an improved version of previous submission. The authors could address the comments in a good way.
Experimental design: NA
Validity of the findings: NA
Additional comments: This version of the paper addresses the comments given in previous submission. There are still challenges intrinsic in the topic addressed but this beyond the solution the authors provide..


Instruction: You are one of the reviewers, your task is to write a review for the article. You will be given the title of the article, the number of the round in which the article is located, and your order among the reviewers.

Input:
Title: INTEROPERABILITY AND FAIRNESS THROUGH A NOVEL COMBINATION OF WEB TECHNOLOGIES
Review round: 2
Reviewer: 3

Output:
Basic reporting: No additional comments.
Experimental design: No additional comments.
Validity of the findings: No additional comments.
Additional comments: The authors have addressed all the comments and suggestions. From my point of view, the paper can be accepted for publication in its current state. To summarise the observed changes:

Introduction: the section has been improved according to my comments. It now provides sufficient information about the context and the contribution.

Terminology and related concepts: the text is now better organised and new references have been included, so every technical term is briefly introduced before its use and/or properly cited. I understand (and agree) that adding long definitions would hamper readability. After the minor changes, the manuscript does a proper balance between the biological and the computing perspectives.

URLs: Thanks for the explanation about the current state of the infrastructure. Probably it was a problem from my side, because I can access now to the UniProtAccessor. However, I cannot reach all the mentioned resources, e.g. https://zenodo.org/deposit/47641/, http://linkeddata.systems:3001/fragments
 and linkeddata.systems in Figure 8. I suggest checking all the (new) URLs before submitting the final manuscript.

A minor comment: HTML and RDF are shown in lowercase in line 845.


Instruction: You are one of the reviewers, your task is to write a review for the article. You will be given the title of the article, the number of the round in which the article is located, and your order among the reviewers.

Input:
Title: ISOLATED GUITAR TRANSCRIPTION USING A DEEP BELIEF NETWORK
Review round: 1
Reviewer: 1

Output:
Basic reporting: By and large, this article checks all the main boxes of a well-prepared submission: the writing is thoughtful and clear; the literature review is reasonably thorough (a couple suggestions below); the work presented asks the primary scientific questions, and makes a solid effort to address them comprehensively; experimental data is made freely available online for subsequent inquiry; and, overall, I quite enjoyed the read.

I've also two pieces of relevant work the authors may consider including (I'd leave it to their discretion though):

A. Barbancho, A. Klapuri, L. J. Tardon, and I. Barbancho, “Automatic transcription of guitar chords and fingering from audio,” IEEE Transactions on Audio, Speech & Language Processing, vol. 20, no. 3, pp. 915–921, 2012

E. J. Humphrey and J. P. Bello. "From music audio to chord tablature: Teaching deep convolutional networks to play guitar." IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), 2014.
Experimental design: The experimental design is generally sound. I've migrated my primary concern (re: data) to the following experimental validity bucket, though it arguably fits into either. All other facets are well motivated and / or standard in this course of work.
Validity of the findings: There are a handful of tiny nits I've noted in the annotated manuscript (attached), but a major concern I'd like to express is with regard to the limited data variance used for experimentation. Specifically, the authors generate a collection of audio files for training and evaluation, but use one guitar sound font for all data. As a result, the train / test distinction of the different folds is rather undermined, and the reader has minimal real insight into how this method may or may not generalize in realistic scenarios, or how much data is required to achieve these results. Realistically, train / test splits should at least occur at the level of the sound font.

This issue is further exacerbated as the algorithm with which the authors compare performance leverages no machine learning (as far as I can tell by reviewing the previous article). Whereas the baseline algorithm is rather generic, the deep learning model has been exposed and optimized to this one sound font. The only fair comparison might be with respect to run-time; however, processing time is only loosely correlated with complexity, e.g. big O, and perhaps the previous algorithm could be optimized, also via GPUs, such that it exhibits comparable computational performance.

Realistically, the choice of remedies I see consist of the following:
- One or more additional guitar sound fonts are obtained, the same GuitarPro files are re-rendered for the test set, and used to recompute Tables 1 & 2. Though probably time consuming, this should (hopefully?) not be especially difficult, and would greatly improve experimental validity.
- Alternatively, the authors should thoroughly acknowledge this limitation in the article as a means of better contextualizing results, and identify more extensive evaluation as a key future work item.

As a final comment, I both understand and empathize with the authors' need to leverage synthesized data, having worked with GuitarPro data a great deal in the past, but it is my feeling that, while correctable, this is the only substantive deficiency of this submission.
Additional comments: Complementing the notes I've scribbled in the annotated version, there are two points I wanted to revisit for emphasis:
- I really like the idea of optimizing the cross entropy loss jointly over the polyphony and pitch estimators. I find it a bit odd though that each logistic is given equal weight, regardless of what it represents, and wonder if this might be biasing the estimators toward better pitch estimates than polyphony (higher contribution).
- There are a variety of modern deep learning "tricks" you may find useful, especially rectified linear units (for hidden layers), dropout, and data augmentation.


Instruction: You are one of the reviewers, your task is to write a review for the article. You will be given the title of the article, the number of the round in which the article is located, and your order among the reviewers.

Input:
Title: ISOLATED GUITAR TRANSCRIPTION USING A DEEP BELIEF NETWORK
Review round: 1
Reviewer: 2

Output:
Basic reporting: The paper is well written and structured. It was a pleasure to read. The authors seem to be on top of the relevant literature and review it adequately, and Introduction is well presented.
Experimental design: The problem is well defined and technical descriptions are clear and unambiguous almost all the time. See my "General comments for the author" for a few places where I think clarification is needed.

What I was missing in this paper was a comparison of different DNN configurations. They authors present an entire system composed of many parts, but the main original contribution regards the DNN (deep neural network) used. The decisions made regarding the DNN structure are just given, without telling how they ended up using those. More specifically, I would have been interested to see a comparison between a) sigmoid and rectified linear unit non-linearity, b) deep NN versus a shallow one, c) using unsupervised pre-training versus simply applying backpropagation on from scratch on the entire network (with sufficient amount of training data). Also a so-called dropout regularization technique is not used here although it has become very popular in DNNs. Is there a reason why it was not used?

Comparing at least some of the above-mentioned configurations would greatly increase the value of the paper.
Validity of the findings: The main shortcoming of the paper regards the evaluation, more specifically:
1) The developed methods are evaluated using only synthesized recordings. I cannot accept the given reasons for that. It is totally reasonable to hire a session musician to perform at least a few songs (from a given musical notation). Perfect temporal alignment with the reference is not necessary in order to use that for evaluation (recall/precision etc.) Optionally, one can also use a six-channel microphone that records each string individually, which makes temporal alignment with reference possible too.
2) Only one guitar mark and model was used in the synthesis, causing the neural network to learn the sound of that particular guitar. Even worse, the same guitar model was used to generate both train and test data. The results do not generalize.
3) Because of the point 2 above, the main conclusion and the claim made in the paper is not valid ("When applied to the problem of polyphonic guitar transcription, deep belief networks outperform stat-of-the-art"). The reference method was generic, not even intended for guitar specifically. That makes comparison with it unfair. The method presented here is overfitted to the particular guitar used.

When comparing different configurations of their DNN, even synthetic recordings could be used, provided several different guitar models would be used for the synthesis (different for training and testing). However there should be at least a small set of actual recordings from a human performer to validate the final performance of the proposed methods.
Additional comments: Detailed minor comments:
* smallers/highest diameter string -> thinnest / thickest string
* at the bottom of page 6 (In paragraph starting "For training and testing the algorithm...": How did you define the offset of a note? Guitar sounds are exponentially-decaying, therefore they become inaudible even without explicit offset. Also there should be some non-pitched transition between notes/chords in continuous playing. How were those dealt with?
* page 10: please define "one error", "hamming loss" and "polyphony recall" exactly.


Instruction: You are one of the reviewers, your task is to write a review for the article. You will be given the title of the article, the number of the round in which the article is located, and your order among the reviewers.

Input:
Title: ISOLATED GUITAR TRANSCRIPTION USING A DEEP BELIEF NETWORK
Review round: 2
Reviewer: 1

Output:
Basic reporting: Basic reporting looks great. I'd suggest the authors consider renaming the section "Transcription Evaluation" to "Experimental Method", "Experimentation", or something to that effect. Parameter sweeps and tuning are still part of finding the solution that's being evaluated, which roll up to experimentation.
Experimental design: Overall, the experimental design is reasonably solid. I do, however, have two smaller comments.

First, regarding L396 on page 12: The text states that "recordings" are partitioned into two sets; is this split conditional on the source songs? For example, imagine two GuitarPro files A and B. A is passed through three guitar models producing A1, A2, and A3, and B is rendered to B1, B2, and B3. Can A and B both occur in the test set? In the spirit of true scientific rigor, audio rendered from the same symbolic source really shouldn't fall across partitions, and I'm curious whether or not this is considered here.

Second, regarding L527-528 on page 16: The authors intend to measure the effect of model depth, i.e. the number of layers, independently, but keep the width, i.e. number of nodes, of each layer constant. This isn't truly an independent assessment, because the number of parameters (and thus model complexity) is certainly increasing. To truly measure the effect of layers, the number of *parameters* should be held constant, as this would offer insight into what may be gained / lost with depth. Otherwise, one would expect that over-fitting will almost certainly happen as the number of parameters increase, especially here given the small size of the dataset and the minimal variance of the sound fonts considered.

That said, I don't think this is an irrelevant experiment, but it's not testing the hypothesis set out by the authors. Rather, I'd be willing to wager that performance on the training and test sets are going in opposite directions here. Reporting performance on the training set (not given) would provide some insight into what's happening here.
Validity of the findings: The majority of the findings reported by the authors are substantiated and insightful. There are a few, though, that I would suggest the authors revisit.

L502-503: The authors offer that "steel samples are generally louder than the electric or nylon acoustic samples", and perhaps that's to account for a difference in performance between the reference system (Zhou) and the one proposed here. I'm curious what the rationale would be in that one system would be more or less affected by the gain of a signal? Or how could this hypothesis be tested? After all, symbolic MIDI / GuitarPro files still encode a range of velocity values, no? I'd suspect it has more to do with the timbre of the sounds than the loudness, in that the nylon and electric guitar are "closer" than the steel sound fount (damped overtone series?).

L522-525: In motivating the exploration of multiple layers in the network (2-4), a parallel is drawn between deep networks and neurobiology. I would strongly advise against trying to make this link. Not only is it debatable, it's an unnecessary distraction that undermines the good work around it. It is sufficient to say "deeper models afford greater representational power and can better model complex acoustic signals" without bringing brains into the mix, and no one can take issue with the claim. Similar comments hold for lines L536-538

L548-553: As a conclusion to the same section named above, the authors offer three explanations for an increase in depth leading to decreases in performance: "First, increasing the complexity of the model could have resulted in overfitting the network to the training data. Second, the issue of “vanishing gradients” Bengio et al. (1994) could be occurring in the network fine-tuning training procedure, whereby the training signal passed to lower layers gets lost in the depth of the network. Yet another potential cause of this result is that the pretraining procedure may have found insufficient initial edge weights for networks with increasing numbers of hidden layers."

Based on past experience, I'd bet it is exclusively due to the first reason named. All speculation would be easily resolved by including performance numbers over the training set, as well as the test set. If training accuracy increases with model complexity, then we have our answer. In a similar manner, I am suspicious that the vanishing gradient problem is to blame, or that pre-training yielded poor parameters. Again, performance on the training set would shed some light on this. Also, I'd offer that some of the narrative be adjusted to reflect that model complexity, not just depth, is being varied here.
Additional comments: Overall the article is in good shape, and I commend the authors for their diligence in continuing to improve the work. My most important feedback (regarding science and whatnot) is named above, but I've a number of much smaller notes to share. Do with them as you will.

L34: It would be more slightly more convincing to find a more modern reference than (Klapuri, 2004) when referring to the state of the art being so far behind human experts, since it's almost 10 years older than the (Benetos, 2012) citation, which is used as evidence that a monophonic transcription is solved.

L81: It's a minor misrepresentation that Humphrey et al, 2012 & 2013 advocate the use of "deep belief networks", which are a specific kind of neural network (RBM pre-training followed by supervised fine tuning). Rather, the articles argue for feature learning and deep architectures generically, e.g. CNNs, DBNs, LSTMs, autoencoders, etc.

L119-127: It's somewhat unclear from this passage that the two systems presented are doing different things on different data, i.e. extensive guitar fingerings from solo guitar recordings versus guitar chord shapes over polyphonic pop/rock music.

L380-381: Why would MFCCs be a good feature for polyphonic pitch tracking?

L485-489: Both seem like reasonable kinds of errors. Do you have any insight to the frequency or prevalence of one over the other? Personally I'd expect the duration merging kind to be more common than the thresholding issue, but that's just a hunch.

L473: Perhaps consider using the Constant-Q transform in future work, which provides a more reasonable trade-off between frequency resolution and time resolution than the DFT.

L560: It's more accurate to say "faster than real-time", right? It's my understanding that the HMM decoding is non-causal, which means the full signal must be processed before an output can be given at t=0. This would be different from an "on-line" system, as in "as one plays music."

L565: Similar to the previous comment, it's a bit of a stretch to claim that the algorithm could be achieved with a microcontroller. It's doubtful that the processing speed seen on a personal computer (with an Intel or AMD processor in the GHz) will translate well to smaller processors with less / slower RAM.

L568: Also in the realm of tempering optimistic claims, the sentence "All that is required is a set of audio files" is a tad ironic, given how difficult it sounds to obtain data in L355-363 and L590-609. Perhaps something along the lines of "When it's possible to find, curate, or synthesize data, this approach is great."

L587: The approach used in this paper is not early stopping, but a fixed number of iterations. Early stopping requires that some measure (typically over a validation set) is computed as a function of iteration / parameters [https://en.wikipedia.org/wiki/Early_stopping].

L590: What about sample-based synthesizers? These are at least real sound recordings, rather than algorithmically defined synthesis equations.


Instruction: You are one of the reviewers, your task is to write a review for the article. You will be given the title of the article, the number of the round in which the article is located, and your order among the reviewers.

Input:
Title: COMPUTATIONAL TESTING FOR AUTOMATED PREPROCESSING: A MATLAB TOOLBOX TO ENABLE LARGE SCALE ELECTROENCEPHALOGRAPHY DATA PROCESSING
Review round: 1
Reviewer: 1

Output:
Basic reporting: The submission must adhere to all PeerJ Computer Science policies.
The submission meet to all PeerJ Computer Science policies.

The article must be written in English using clear and unambiguous text and must conform to professional standards of courtesy and expression.
The article should be revised for style, grammar and typos. The authors should avoid the use of technical jargons

The article should include sufficient introduction and background to demonstrate how the work fits into the broader field of knowledge. Relevant prior literature should be appropriately referenced.
The authors use abundant literature in the manuscript. However, they should provide more relevant references and insert a section for already existing automated MATLAB-Octave pipelines (See. http://psom.simexp-lab.org/).

The structure of the submitted article should conform to one of the templates. Significant departures in structure should be made only if they significantly improve clarity or conform to a discipline-specific custom.
No comments

Figures should be relevant to the content of the article, of sufficient resolution, and appropriately described and labeled.
Labeling of figures and explanations needs important improvements. The authors should provide labels and descriptions that make the figure a self-contained unit.
 e.g.
Figure 2: Definitions and abbreviations are not explained
Figure 3: The authors does not describe or provide relevant information on the figure
Figure 4: Size of figure should be reduced and typo on word ‘horizontal’ must be fixed. X label is missing in the figure.
Figure 6:  The authors should include a legend in the figure and define or remove abbreviations as ‘pdf’ or ‘sd’

The submission should be ‘self-contained,’ should represent an appropriate ‘unit of publication’, and should include all results relevant to the hypothesis. Coherent bodies of work should not be inappropriately subdivided merely to increase publication count.
In the manuscript the authors reference the wiki page developed for the toolbox to avoid deeper explanations on the matter of discussion. This should be avoided as this affect the unity and ‘self-containing’ aspect of the manuscript.

All appropriate raw data has been made available in accordance with our Data Sharing policy.
No comments

Formal results should include clear definitions of all terms and theorems, and detailed proofs.
No comments
Experimental design: The manuscript under revision propose an automated pipeline for preprocessing and compare different streamings of analysis for electroencephalographic (EEG) data. The development of the toolbox , Computational Testing for Automated Preprocessing (CTAP), is proposed as an extension of the already existing and popular EEGLAB Toolbox (Delorme and Makeig, 2004) developed  at the Swartz Center for Computational Neurosciences at UCSD. While it is nice to see a new contribution to the EEGLAB community, it is necessary to make some important points on the proposed manuscript/development.
In the manuscript, when reviewing the previous work on the field, the authors suggest that EEGLAB is a GUI based tool with restricted ability to be used for batch or custom data analysis scripts. Later the authors use this to justify the development of the CTAP.  However, the reality is that EEGLAB provide both GUI and scripting based capabilities. The authors are aware of this, since these scripting capabilities made possible their own development using EEGLAB as a base, as well as the many contribution received as EEGLAB's extension or plugins. This should be mentioned in the manuscript.

Regarding the relevance of the contribution. At the beginning of the manuscript,  the authors claim to propose a pipeline  to compare different streamings of EEG data analysis. This is a very good idea and and also a need of the EEG community. However, what is presented instead is a framework that comes on top of EEGLAB scripting capabilities (core and plug-in), leaving the main and most exciting part of the contribution for a future work.

I understand that the development of this framework is necessary for the later development of  a functionality to compare different streaming of preprocessing, however, without this ‘comparing’ functionality, the contribution of the current work is reduced to a new wrap on an already existing functionality in EEGLAB.
Validity of the findings: No Comments
Additional comments: As mentioned before, the idea of providing and environment for comparison of different processing streams of EEG data is bold and would be an important contribution to the EEG community. However, this contribution is not included in this manuscript. Apparently, a few scripts could do the trick. Given the ability to do exactly the same thing as the CTAP does with just a bit of scripting using EEGLAB, the authors should consider to hold on the submission and make it more valuable by implementing the ideas enunciated in the section ’Future work’.

The authors should also consider an expanded view on artifact rejection and asses not only for eye blinks but for all the many artifacts the EEG data is sensitive to.

I sincerely encourage the authors to continue working on this development and make a new submission of the manuscript attending to the formulated criticisms.


Instruction: You are one of the reviewers, your task is to write a review for the article. You will be given the title of the article, the number of the round in which the article is located, and your order among the reviewers.

Input:
Title: COMPUTATIONAL TESTING FOR AUTOMATED PREPROCESSING: A MATLAB TOOLBOX TO ENABLE LARGE SCALE ELECTROENCEPHALOGRAPHY DATA PROCESSING
Review round: 1
Reviewer: 2

Output:
Basic reporting: The manuscript „Computational Testing for Automated Preprocessing: a Matlab toolbox for better electroencephalography data processing“ by Cowley and colleagues describes the general structure and workflow of a MATLAB toolbox for EEG data processing. The toolbox extends the widespread EEGLAB MATLAB toolbox by an additional command line oriented user interface optimized for batch pre-processing of EEG data and a generator EEG test signals.

Language and writing are accessible and appropriate. The manuscript does not present original research but the description and documentation of a software (interface) for more efficient data analysis. I’m not sure whether the manuscript is within the scope of the journal. In general, most of the content of the manuscript would be expected in a software manual rather than in a research article. The manuscript could be massively improved if the focus is moved away from the mere documentation of the software (interface and workflow) towards the genuine scientific contribution, e.g. invention and/or implementation of new algorithms of signal processing and classification. Furthermore, the manuscript could be improved by adding empirical data. E.g., trying to demonstrate empirically that the approach and the implemented algorithms (e.g. blink detection and classification etc.) indeed enhance the quality of the pre-processed data.

From the information provided in the manuscript I cannot follow the claim in the abstract that the toolbox helps avoiding manual decision making to reduce subjectivity and low replicability. I do not see how the software reduces manual decision making except providing default values for some pre-processing procedures. The general validity of default values in such a diverse field as psychophysiology is questionable. Furthermore, the abstract promises testing and comparison methods. This appears to refer mainly to synthetic test signals but not to real world data. While I see the requirement to evaluate pre-processing procedures with test signals and do see the benefits of standardized and replicable test signals-as provided by the toolbox-I do not see how the toolbox helps or enforces validation with real data.

Other:
Line 22: “by” missing?
Line 62: The chapter does not provide anything essential for the manuscript. Considerably shorten? Omit?
Line 77-78: This statement is incorrect. The GUI is part of EEGLAB but the software can be used completely without from CLI or script (as the authors actually do and thus, should know).
Line 89: BrainVISION Analyzer
Figures 4 and 6: Units missing or incorrect?!
Experimental design: No experimental work is described in the manuscript.
Validity of the findings: No new findings are described in the manuscript.

An in-depth review of the toolbox code is not possible within the scope of a manuscript review. For curiosity I had a look at minor parts of the code, more or less accidentally starting with data filtering. The default IIR elliptic filter is not suitable as a general purpose filter for EEG data. The IIR filter is not part of the default EEGLAB toolbox but a plugin which appears to be no longer maintained since quite some time and known to be problematic in several aspects. The alternative EEGLAB FIR filter routines used by the toolbox are known to be broken at least since 2012 (Widmann et al., 2012, DOI: 10.3389/fpsyg.2012.00233), should no longer be used and were replaced. The documented equation for default filter order is incorrect (in particular for the IIR but also for the FIR filter). The defaults for the cutoff frequencies are inappropriate for general purpose EEG analysis (see e.g. Acunzo et al., 2012, DOI: 10.1016/j.jneumeth.2012.06.011; Tanner et al., 2015, DOI: 10.1111/psyp.12437).
Additional comments: I sympathetically see the requirement to supplement such a toolbox with a peer-review publication. However, I would suggest to replace the rather documentational aspects (much better fitting into a software manual) by (a) empirical validation of the toolbox and (b) focus on the description and evaluation of the genuine scientific contribution (e.g., blink detection, detailed description and evaluation of the validity of the synthetic test signals, description and evaluation of provided defaults).


Instruction: You are one of the reviewers, your task is to write a review for the article. You will be given the title of the article, the number of the round in which the article is located, and your order among the reviewers.

Input:
Title: COMPUTATIONAL TESTING FOR AUTOMATED PREPROCESSING: A MATLAB TOOLBOX TO ENABLE LARGE SCALE ELECTROENCEPHALOGRAPHY DATA PROCESSING
Review round: 1
Reviewer: 3

Output:
Basic reporting: Overall, the paper would be clearer if edited in plain English as much as possible, and using examples to clarify many unclear and often unnecessary technical terms. I get what you’re trying to do, and this is important work, but the paper reads as a rough draft at the moment: it really needs to be restructured and edited.

There are typos throughout - please edit carefully.

Avoid the quotation marks of mystery and be specific instead, for instance in:
complex ’standard’ EEG processing
’quality control’
of ’bad’ data
’mop up’
’tricky’
’gold standard’

Avoid monstrosities such as “learning curve”.

## abstract
“the pre-processing of EEG data is quite complicated”. Unless you provide an example, it remains unclear why this is the case. It would be fair to say that there are many options, but you can see certain large effects without pre-processing at all - which is certainly not the case for other brain imaging techniques.

## Introduction
Overall, the introduction would be much better restructured by pointing out the main steps involved in EEG preprocessing,  instead of hinting at its complexity, as well as existing tools in Matlab & Python. Then describe what’s lacking and your contribution. At present, you ignore all the pipeline toolboxes, and you fail to precisely describe what is lacking.

This sentence is difficult to parse and refers to undefined “electrophysiology data”: “However, among those types of human electrophysiology data recorded from surface electrodes (to which EEG is most similar in terms of recording methods, see e.g. Cowley et al. (2016) for a review), EEG data is comparatively difficult to pre-process.”
EEG is difficult to pre-process compare to what?

The class A and class B terminology does not bring anything: use clear sub-headings instead, and focus on examples, to avoid technical sounding but vague terms.

“EEG data is high-bandwidth” -> EEG data can be very large.

“Due to the inverse problem it is not possible to
precisely determine a ’ground truth’ for the signal”
It’s unclear how this relates to problems with preprocessing.

## Related work
This section comes far too late and should be integrated in a more comprehensive introduction.

“EEGLAB is a graphical user interface (GUI)-based tool” - this is inaccurate, because EEGLAB has been developed to be used at either GUI or script level.

I don’t see the point of Figure 1: it is rather confusing. The terms would be better defined in the text.
Experimental design: “CTAP is built on Matlab (r2015a and higher)” - does it mean it was tested with 2015a and higher, or absolutely not usable with 2014b and under?
Same question with EEGLAB.

## Approach
“Although CTAP works as a batch processing pipeline, it supports seamless integration of manual operations. This works such that the user can define a pipeline of operations, insert save points at appropriate steps, and work manually on that data before passing it back to the pipe.”
This can be done easily using EEGLAB + scripts, and is covered in the EEGLAB tutorial. How does your approach differ? Also, how does it differ from PREP and other tools?

## Methods
After explaining that CTAP differs from existing toolboxes, the methods section starts with:
“The core activity of CTAP is preprocessing EEG data by cleaning artefacts, i.e. detection and either correction or removal of ’bad’ data, that is not likely to be attributable to neural sources.”
which is what existing toolboxes do. So again, the introduction must do a much better job at clarifying what is new here.

## Configuration
In the 2 step example, please do not use *i* as an index, as in:
`i = 1; %stepSet 1`
`s` or `step` would be better coding.

## Pipe execution
Can you provide an estimate of the space needed to store the typical directory tree, given that all intermediate stages are saved? I realise this will depend on epoch lengths and if time-frequency decomposition is performed.

## CTAP outcomes
For some of the main outcomes, such as blinks, bad segments, ICA, please clarify which tools are used, for instance from EEGLAB exclusively? I see that the functions and algorithms are described later, in results. These explanations should be part of methods.
Validity of the findings: ## Results
Overall, the result section would be more convincing if it provided a more detailed walkthrough of the preprocessing of a dataset, focusing on the figure outputs.

One important aspect is missing, which is promised in the abstract: “testing and comparison of automated methods”. Can you explicitly describe how this can be done using CTAP? Do you have special functions for instance to generate figures comparing blink correction techniques? The discussion mentions that such features are not yet available, so part of the software description, including in the abstract, is misleading.

Please define “trimmed sd”. Do you mean the SD of the trimmed mean. If so, what amount of trimming is applied?

## Discussion
I agree with your points about GUIs and software development.
Additional comments: No additional comments


Instruction: You are one of the reviewers, your task is to write a review for the article. You will be given the title of the article, the number of the round in which the article is located, and your order among the reviewers.

Input:
Title: COMPUTATIONAL TESTING FOR AUTOMATED PREPROCESSING: A MATLAB TOOLBOX TO ENABLE LARGE SCALE ELECTROENCEPHALOGRAPHY DATA PROCESSING
Review round: 2
Reviewer: 1

Output:
Basic reporting: I'm satisfied with the changes.
Experimental design: I'm satisfied with the changes.
Validity of the findings: I'm satisfied with the changes.
Additional comments: Your new version is much better - good job integrating all the comments.
I hope your toolbox will contribute to more transparent and reproducible EEG analyses.
Happy software development in 2017!


Instruction: You are one of the reviewers, your task is to write a review for the article. You will be given the title of the article, the number of the round in which the article is located, and your order among the reviewers.

Input:
Title: EXPLORING TWITTER COMMUNICATION DYNAMICS WITH EVOLVING COMMUNITY ANALYSIS
Review round: 1
Reviewer: 1

Output:
Basic reporting: The paper is clearly and unambiguously written, with exception of lines 71-73 and 314-318. The description of the TISCI measures in a single sentence might confuse the reader and require multiple readings.

The introduction and background are sufficient to place this work into the broader field of knowledge, and the relevant prior literature is referenced.

The structure of the paper conforms to the PeerJ standard.

There are multiple issues with regard to the figures:
Fig 2 (d) has incorrect timestamps.
Fig 6 (d) has the circle purple dynamic community taking two different values for the population of recurring users for 03/Jan
Fig 7 (b,c) are illegible (low quality).
The combination of the heatmap (dark blue for higher values) and the characterization of the community (in black) used in Fig 7 (a), 8, 9, and 10 makes the characterization of the communities with higher population.

The raw data is supplied through a listing of the unique ids of the tweets used in for the 4 twitter datasets.
Experimental design: The approach is technically sound and thoroughly justified.
However, I have minor ethical concerts wrt. disclosing the twitter identities of the most influential users.
Validity of the findings: No Comments
Additional comments: The authors present a framework for automated information extraction in online social network (twitter) through dynamic community detection and their ranking based on varied set of measures (TISCI).
The framework is thoroughly described, and applied to four twitter datasets acquired by the authors.

In addition to the lack of clarity in the two sentences (71-73, 314-318), there is a typo in line 252: D (x \leq 1 \leq D \leq N), should be D (1 \leq x \leq D \leq N).
Table 4 has inconsistencies wrt. the formating of the numbers for # of tweets Sherlock and # unique users US Election.

I would suggest the introduction of a small table in description of each of the datasets such that the key findings of this framework could be easily summarized.
This would help to illustrate in what way the proposed framework could "assist the work of journalist in their own story telling, social and political analyst scientists[...]".


Instruction: You are one of the reviewers, your task is to write a review for the article. You will be given the title of the article, the number of the round in which the article is located, and your order among the reviewers.

Input:
Title: EXPLORING TWITTER COMMUNICATION DYNAMICS WITH EVOLVING COMMUNITY ANALYSIS
Review round: 1
Reviewer: 2

Output:
Basic reporting: See below.
Experimental design: See below.
Validity of the findings: See below.
Additional comments: In this paper, the authors propose a dynamic/time-evolving community detection framework and apply it to twitter networks.
Under this framework, two sub-problems are addressed:
i) how to identify dynamic communities from temporal graphs?
ii) how to identify rank resulting communities on the basis of structural and temporal properties?

For the first sub-problem, the paper proposes the use of existing community detection tools first and then a Jaccard index way of associating communities from different timesteps to one another. In this case, it justifies the use of InfoMap which is a widely used community detection tool for standard graphs.
Subsequently, every pair of communities from different timesteps are compared using a Jaccard similarity to match communities.
This provides a way to associate community merges/splits/death/birth events from timestep to timestep.

For the second sub-problem of ranking communities, a normalized page rank approach is used that combines persistence, stability, popularity, integrity, etc. of communities.

The approach is then evaluated on multiple twitter datasets - US/Greece elections, Sherlock, etc.
Results are presented by and large as findings of the detection framework. Top 10-20 communities are presented from each dataset and there is a discussion on what those communities represent.

Major comments:

The problem of dynamic community detection is an important problem with a wide range of applications.

Given that there are already several methods in literature (many of them cited in this paper, some missing), it becomes important for any paper on this topic to clearly state what is the achieved improvement, both in concept/algorithm and in results. This paper, by and large, uses previously devised methods and known techniques and therefore it is difficult to see what is new about the contributions made in this paper.
The use of a standard community detection tool such as InfoMap first to detect communities from every step and then match them across is the classical incremental approach that was also used in prior works such as Greene et al. 2010. In fact the approach of matching communities from different timesteps using Jaccard similarity was also proposed in Greene et al. 2010 paper. A similar approach was also presented (earlier) by Tantipathananandh et al. 2007 and  later this line of work was extended to include cost functions that straddle across time boundaries. The sequence of work by this group is only briefly acknowledged but never expanded. It needs to be.

There is also work in the literature that tackles a more global approach toward defining a dynamic community. For instance the following work by Mucha et al. 2010, is one such example, which is not cited in the paper:

Mucha, Peter J., Thomas Richardson, Kevin Macon, Mason A. Porter, and Jukka-Pekka Onnela. "Community structure in time-dependent, multiscale, and multiplex networks." science 328, no. 5980 (2010): 876-878.

There is also this review paper:

Cazabet, Rémy, and Frédéric Amblard. "Dynamic Community Detection." In Encyclopedia of Social Network Analysis and Mining, pp. 404-414. Springer New York, 2014.

The paper needs to state what is different about their proposed method compared to these highly releted works, why that matters, and then substantiate those claims of advantages and/or limitations using a comparative study in the experimental results section.

Paper's methodology and writing:
I found the reading of this paper very difficult. Its not just the grammar or typos and such. But it is also the verbosity and the lack of rigor in terms of formalisms. More specifically, there is a lot of sloppy notation and undefined terms that make the understanding of the proposed method impossible. Here are some key instances:
- First, no where is the notion of a "community" and how it differs from a "dynamic community" explicitly defined.
- Key notation such as C_{i,j}, T_{i,j}, etc are never really defined anywhere. This obviously comes in the way of understanding.
 For instance, if we look at Fig. 5, what does each of the T on the left hand side stand for, and what do their subscripts mean? What is the basis for edge labelling? The caption says there should be 5 dynamic communities but if I look at the figure, it shows actually C_1 ... C_6.

If you look at the Greene paper, for example, they define every term they use very clearly.
 I suggest adding a separate subsection for Definitions and notation prior to the description of the approach, where everything is defined unambiguously and explicitly.

As for verbosity, every effort should be made to make the reading easy - i.e., use short paragraphs (not long montonous strings of text).

The paper uses InfoMap and the authors try to justify its use by comparing it with other methods such as Louvain which are also widely used. However, the comparison they show (Figs. 3 & 4) do not really help in understanding the key differences. The authors should use some quantative way of describing the differences/strengths and weaknesses.

As for the Jaccard similarity test,
a) are communities from only successive timesteps compared or could communities from vastly different timesteps be also compared?
b) how does the use of a threshold impact the result?

As for the results, the authors have done a decent job of trying to explain their ranked community outputs. However, its not clear a) whether the proposed approach is missing out on some key communities; b) or whether there are no further ways to combine the output communities into larger communities and still maintain their significance.

The authors should consider generating synthetic inputs through simulations where there are known community structures and use those for validation first.

The authors should also compare their outputs with other dynamic community detection tools (see. related work comment above).

The quadratic complexity of the tool (in terms of timesteps) is also a problem and the authors should address that and performance reported (they say they can do it through LSH).

Minor comments:

- pg. 10, line 300: has an incomplete sentence.


Instruction: You are one of the reviewers, your task is to write a review for the article. You will be given the title of the article, the number of the round in which the article is located, and your order among the reviewers.

Input:
Title: FINDING MELANOMA DRUGS THROUGH A PROBABILISTIC KNOWLEDGE GRAPH
Review round: 1
Reviewer: 1

Output:
Basic reporting: This manuscript proposes an integrated network analysis system to detect novel drug-disease associations. Various drug and disease related data sources were integrated and a probabilistic measure to evaluate the reliability of medical entity links was utilized. Several high quality melanoma drugs were generated and evaluated by recent literatures.
Due to the complicated underlying network mechanisms of disease phenotypes and the perturbation of drug treatment, it is difficult to find and confirm novel drugs for disease treatment. This manuscript do provide an interesting computing approach to find novel drug candidates for complex diseases.
Experimental design: This manuscript focuses on proposing a complex network analysis system for novel drug detection for disease treatment. The authors have introduced the related data sources, data fusion and network visualization functionalities. An instance on novel melanoma drug detection is introduced and evaluated on the reliable and novelty.
Validity of the findings: The performance of the generated results from the network analysis system has been evaluated by the measures like precision, recall and F1 measure and the results showed that the system has obtained acceptable performance. Furthermore, 25 detected melanoma drugs were validated by related literature as well as a search for registered clinical trials in ClinicalTrials.gov.
Additional comments: This manuscript describes an integrated network analysis system on drug-target-disease associations to predict novel repositioning drugs. It integrated various drug and disease related data sources, such as drug-target associations, genotype-phenotype associations and protein-protein interactions and proposed a probabilistic approach to define the direct and indirect associations between these entities. The platform consists of a graphical web application, an application programming interface (API), and a knowledge base. The knowledge graph was stored in semantic web schema and it can be visualized as interactive networks using social network packages. The system has acceptable performance on predicting novel drugs for diseases and, In particular, the platform has generated 25 high quality Melanoma drugs, which have been cross evaluated by literature query and clinicaltrial.gov database. Therefore, this is a very interesting study that has delivered meaningful and biological useful results on novel drug indication prediction. Overall, an excellent work on network pharmacology.
The minor improvement would be that a clear description on the calculation of 'joint probability' should be provided possibly in Section 4.2. Currently, there is only introduction of the principle to set the weights on direct links. However, the weight(or probability) of the indirect links(i.e. drug-disease) is a key development of the proposed system and thus should be described.


Instruction: You are one of the reviewers, your task is to write a review for the article. You will be given the title of the article, the number of the round in which the article is located, and your order among the reviewers.

Input:
Title: FINDING MELANOMA DRUGS THROUGH A PROBABILISTIC KNOWLEDGE GRAPH
Review round: 1
Reviewer: 2

Output:
Basic reporting: The structure of the paper looks reasonable. The paper is easy to follow.
Experimental design: The goal of the paper is to use an evidence-based (in the form of probability) knowledge graph to help identify novel and known drugs for melanoma. The probability is computed based on so called expert-driven measures of the reliability of experimental methods that come from existing studies. In order for other to reproduce the results, it is necessary to provide the detailed formula for computing and combining the mentioned measures, in the final draft of the article. Also, little justifications were provided to the rather high probability (0.95) for the second confidence level.
Validity of the findings: The presented knowledge graph uses 5 data sources: iRefIndex, DrugBank, UniProt Gene Ontology Annotations, the Online Mendelian Inheritance in 168 Man and the COSMIC Gene Census. Associations are represented as ontology based linked data. However, it is unclear how such associations are captured. Some important justifications are still missing, including the two thresholds of interaction steps (3) and joint probability value (0.93).
Additional comments: The paper presents a seemingly very promising approach; however, some key details are not clarified in the current draft of the paper.


Instruction: You are one of the reviewers, your task is to write a review for the article. You will be given the title of the article, the number of the round in which the article is located, and your order among the reviewers.

Input:
Title: CHALLENGES AS ENABLERS FOR HIGH QUALITY LINKED DATA: INSIGHTS FROM THE SEMANTIC PUBLISHING CHALLENGE
Review round: 1
Reviewer: 1

Output:
Basic reporting: The paper is written in clear, professional English. The structure of the paper is clear and it corresponds to the presented topic although it does not fully conform to the recommended PeerJ standard. The Introduction and Background sections provide the necessary context, the referenced sources are relevant to the topic. The presented tables provide important and useful additional information. The table formatting should be improved, for example the structure of Table 2 is not very clear and it takes time to understand the meaning of the rows and overlapping columns. Raw datasets mentioned in the paper seem to be available in a public repository at GitHub.com.
Experimental design: The paper focuses on summarizing the experience gained during three years of organizing The Semantic Publishing Challenge. Thus, it is not a typical research paper presenting hard experimental results. The relevant methods of linked dataset creation are mentioned only briefly and the main focus is given to sharing the experience and "lessons learned". In this part, the summarization of the findings corresponds to the evolution of the Challenge - authors thoroughly describe the aspects of the Challenge that turned out to be beneficial or counterproductive. The description is sufficiently detailed. The only thing I am missing is a more thorough description of the process of preparing the "golden standard" data used for evaluating the submitted datasets and evaluation process in general. Many questions arise here: Which tools have been used for creating the "golden standard"? Was the result validated with the source documents (how?) or was any other data source used for its validation? Does some of the submitted tools actually provide better results in some situation that the golden standard and could such situation be detected properly?
Validity of the findings: The findings regarding the Challenge organization are compared and sufficiently validated with the results of a survey conducted among the organizers of similar challenges. Moreover, the paper provides an interesting statistical summary of the properties of the obtained data sets including the usage of the existing ontologies, their concepts, etc. which come from the available raw data.
Additional comments: Although the topic of the paper is not typical for informatics and computer science, it is definitely very interesting and the experience is worth sharing. I find the paper well structured and the findings are presented in detail. I recommend only minor revisions regarding (1) the table data presentation and (2) more detailed description of the Challenge evaluation process and the "golden standard" preparation.


Instruction: You are one of the reviewers, your task is to write a review for the article. You will be given the title of the article, the number of the round in which the article is located, and your order among the reviewers.

Input:
Title: CHALLENGES AS ENABLERS FOR HIGH QUALITY LINKED DATA: INSIGHTS FROM THE SEMANTIC PUBLISHING CHALLENGE
Review round: 1
Reviewer: 2

Output:
Basic reporting: "No Comments"
Experimental design: "No Comments"
Validity of the findings: "No Comments"
Additional comments: The paper provides a description and analysis of the Semantic Web Publishing Challenge, deriving a set of challenge organization best practices and providing a description of Linked Data produced as a result of the challenge. The proposed analysis is relevant and the focussed scope of the analysis corpus (as well as some of the author's involvement with the challenge organization) brings the opportunity for a deeper analysis of the problem space. The paper needs more effort on the analysis and on communication side. Improvement points are provided below.

- As it is the paper is rather descriptive and fails to create a process of abstraction and generalization of interesting patterns within the challenge. This is the main limitation of the paper. To address this, the paper needs to be seriously reworked. For example the tables in the paper concentrate on comparing lower-level attributes, rather then using higher level categories which can shed some light on the challenges and insights of Linked Data production and publication.

- The authors here have a great opportunity of providing the core insights, from a 'lab' perspective (the challenge here working as an experiment) on the critical aspects of representation, infrastructure and cultural that influence the practice of Linked Data publication. Can you dig into the dataset heterogeneity problems, for example? Which aspects of the representation provide more complete/high quality Linked Data?

- The paper is well-written but it lacks structure. As it is, the paper is a collection of low-level descriptions of observations and attributes of the challenges put together as a continuous text flow. The paper fails to generalize the observations into higher-level categories. Introducing more structure into the paper (Bullet points, subsections) is a very central improvement for the discourse.

- Some more 'soft' but very fundamental aspects are left out of the analysis at Section 3. In which communities the Semantic Web Publishing challenge had better traction (Semantic Web, Biomedical domain)? How the perception about the core communities which would be the main stakeholders in the challenge changed?

- Comparative analysis with related work is limited (and for a good reason as there are not many works on the evaluation of campaigns). One suggestion would be to better link Section 4 with related work on Linked Data Quality? Information retrieval has a long tradition on evaluation campaigns. Can you link to any work on this community?


Instruction: You are one of the reviewers, your task is to write a review for the article. You will be given the title of the article, the number of the round in which the article is located, and your order among the reviewers.

Input:
Title: SYMPY: SYMBOLIC COMPUTING IN PYTHON
Review round: 1
Reviewer: 1

Output:
Basic reporting: Please, see the review.
Experimental design: Please, see the review.
Validity of the findings: This category doesn't apply as this is a technical report and not a research paper. Further, it doesn't deal with any data nor it has findings of its own.
Additional comments: No additional comments


Instruction: You are one of the reviewers, your task is to write a review for the article. You will be given the title of the article, the number of the round in which the article is located, and your order among the reviewers.

Input:
Title: SYMPY: SYMBOLIC COMPUTING IN PYTHON
Review round: 1
Reviewer: 2

Output:
Basic reporting: No Comments.
Experimental design: No Comments
Validity of the findings: There is a minor problem with the supplementary notebook

The final cell is

>>> circuit_plot(fourier, nqubits=3);
plt.savefig('./images/circuitplot-qft.pdf', format='pdf')

This fails if the user does not have an images folder.

I suggest removing the >>> before each line. It looks strange in a notebook.
Additional comments: Sympy is a superb package and I am happy to see that there is now a paper that describes its current state. Here are a few comments that you may wish to consider before publication.

Line 105
It is generally frowned upon to import all symbols from a Python module in this manner.
I understand why you are doing it in this paper, it makes subsequent sympy commands less verbose.
It might, however, encourage bad practice. It may also lead to a poor user experience for newbies.

For example, say I had the following code

```
from numpy import sin,array

test = array([1,2,3])
sin(test)
```

I decide that I want to use sympy for something and, following your example, do

```
from numpy import sin,array
from sympy import *

test = array([1,2,3])
sin(test)

x, y, z = symbols('x y z')
```

The code will break because sympy has it's own sin that gets imported that doesn't work on numpy arrays. As a newbie, I might not know this.
I tested using Sympy 1.0 and Python 3.

Later in the text, a similar assertion is made by you (line 485) in reference to a different package that may break sympy.

Lines 151-153
Minor comment: srepr is a useful command. As someone who also uses Mathematica, I wonder if there is a sympy version of the TreeForm command which produces a visualisation of the expression tree, or maybe an output format that I could pass to a graph library for visualisation?

Section 2.3 - Assumptions
Is there a way of the user listing all available assumptions?

Section 2.4
Line 216 - This is due in part because the same language, Python, is used both for the internal implementation and the external usage by users.

This reads badly. Perhaps the following might be better?

This is due, in part, to the fact that the same language, Python, is used both for the internal implementation and the external usage by users.

Line 221:
the phrase 'Expression tree' is cited but this is not the first time you've used it. Perhaps cite earlier? Line 129 perhaps?

Page 8: Footnote 5
The line reads
The measure parameter of the simplify function lets specify the Python function used to determine how

should this be..

The measure parameter of the simplify function lets the user specify the Python function used to determine how

Section 4.1
Should mpmath be cited here?


Instruction: You are one of the reviewers, your task is to write a review for the article. You will be given the title of the article, the number of the round in which the article is located, and your order among the reviewers.

Input:
Title: SYMPY: SYMBOLIC COMPUTING IN PYTHON
Review round: 1
Reviewer: 3

Output:
Basic reporting: Generally good. A part that confused me is the assertion (footnote 3) that "If
A and B are Symbols created with commutative=False then SymPy will keep A·B
and B·A distinct." Does that mean that BOTH of them must be created this way,
and that A and x (if x is created normally) will commute? Is there any way to
declare commutators? How does one guarantee that other pieces of code, e.g.
Gaussian elimination, respect non-commutativity?

References reasonable, though [8], an excellent reference, makes almost
precisely the opposite point to that for which it is cited - "simplification
is not well-defined". I think the authors are trying to follow [8]'s
definition of simplification.
Experimental design: The referee is asked to comment whether the research is withinthe scope of the
journal, defined as "PeerJ is an Open Access, peer-reviewed, scholarly
journal. It considers articles in the Biological Sciences, Medical Sciences,
and Health Sciences. PeerJ does not publish in the Physical Sciences, the
Mathematical Sciences, the Social Sciences, or the Humanities (except where
articles in those areas have clear applicability to the core areas of
Biological, Medical or Health sciences)."

Hence I fear a software description on the maths/cmputing boundary with no
cited applications is out of scope.
Validity of the findings: There are no findings as such, since this is a software description, not an
experimental paper.
Additional comments: Nice paper - pity it seems totally out of scope. Why not J. Symbolic
Computation or some such?


Instruction: You are one of the reviewers, your task is to write a review for the article. You will be given the title of the article, the number of the round in which the article is located, and your order among the reviewers.

Input:
Title: SYMPY: SYMBOLIC COMPUTING IN PYTHON
Review round: 2
Reviewer: 1

Output:
Basic reporting: The structure of the paper and the supplement is now improved, compared to the original submission, and the materials are now much easier to read and comprehend.
Experimental design: No Comments
Validity of the findings: No Comments
Additional comments: The paper

The abstract and the paper itself mention the paper's supplement in several places. I believe it would be beneficial to add a short paragraph, possibly after line 90, that would list the supplement's contents in a manner similar to that of the paper's contents (lines 83-90), with the list submodules, to make it easier for an interested reader to find their way around it.

In line 323, it should say "...meaning that the entries...".

In line 673, it should say either "poorer than that of its commercial competitors" or "poorer than its commercial competitors'" (ending in apostrophe).

In line 813, one "PhD thesis" seems to be superficial.

The supplement

Figure 1 is hard to read. I suggest putting the labels on the outside of the loops.

Figure 2 has the wrong link towards SymPy Gamma's computation of `tan(x)` instead of `integrate(tan(x))`.


Instruction: You are one of the reviewers, your task is to write a review for the article. You will be given the title of the article, the number of the round in which the article is located, and your order among the reviewers.

Input:
Title: SYMPY: SYMBOLIC COMPUTING IN PYTHON
Review round: 2
Reviewer: 2

Output:
Basic reporting: No comments.
Experimental design: No comments.
Validity of the findings: No comments.
Additional comments: This is a great paper. Thank you for addressing my concerns.


Instruction: You are one of the reviewers, your task is to write a review for the article. You will be given the title of the article, the number of the round in which the article is located, and your order among the reviewers.

Input:
Title: A PCA-BASED BIO-MOTION GENERATOR TO SYNTHESIZE NEW PATTERNS OF HUMAN RUNNING
Review round: 1
Reviewer: 1

Output:
Basic reporting: The paper describes an approach to synthesize  running motions from examples by decomposing the motion into a set of variables such as gender, age,height, etc.
 In general the paper addresses an interesting problem, the experiments are well done and results are interesting. However, the technical description of the paper does not meet the necessary quality for a journal publication. The authors are encouraged to rewrite the paper in a more mathematically formal way since the work is interesting.

Below is a more detailed summary of my concerns:

 1) The dataset is fine but contains almost no variability on the data the researchers want to measure. The researchers indicate that one  of their goals is to
"determine a statistical model capable of synthesizing new
 realistic running motion from a set of desired data: age, gender, height, body mass index (BMI)
 and velocity."

 However, the limited number of people in their experiments does not allow to do that. Which are the height and BMI of the subjects included? It is not indicated.
 Also, according to the tables of age, there is little variation in that dimension to obtain reliable models.

 2) line 170. They indicate 3200 columns. However, it is not clear why. They should clarify on that point. If they are using 20 joints, it makes roughtly 60 variables for the whole model in a given time instant. So, it makes around 50 measures, which at a frame rate of 120Hz, is no more than half second. Is that right?
 In any case, the authors must indicate much more clearly the number of variables of their model, a picture of it and where the 3200 columns come from.

 3) This links to the Mathematical procedure in line 139. It would be much more technical to have steps 1-3 be somehow graphically shown with some figures.

 In general there is a lack of mathematical formalization of the procedure.

 4) The explanation of the PCA dimensionality reduction is not very technical.

 5) The term bio-motion generator is randomly introduced in the text. First in the title, but it does not appear again until line 173 (near the middle of the paper).

 6) The authors apply a PLS approach coupled with a LRM to do the inference from age,gender,etc to movement. However, the authors should provide  a more complete explanation of these techniques.

 7) line 208. The authors should explicitly indicate the domain of x (line 208), i.e., what is the size of a_i. Also, which is the domain of variable "gender". In general a more technical and formal description of all variables is required.
line 220, coefficients alpha and beta, have 12 components. Why 12?

 8) Equations 6,7 are not  way of calculating values (line 245), they are representation of the curve values.

 9) ICC should be an equation, though, instead of just text (line 249)

 10) The value Se in Eq 8, is not clearly indicated how it is computed.
Experimental design: No Comments
Validity of the findings: No Comments
Additional comments: No additional comments


Instruction: You are one of the reviewers, your task is to write a review for the article. You will be given the title of the article, the number of the round in which the article is located, and your order among the reviewers.

Input:
Title: A PCA-BASED BIO-MOTION GENERATOR TO SYNTHESIZE NEW PATTERNS OF HUMAN RUNNING
Review round: 1
Reviewer: 2

Output:
Basic reporting: The english language written in the paper is OK. The paper is easy to read and to understand

The background of the problem is well shown into introduction section and the references used are well chosen mixing old relevant and recent references on the problem.

In general, the structure of the paper conforms the recommendations of Perj Standard. The authors include a section of methods. The section of Materials is named  by the authors "instrumentation" and it include reasonings on the methodology used.
This organization can be improved including all expositions on methodology in section Methods that could  be named Materials & Methods.
Experimental design: The authors propose to use PCA to selection of main features on movement of an human to generate new movements according with other variables (age and  gender) that have been not considered for this problem. The idea is not very original because the authors propose a similar methodology similar to published by Vinué et al., 2014 (see paper lines  88 -95).
The methodology proposed is OK but, in my opinion, there are two relevant problems not considered by the authors. This problem is related with validation of the proposed methodology.
Validity of the findings: First problem: The authors use a commercial system based on 17 inertial sensors but there is not any information on the previous validation system. Without this prerequisite all data may not be valid.

Second problema: Because the main objective of work is to predict movements of humans with similar features but different age and/or gender, where is the real data sample  to confirm the results obtanided?
Additional comments: In my opinion the idea is well but to confirm the conclusions of the paper is needed a major effort to obtain a real sample of humans with similar features of body and differents age and gender, and it is not easy.

Also, the authors should have a effort to clarify the differences between the proposed methodology and the methodology proposed by Vinué et al., 2014 to judge the originality of the work


Instruction: You are one of the reviewers, your task is to write a review for the article. You will be given the title of the article, the number of the round in which the article is located, and your order among the reviewers.

Input:
Title: A PCA-BASED BIO-MOTION GENERATOR TO SYNTHESIZE NEW PATTERNS OF HUMAN RUNNING
Review round: 2
Reviewer: 1

Output:
Basic reporting: The authors have made some changes on the paper considering our review. Some aspect of the model have been clarified. Nevertheless, my main concerns are two. First, the dataset contains very few people for a model with so many parameters, and the variability is very reduced (Table 1). Second, there is no comparison with existing techniques
Experimental design: No comparison with other techniques.
Validity of the findings: Because reduced size of the dataset, the validity of the findings can not be assesed.
Additional comments: No additional comments


Instruction: You are one of the reviewers, your task is to write a review for the article. You will be given the title of the article, the number of the round in which the article is located, and your order among the reviewers.

Input:
Title: A PCA-BASED BIO-MOTION GENERATOR TO SYNTHESIZE NEW PATTERNS OF HUMAN RUNNING
Review round: 2
Reviewer: 2

Output:
Basic reporting: The authors have managed well the answers corrections according to my previous suggestions. So my opinion is favorable to accept the paper
Experimental design: With limitations but is OK
Validity of the findings: No Comments
Additional comments: No Comments


Instruction: You are one of the reviewers, your task is to write a review for the article. You will be given the title of the article, the number of the round in which the article is located, and your order among the reviewers.

Input:
Title: LEARNING STRUCTURAL BIOINFORMATICS AND EVOLUTION WITH A SNAKE PUZZLE
Review round: 1
Reviewer: 1

Output:
Basic reporting: The content of this paper is well organized and the text is clearly written. However there are some typos, which could benefit from a proofread. I have only two minor comments about the figures:

Figure1 could be dissected into 3 parts and labeled as "a","b","c". a) maximally compact conformation, b)intermediate conformation resembling a protein hinge, c) linearly arranged conformation resembling intrinsically disordered regions.

Figure 2 needs y axis label.
Experimental design: The authors are describing a methodology for teaching graduate students "protein folding and evolution" with the help of Snake Puzzle analogy, which they have been using in their Master course at Universidad Autonoma de Madrid.

They provide sufficient information to reproduce their method.
Validity of the findings: The proposed game/method to enhance the teaching practice is sound and robust. I believe that the students coming from different backgrounds will benefit from this paper.
Additional comments: No additional comments


Instruction: You are one of the reviewers, your task is to write a review for the article. You will be given the title of the article, the number of the round in which the article is located, and your order among the reviewers.

Input:
Title: LEARNING STRUCTURAL BIOINFORMATICS AND EVOLUTION WITH A SNAKE PUZZLE
Review round: 1
Reviewer: 2

Output:
Basic reporting: The paper designed a snake puzzle to teach students the  protein folding problem. The puzzle is similar to the e lattice models of protein structures. The puzzle is interesting. However, many parts of this paper are just reintroduction of the existing knowledge, such as RMSD, structure alignment, protein structure classification, etc.
Experimental design: From the abstract and introduction, the authors claims that to design teaching basic concepts of protein folding with the snake puzzle. However, the major of the paper is presenting the common concepts in the protein folding and relevant area. The experimental design is loosely related to the topic claimed.
Validity of the findings: no comments.
Additional comments: There are three problems that the authors might want to address in this paper. First, the authors may want to design the a puzzle to help teaching protein folding. Second, the authors may want to solve the puzzle. Third, the authors may want to solve the protein folding problem under lattice model. In any case, the authors should state clearly which problems to be addressed, and how the paragraphs are related to these problems. In addition, the commonly known paragraphs can be shorten, and references can be added.


Instruction: You are one of the reviewers, your task is to write a review for the article. You will be given the title of the article, the number of the round in which the article is located, and your order among the reviewers.

Input:
Title: LEARNING STRUCTURAL BIOINFORMATICS AND EVOLUTION WITH A SNAKE PUZZLE
Review round: 1
Reviewer: 3

Output:
Basic reporting: This paper proposes a working unit to teach a course about protein folding and evolution.  The authors proposes to use a wooden snake puzzle as a starter, where various concepts on combinatorics, modeling, folding, alignment are introduced, and related software in the literature can then be applied to solve a problem.  After that,
other concepts like evolution model and related statistical analysis are developed.

This working unit has been used in teaching a master course with students of inter-disciplinary background.
Experimental design: No comments.
Validity of the findings: The description of the working unit is very detailed, and interesting.
I believe this working unit is suitable for a master course as suggested.
The only thing I am concerned is how to test the relative effectiveness of using this working unit, as opposed to other ways (such as:  teaching the same topics, but working directly with real data instead of the snake puzzle).
Additional comments: The paper is generally well-written.
There are some potential very grammatical mistakes that the authors may wish to go through the whole manuscript carefully, such as:

Line 50:  "... that help bridging the gap ..."  -->  "... that help bridge the gap ..." ?


Instruction: You are one of the reviewers, your task is to write a review for the article. You will be given the title of the article, the number of the round in which the article is located, and your order among the reviewers.

Input:
Title: MULTIPLE COMPARATIVE METAGENOMICS USING MULTISET K-MER COUNTING
Review round: 1
Reviewer: 1

Output:
Basic reporting: No comments.
Experimental design: In the supplementary figure 1, why the running time becomes constant after k=17? The running time should grow exponentially with the kmer size. My interpretation is that the authors only picked 1M reads from each 20 samples. As a result the total number of kmers is about 20M which is much smaller than 4^17. I think the plot is unfair.

About a related issue in the same figure, when the data set is fixed, why the disk usage grows with the kmer size? Is it due to the difficulty of compression when there are more distinct kmers? Even so, I am surprised to see that the curve did not flat out until k>=51.
Validity of the findings: I’m wondering whether the sample size has an impact on the correlation, i.e. some samples have much more reads than other samples. Though some of the distance has normalization factor, can it normalize the factor of the data set size?
For example, in the BrayCurtis distance calculation, if one data set is just repeats of the other data set 100 times, then the distance will be close 1, where the distance should be 0.
Additional comments: In this paper, the author implemented a highly-parallel program Simka that can count the kmers from many metagenomic samples while computing the ecological distance with additive property between the samples. The authors did a thorough analysis showing that analyzing kmers gives similar results when using other more complex analyzing methods. Thus Simka can be applied to analyze large-scale metagenomic experiments.

The framework of Simka is solid. It is quite scalable with respect to time and memory footprint. However, Simka heavily uses disk and is not scalable with respect to disk usage.