Instruction: You are one of the reviewers, your task is to write a review for the article. You will be given the title of the article, the number of the round in which the article is located, and your order among the reviewers.

Input:
Title: EVALUATING NAMED ENTITY RECOGNITION TOOLS FOR EXTRACTING SOCIAL NETWORKS FROM NOVELS
Review round: 1
Reviewer: 2

Output:
Basic reporting: The manuscript entitled "Evaluating social network extraction for classic and modern
fiction literature" presents a study that provides a comparison between classic and modern literature through the use of natural language processing tools for the automatic extraction of social networks.

The manuscript is very well written and there are just a few typos and corrections to be performed in the text.
- line 45: missing space between "time" and "Biber"
- line 66: the second "analysis" should be removed
- line 98: the word "obtaining" is repeated
- line 99: the F-1 score should be 0.936?
- line 184: missing space between "removed" and "(1)"
- line 214: sentence needs correction "by the running the"

The literature references, the main background and the introduction of the theme are well presented, although the main motivation for extracting social networks from books is not well stated. This should be better explored to bring the reader's attention for what concerns the possible applications of such social networks.

The manuscript is well structured, presents a logical sequence of sections and a good summary of the available data and the obtained results. The data used in the manuscript is also available online through a link provided by the authors, except the books with copyright restrictions. However, the gold standard annotations of these books are also provided in the project's web page.
Experimental design: The purpose of the study described in the manuscript is well defined and the main topic is relevant to the scope of the journal. The methodology is also well described allowing the reproducibility of the work. As the main application of the study, the authors try quantify the differences and similarities between classic and recent fiction literature, which is a gap that has not been adressed yet by previous related works.
Validity of the findings: The authors provided an in-depth comparison of four Named Entity Recognition systems. They identified some issues that can be related to the performance of these systems in recognizing characters, such as real word names. They showed how performance can be increased by applying some transformations to the corpus which are able to handle those issues.

This study draws attention on how Networks can contribute as an analysis tool in the context of natural language processing. However, for what concerns the extraction of social networks from the novels, the quantitative analysis showed no significant differences between classic and modern literary books. I would like to highlight the following issues:

- The network measures are very different within each group. Given such differences in the number of nodes and the number of edges, it becomes difficult to compare these networks among themselves just by analysing the mean and standard deviation. Are the networks with such different degrees comparable to each other?

- Throughout the description of the results and their respective analysis, the authors raised some alternatives from the methodological point of view. For instance, the use of conversational networks instead co-occurrence networks and the use of a list of aliases. To which extent these alternatives can influence the conclusions about the differences and similarities between classic and modern books? This should be elaborated in the discussion or conclusion parts.

- No significant differences were found between the performance of classic and modern novels, and also between the two types of social networks. Would this result be beneficial from the point of view of network analysis, given for instance, the community structures and the social interactions described in the books?

- Did the authors consider the possibility of using machine learning techniques to distinguish both classes of literary books?

- In Table 10, the authors present the network measurements for only 18 books of the dataset for each class. Why two books were removed for each class in this table? In addition, the mean values presented for each measure do not correspond to the actual mean of each column. This table should be reviewed.
Additional comments: No additional comments


Instruction: You are one of the reviewers, your task is to write a review for the article. You will be given the title of the article, the number of the round in which the article is located, and your order among the reviewers.

Input:
Title: EVALUATING NAMED ENTITY RECOGNITION TOOLS FOR EXTRACTING SOCIAL NETWORKS FROM NOVELS
Review round: 1
Reviewer: 3

Output:
Basic reporting: no comment.
Experimental design: Please see the detailed comments below. There is a research question, but the gap is not clearly defined. There is clearly a very rigorous investigation. The approach is clearly described, as could be replicated.
Validity of the findings: Please see the detailed comments below. There are some questions about the two selected cases, but I trust the method is sound. The discussion is lacking in specification, but I believe this can be improved in two ways: either reordering the argumentation, or developing two separate papers. I have detailed this in the comments below.
Additional comments: This paper reviews the NLP tools used to automatically identify literary networks. It reviews four popular and thus commonly implemented NER tools. It also generates networks from the results of NER from the tool BookNLP. This paper has a lot of potential, the authors have done a valid and systematic comparison of the tools, but there is a desperate need to clarify concepts, contributions and the argument, as to enhance this contribution. In addition, I would strongly advise the authors to consider developing this into two different papers to support the two differing yet complementary arguments that they attempt to make in the paper. One comparing the NLP tools for conducting NER and another of the pipeline from NLP tools of NER to networks and thus the network analysis of the literary networks. I think this would do the two arguments and contributions much more justice. For this review, I have assumed they will keep with the dual argument and have made a set of suggestions to improve the paper, with mentions when necessary as evidence for separate papers.
As I mentioned above, there is a desperate need to clarify concepts, contributions and the argument(s), as to enhance the contribution(s). For example: Already in the introduction the argument is lacking clarity which leaves the reader confused: first the authors say NLP is important and we need compare these tools (but fail to tell the reader why), then the authors state that actually the problem is that we do not have comparisons of the 18th + 19th century texts with modern texts as the styles are different and thus we cannot compare the networks (but still the reader does not know what literary network analysis tell us and why that would matter). And then the authors say we need to understand “the potential differences or similarities in terms of (1) performance of natural language processing techniques, (2) social network structure, and (3)overall writing style”. Then in your research question you state you are looking only at fantasy novels, but with no context as to how that is an element of the overarching research question. The reader cannot grasp what the main contribution is as the authors fail to present a logical argument and detail knowledge (what is currently now presented as assumptions). Please see this as an opportunity to clarify your points.
There is a great need to clarify your contribution and build an argument so that these three above points build upon each other.  Currently they are just muddled through each other. Being much more precise about your boundaries is going to enhance your argument immensely. There are a number of arguments that need to be established, that have either been overlooked or assumed, where it would serve the authors to be much more precise:
First, in the first paragraph of the introduction the authors need to establish: what are literary networks and what can they tell us? (do not mention NLP yet). Please explain and cite what it is meant by literary networks. You seem to infer it is social networks and you reference Moretti in the first paragraph, but then in paragraph 3 it is stated that the concept and state of interest is the language and style that changes over time. Thus there is a step missing for the reader, it would serve one well to clarify your scope to identify how others have considered and identified links – are we talking about references between characters, across novels/works, etc? Also what does the mapping of literary networks serve to explain? I do know this elaborated upon in the section Related Work, but this is separate from these techniques, this has to be established as to understand why NER is being used as a technique for this approach.
In addition, as part of the second argument, the authors state later “we have not found any other studies that compared the performances of social network extraction on classic and modern novels; or compared the structures of these networks”, which gives the reader the assumption that this is important, but it has not been established in the introduction why this is important and why it is a gap that needs to be filled. Also note this gap is different than the one the authors attempt to establish later – comparing time periods. It is fine to have two contributions, but then please highlight them as two gaps and very precisely how they build upon each other. Again this question arises- could this potentially be two separate papers, which would allow you elaborate in much greater detail on both contributions.
Second, review the current tools that have been used to identify literary networks. Currently, indirectly the authors state that NLP is increasingly being used, and that there are multiple tools, but they do not directly establish why it is important to review them. You may want to state here that you are only focusing on computational tools, as to exclude a review of close reading methods. This review should summarize with your key argument (as presented now) that there is a need to compare networks over time. Please state why this is so important. I believe you attempt to do this, but not directly, you need to rehash the state of the art here.
Third mention the example, related to the periods. Do note: If you are going to use literary novels as your source and thus target literary scholars (as I am assuming you are), you need to get the terminology correct. 18th and 19th century texts are not classics. In popular understanding they may be largely referred to as classic/timeless novels that make up a canon of must read texts, but these are not classic texts. In literary studies, the classics refer to works from ancient Greece and Rome.  This period you refer to, pending how far you go into the 19th century, can be referred is often as the Enlightenment – starting around 1660, with the crowning of the exiled Charles II, until the beginning of the 19th century and the reign of Victoria. Although it may also include the period of Romanticism (early 1800s to Victorian period). If you are largely speaking of English texts then you may consider to mention it as Victorian- under the reign of Queen Victoria (1837–1901). In any case you need to pick another term as it is incorrect to call this this classics when you are referring to literary studies. The use of modern is applicable, as if I understand it is looking at the 20th century. Although, you are going to have to justify later why you looked at fantasy novels as your modern set, and not canonical works as you did for the 18th and 19th century. Please see the comments related to fantasy as a selection.
Fourth, the research questions would be fine if it did not include the fantasy mention. The problem lies in that fantasy is a literary genre, not a period (as you mentioned before as your core gap) and classics as you refer to them are not a genre. These top novels of the 18th and 19th century are a canonical distinction. And one could argue fantasy is not related to your research question, but rather a methodological choice to limit even further the assumed variation between periods. Thus the fantasy bit can be included in the write up of the approach. On the other hand in selecting these two cases the authors have actually selected two incomparable cases: an overarching representation of 18th and 19th century novels and a specific genre from the 20th century. So either select for representativeness of a period or on genre or both. This is very important given that the authors state that they seek to contribute to the understanding of how to study or compare differences between these networks, but then select comparable cases from that has similar characteristics for the two cases. I suggest/guess, there must be another Guardian list of  20th or 21st century novels that makes up a canon.
Fifth, mention here the approach, how the research questions be answered. As is mentioned “The contributions of this paper are (1) an annotated gold standard dataset with entities and coreferences of 20 classic and 20 modern novels, (2) a comparison and an analysis of four named entity recognition on 20 classic and 20 modern novels, (3) a comparison and an analysis of social network analysis measures, and (4) experiments and recommendations for boosting performance on recognising entities in novels”. Explain shortly the steps of each of these.
Then begin the related work section. I can follow the steps made and the analysis. Although I think the authors can expand greatly on the discussion. This is particularly true in the case of the network measures, the authors measure a number of measures, show lots of visualisations, but to do them any justice. The author need to elaborate on why these measures, how do they compare, how do they contribute back to this gap. It is one thing to say, look we can do this, but given your initial framing for both technical NLP and literary researchers I would challenge you do more and expand on this.
In regards to analysis of the networks, it is very superficial. Since you aim to contribute to literary network analysis, it would be good to employ a network analyst who can assist on integrating the measures, methods, etc and to link to the contributions the authors attempt to make. For example, the authors have measured so many aspects of the network, let that terminology guide you. Some examples: very full network is not a network term; if you want to describe a full network measure density and describe the networks as a continuum from sparse and dense. There are assumptions such as: “the relationship between the main characters of this novel can easily be identified” – please explain how they are identified? And note which characters these are. Also a visualisation is not an analysis, it is a visualisation, one may be able identify clusters by the eye as suggested - “The visualisation of such a network also offers a prompt manner to identify social clusters.”, but to serve as evidence one needs to use graph theory and let the mathematically evidence serve as the evidence of clusters. Thus, if you want to discuss clusters use a clustering algorithm. Again, unfortunately, the authors are just selling themselves short by not maximizing the use of the network measures and terminology. Consequently, the discussion and contributions fall short, which is a pity.
Overall, I truly think the paper is very interesting and needed in this field; but the authors are not doing their work justice, by stuffing them into one paper. I think it would serve them better to develop two papers. One that makes a case for comparing tools and thus time periods and then a second for the network structures. This comes to light a number of times, both in the current lack of specificity of the argument and that in Section 5, the authors only generate the networks from one tool – BookNLP, which is a different initial approach to comparing the tools. In addition, I would argue that these contributions, although related, actually speak to different sub-sets of the research community- 1) using NER tools, and then 2) those that use NER tools for literary networks specifically and how to understand these structures. In developing two papers the authors can expand on their discussion of the findings for both NER tools and networks.
There are number of instances where you are missing commas, which I did not note here, as I am assuming this is related to the Latex file, please check accordingly.


Instruction: You are one of the reviewers, your task is to write a review for the article. You will be given the title of the article, the number of the round in which the article is located, and your order among the reviewers.

Input:
Title: EVALUATING NAMED ENTITY RECOGNITION TOOLS FOR EXTRACTING SOCIAL NETWORKS FROM NOVELS
Review round: 2
Reviewer: 1

Output:
Basic reporting: Several aspects suggested in the first revision were corrected, however there are still some errors or unclear pieces of text regarding the English writing of the manuscript. Some sentences need to be corrected because they still were not clear. In other cases, some sentences are very long, it would be necessary to use punctuation marks to separate them to better understand such sentences.  The sentences could be found at lines:
	- 57 to 59
 	- 106 to 107
 	- 113 to 114
 	- 129 to 130
 	- 133 to 134
 	- 155 to 156
 	- 189 to 190
 	- 240
 	- 283
 	- 305 to 306
 	- 310 to 312
 	- 318 to 319
 	- 414 to 416
Experimental design: no comment
Validity of the findings: no comment
Additional comments: The research article is good, it presents different relevant results, and it makes many comparisons. And also it gives suggestions for future work. The article showed a large number of results, comparing the two types of literature proposed (classical literature vs. modern literature).

The authors addressed all the points raised by the previous revision. However, it is required to make some changes related to the English in the text in some sentences. The sentences that need to be reviewed are indicated in the basic reporting section.


Instruction: You are one of the reviewers, your task is to write a review for the article. You will be given the title of the article, the number of the round in which the article is located, and your order among the reviewers.

Input:
Title: EVALUATING NAMED ENTITY RECOGNITION TOOLS FOR EXTRACTING SOCIAL NETWORKS FROM NOVELS
Review round: 2
Reviewer: 2

Output:
Basic reporting: no comments
Experimental design: no comments
Validity of the findings: no comments
Additional comments: My main questions have been addressed by the authors. The performed changes resulted in a more clear and organized text.


Instruction: You are one of the reviewers, your task is to write a review for the article. You will be given the title of the article, the number of the round in which the article is located, and your order among the reviewers.

Input:
Title: EVALUATING NAMED ENTITY RECOGNITION TOOLS FOR EXTRACTING SOCIAL NETWORKS FROM NOVELS
Review round: 2
Reviewer: 3

Output:
Basic reporting: This revision of the text is vastly improved, from the previous version. The methods, findings and discussion sections in particular, have improved in clarity, specifically that the approach is now explained in detail, and the decisions explicated to the research question. That being said, I still think that the positioning of the paper can be even more specific.

This increased specify begins with the first paragraph to the research questions, in the introduction. The very first paragraph attempts to position the study in a larger field, but in doing so it not only incorrectly positions the contribution and scope to making contributions to literary theory (which the paper does not do), but also casts the scope so large to quantification of literary studies; when in fact your study is much more specific contributing to knowledge specifically to character network identify through NLP techniques. As the Game of Thrones example proves and which you state on page 13 line 461 “ This is a design choice that [462] will depend on the type of research question one wants to answer by analyzing the social networks”. Yes! Yes! This is your contribution! – in order to understand the validity and reliability of these character networks and thus our understanding of social structures in novels, this starts from the nature that the nodes and edges were automatically identified using the NLP/NER tools. This is so fundamentally important, and your approach gives scholars great insights into which tool is best suited for their study or research question. Thus drop the entire framing of contributions to literary theory, claim your findings and their boundaries and roll with them!

Thus, I would suggest a re-positioning. This would mean that I would highly recommend removing the first paragraph completely, and instead start out with why the understanding of social networks in novels are important in literary studies. Then narrowing your scope to how these networks can be identified, and the NLP approaches differ and how. Thus you then have one more general research question: To what extend do NLP techniques using NER differ in identifying social networks in novels? / as to how to operationalization networks (really what is a node and edge) and how that translates to our understanding of the projection of the networks and thus analysis of the networks. Where you first look at the NER process and compare it, and then the network itself and its relationship to the extraction/identification.

In addition, this suggests that your framing of the importance and selection of the two time periods instead becomes part of your case selection, not part of your aim. It is not your aim to contribute to our understanding of different periods, you even say that no differences have been found and no other research has been done this (starting at page 2). Thus I would highly recommend you delete this claim/positioning. Instead mention the time periods and genres in the methods section as a variable you consider to insure that it is applicable to “all” periods; much like you compared these networks to real world networks. This serves to further ensure the validity of the findings, but it is actually irrelevant to the approach and contribution you aim to make.
This then suggests that the title becomes more general and thus more powerful! – evaluating social network extraction tools for fictional literature.

In addition, given one of your core audiences is literary scholars, I still adamantly oppose to you using the word classics, these are canonical works as you also mention now in a footnote. But no literary scholar is going to take your work seriously if you from the outset incorrectly categorize this core term in their field.
Experimental design: See the above comments.
Validity of the findings: See the above comments.
Additional comments: I want to emphasize, the paper has improved greatly, and these suggestions suggest a further specification of the contribution and message, that are minor revisions that should even further enhance the impact of this work.


Instruction: You are one of the reviewers, your task is to write a review for the article. You will be given the title of the article, the number of the round in which the article is located, and your order among the reviewers.

Input:
Title: EVALUATING NAMED ENTITY RECOGNITION TOOLS FOR EXTRACTING SOCIAL NETWORKS FROM NOVELS
Review round: 3
Reviewer: 1

Output:
Basic reporting: Several aspects suggested in the previous revision were corrected, however, there are still some errors or unclear pieces of text regarding the English writing of the manuscript. The unclear sentences or some mistakes could be found at lines:
- 21-23: Furthermore, we identify several issues that complicate named entity recognition in our set of novels and we present methods to remedy these.
- 33-34 The most commonly used approach for extracting such networks, is to first identify characters …
- 136-137: One difficulty of character detection is the variety of aliases one character might go by, or; coreference resolution.
- 367: However, other than that, none of the other measures differ significantly …
- You must use a unique pattern to talk about F1-score measurements:
     * 121: F1-scores between: .45 and .54. Vala et al. …
     * 130: obtaining an F1 score of 0.936 for ...
     * 308: yields a F1 score of 51.91. …
     * 545-546: F1 score rose from 13.91 to 53 ...
Experimental design: no comment
Validity of the findings: no comment
Additional comments: The authors addressed most of the points raised by the previous revision. However, it is required to make some changes related to the English in the text in some sentences. The sentences that need to be reviewed are indicated in the basic reporting section.


Instruction: You are one of the reviewers, your task is to write a review for the article. You will be given the title of the article, the number of the round in which the article is located, and your order among the reviewers.

Input:
Title: EVALUATING NAMED ENTITY RECOGNITION TOOLS FOR EXTRACTING SOCIAL NETWORKS FROM NOVELS
Review round: 3
Reviewer: 2

Output:
Basic reporting: no comments
Experimental design: no comments
Validity of the findings: no comments
Additional comments: The reviewer's suggestions concerning the contributions of the paper were incorporated in this second revision by the authors. The paper was greatly improved.


Instruction: You are one of the reviewers, your task is to write a review for the article. You will be given the title of the article, the number of the round in which the article is located, and your order among the reviewers.

Input:
Title: EVALUATING NAMED ENTITY RECOGNITION TOOLS FOR EXTRACTING SOCIAL NETWORKS FROM NOVELS
Review round: 3
Reviewer: 3

Output:
Basic reporting: The paper has improved greatly in precision and clarity.
Experimental design: Sufficient.
Validity of the findings: Sufficient.
Additional comments: I recommend this paper for publication, and have no further questions or concerns.


Instruction: You are one of the reviewers, your task is to write a review for the article. You will be given the title of the article, the number of the round in which the article is located, and your order among the reviewers.

Input:
Title: ENHANCING THE PERFORMANCE OF THE AGGREGATED BIT VECTOR ALGORITHM IN NETWORK PACKET CLASSIFICATION USING GPU
Review round: 1
Reviewer: 1

Output:
Basic reporting: The paper presents an implementation of a parallelized ABV algorithm on a GPU.

Language and clarity: The paper is generally written in clear English. There are some minor language issues which should be fixed (for example in lines: 40, 54, 77, 86, 132, 260, 266, 278, 301). However, the clarity of the main ideas in the paper could be improved.
For example, some parts of the text should be revised to make the text easier to follow.
1) The contributions listed in intro are not clear. Maybe provide more detail on the first contribution.
2) The background should be more clearly written. E.g., it dives right in to how the constructs of ABV work without explaining what is the purpose of the algorithm.
3) Minor issue - In the second section, I am not familiar with the use of the term instruments in this context, why not rename the section to background?

Ethics: Figure 2 is copied directly from the paper by Baboescu and Varghese, though it is not mentioned that the figure is from that paper.

Analysis: The authors present equations that represent the  total time and memory of the execution. However, they do not explain the equations or prove their correctness. It may be that some of these results follow from the use of Ma et al. though that should be more rigorously exhibited and explained. In general the analysis section is hard to follow and the results are not sufficiently discussed.
Experimental design: The authors present a new way to improve the implementation of the aggregated bit vector packet classification technique presented by Baboescu and Varghese by parallelizing parts of the algorithm on GPUs.

Research question and comparison to other work: The authors present the need to speed up packet classification and propose an interesting technique. This solution should be better placed within the context of other existing solutions. There is some comparison though it is hard to infer an apples-to-apples comparison from the short comparison and table that is given.

Generalizing the result and evaluation: The discussion of the implementation and the evaluation discuss the case for two header fields. The authors should consider ways to generalize to k header fields.

Evaluation results: The purpose of the graph showing the analytic speedup is not clear, since it shows the same results for all number of filters. If the point is to compare with the results in figure 7 you may want to place the results on the same graph.
Minor issue: the legends in the graph should be labled
Validity of the findings: Data set: The authors make use of the Classbench simulator for generating files for evaluation. The toolset is publicly available making the simulations potentially reproducible, though it is no longer supported and I am not sure if it can still be used in this case.

Conclusions are provided as part of the evaluation though they should be better summarized.
Additional comments: The paper is generally well written and easy to follow, though it lacks detail of analysis and comparison to related work. Furthermore, it is not clear what would be the impact of generalizing to more fields. If that is somehow implied in the paper it is not clear.


Instruction: You are one of the reviewers, your task is to write a review for the article. You will be given the title of the article, the number of the round in which the article is located, and your order among the reviewers.

Input:
Title: ENHANCING THE PERFORMANCE OF THE AGGREGATED BIT VECTOR ALGORITHM IN NETWORK PACKET CLASSIFICATION USING GPU
Review round: 1
Reviewer: 2

Output:
Basic reporting: This paper presents a parallel implementation of aggregated bit vector algorithm for packet classification on GPU.

The paper is well written. Authors have provided sufficient literature references and background context. Experimental results are aligned with authors' analytical performance model. Terms and parameters are well defined.
Experimental design: The authors have used three major performance metrics for evaluation, including (1) classification time which is the time to classify all the packets, (2) throughput which is number of packet classified per second, and (3) speedup which is computed as the classification time of a serial CPU implementation over the classification time of the proposed parallel GPU implementation. Authors observe that the GPU implementation achieves 60-100 speedup over CPU implementation.

I suggest authors to add the performance of single packet classification latency. Because for latency-sensitive networks, this latency is more import than throughput.
Validity of the findings: The supplemental files contain the source code of the implementation and the packet files used for experiments. Those files match the authors' description in the paper.

Authors have provided the analysis of empirical results and their analytical performance models, indicating that their model can help predict accurate performance.
Additional comments: This paper is well written and structured. And it provides both empirical results and performance analysis.

I have the following suggestions:

1. The content between Line 278-281 is not reflected in Figure 3 (ABV pseudo-code). It makes me confused about how the other three fields are handled in your approach. I suggest to give more details here.

2. Table 5 seems to be an unfair comparison. This is because Varvello's CPU runs at 2.66 GHz (so their speedup number is relatively small due to a fast CPU), while your CPU runs at 2.22 GHz. A fair comparison should be based on the same baseline platform.

3. Line 399 states the memory complexity linearly increases. However, this is not true based on Figure 10. Please correct this statement.

4. Figures 8 and 9 should use KPPS as unit as stated in Line 364.

5. Fix these two sentences,
Line 75, "has an appropriate structure the lets it be highly"
Line 370, "it requires 9748/5"


Instruction: You are one of the reviewers, your task is to write a review for the article. You will be given the title of the article, the number of the round in which the article is located, and your order among the reviewers.

Input:
Title: ENHANCING THE PERFORMANCE OF THE AGGREGATED BIT VECTOR ALGORITHM IN NETWORK PACKET CLASSIFICATION USING GPU
Review round: 2
Reviewer: 1

Output:
Basic reporting: no comment
Experimental design: no comment
Validity of the findings: no comment
Additional comments: The paper has greatly improved and most of the previous concerns have been addressed.
Some comments:
1) Regarding the generality of the fields, the comment added that "For this purpose other fields of packet are inspected linearly" is not clear and still requires further explanation.
2) I do not see the needed citation in figure 2. I think it should be explicitly linked the figure in the caption or the related text about the figure.


Instruction: You are one of the reviewers, your task is to write a review for the article. You will be given the title of the article, the number of the round in which the article is located, and your order among the reviewers.

Input:
Title: ENHANCING THE PERFORMANCE OF THE AGGREGATED BIT VECTOR ALGORITHM IN NETWORK PACKET CLASSIFICATION USING GPU
Review round: 2
Reviewer: 2

Output:
Basic reporting: Overall, the paper is well written and easy to follow.
But there are a few minor reporting issues to be fixed.

Line 85-87: It is confusing here by saying 66.54 times throughput improvement corresponds to 100 times speedup. By reading evaluation session, it looks like 66.54 is average improvement number for throughput, while 100 is the maximum number for speedup?

Line 109: missing space, (Nakano 2013a)CUDA

Line 128, (Cheng et. al 2014) occurs twice

Line 408 (3) should be (4) because Line 329 already has an Equation (3)
Experimental design: Figures 6 and 7 show the comparison between predicted speedup and actual speedup. It would be better to discuss how close the predicted performance (throughput or latency, or total time) is to the measured performance as well.
Validity of the findings: No comment
Additional comments: In addition to the above minor issues, I also suggest authors to briefly discuss the motivation to select the ABV algorithm to accelerate on GPU. This discussion can be added in Introduction.


Instruction: You are one of the reviewers, your task is to write a review for the article. You will be given the title of the article, the number of the round in which the article is located, and your order among the reviewers.

Input:
Title: TOO TRIVIAL TO TEST? AN INVERSE VIEW ON DEFECT PREDICTION TO IDENTIFY METHODS WITH LOW FAULT RISK
Review round: 1
Reviewer: 1

Output:
Basic reporting: See my general comments
Experimental design: See my general comments
Validity of the findings: See my general comments
Additional comments: The authors propose an approach to identify methods that are less likely to be buggy. Their work has different stakeholders ranging from developers who can minimize their test efforts to researchers who can improve their prediction model, not only within a single application. But, also for cross-project prediction models.

The authors base their approach on mining association rules. It consists of mapping a set of 39 source code metrics to the dependent variable that defines if a method is faulty or not. By this approach, the authors established a set of rules, each of which has support higher than 20% and a confidence level lower than 90%.

On six projects from the Defects4J dataset, the authors validated their model using ten cross-validation technique, while they studied the performance of their models on other projects using five projects to build the model and one project as a testing data. The authors were able to achieve high performance.

The strengths of the paper:
- Very nice idea.
- The article is well written and motivated.
- A solid approach that is well enforced by existing research work.
- Interesting findings.

Weaknesses:

It is not 100% clear if a low-risk method is a method that never had a bug in the past, or if the authors consider fixed methods to be correct methods. The authors said that "First, we checked out the source code from the project repository at the original bug-fix commit and stored it as fixed version." It is not clear if they consider one method through the whole history as buggy or not, or if they can consider one fixed method as a correct method, while they consider it as a buggy method before the fix.

That opens another concern how the authors know if a fixed method doesn't contain another bug, which might be fixed in the future?

While the number of independent metrics used in the paper is significant, building rules from high-level metrics would also be interesting. Instead of the number of try blocks, throw statements, catch clauses, it would also be interesting to define a category that encapsulates all of these in one group such as exception. Then, study the impact of these groups so that readers can know the effect of a practice instead of just primary metrics.

The formulation of RQ1 is very confusing, while it is just evaluating the precision of the model. Simplifying it would be interesting.

Figure 3 shows a modification in the API of a method. So, how do you distinguish between methods? Do you consider just method names? Names and parameters? That is because one can define multiple methods with the same name and different parameters. From the paper, it is not clear if the method (as an example) in Figure 3 becomes another method or not.

Discuss how do you count the confidence and support.


Instruction: You are one of the reviewers, your task is to write a review for the article. You will be given the title of the article, the number of the round in which the article is located, and your order among the reviewers.

Input:
Title: TOO TRIVIAL TO TEST? AN INVERSE VIEW ON DEFECT PREDICTION TO IDENTIFY METHODS WITH LOW FAULT RISK
Review round: 1
Reviewer: 2

Output:
Basic reporting: Overall the paper is well written and easy to read. The case study is designed and conducted rigorously. The authors also thoroughly discussed the threats to validity from different angles.
Experimental design: The author claims repeatedly in the paper that recall do not matter but I do not find their argument convincing, In fact, the larger the number of low-risk methods retrieved, the smaller the number of methods that are left to be carefully inspected. Which would result in lower testing resources spent!
Validity of the findings: To justify the need of training classifiers specifically for the detection of trivial methods, I believe that the author should compare the obtained results with those of traditional bug prediction approaches where methods with low probabilities of containing a fault are considered “trivial to test”. In fact, many traditional bug prediction approaches are binary classification approaches, that focus both on detecting code with high and low risk of fault. Hence, they can also be used to detect methods with low risk of fault. A thorough comparison between these traditional approaches and the proposed approach is necessary.

-Also, it would have been interesting to see some assessment of the effort required from developers to examine the identified ‘trivial’ methods.  It is very possible that these trivial methods are also very easily identified by developers as low risk (e.g., empty methods, getters, and setters).
Additional comments: This paper proposes an inverse view on software defect prediction, where the idea is to identify methods unlikely to contain faults, rather than those likely to contain faults. The authors extracted metrics from the source code of the projects in Defects4J database and used the Apriori algorithm to create rules to identify non-faulty methods. Through a case study with six Java open-source projects containing precise fault data at the method level. The authors show that their proposed approach can identify between 32% and 44% of the methods of a project to have a low fault risk. These methods are about six times less likely to contain a fault compared to other methods in the studied database. In cross-project predictions, the identified methods are eleven times less likely to contain a fault.

+ Overall the paper is well written and easy to read. The case study is designed and conducted rigorously. The authors also thoroughly discussed the threats to validity from different angles.

Although I like the idea of the paper, I have some doubts about its usefulness for the following reasons:

-The reported recall values are quite low in general (e.g., 18.4% for Math), which mean that many of the “trivial” methods are missed, which is detrimental to the objective of reduction of the testing effort. The author claims repeatedly in the paper that recall do not matter but I do not find their argument convincing, In fact, the larger the number of low-risk methods retrieved, the smaller the number of methods that are left to be carefully inspected. Which would result in lower testing resources spent!

- The paper reports that “28.6% of all methods with 13.8% of the SLOC are classified as LFR and contain 4.1% of all faults”. I wonder what is the severity of these faults that are contained in the false positive? The authors briefly discussed them in Section 6.1, but I believe that a more detailed investigation is needed.

-Also, to justify the need of training classifiers specifically for the detection of trivial methods, I believe that the author should compare the obtained results with those of traditional bug prediction approaches where methods with low probabilities of containing a fault are considered “trivial to test”. In fact, many traditional bug prediction approaches are binary classification approaches, that focus both on detecting code with high and low risk of fault. Hence, they can also be used to detect methods with low risk of fault. A thorough comparison between these traditional approaches and the proposed approach is necessary.

-Also, it would have been interesting to see some assessment of the effort required from developers to examine the identified ‘trivial’ methods.  It is very possible that these trivial methods are also very easily identified by developers as low risk (e.g., empty methods, getters, and setters).


Instruction: You are one of the reviewers, your task is to write a review for the article. You will be given the title of the article, the number of the round in which the article is located, and your order among the reviewers.

Input:
Title: TOO TRIVIAL TO TEST? AN INVERSE VIEW ON DEFECT PREDICTION TO IDENTIFY METHODS WITH LOW FAULT RISK
Review round: 2
Reviewer: 1

Output:
Basic reporting: The paper is well written in general.
Experimental design: Experiments are rigorous in general.
Validity of the findings: The reported results are valid. Thank you to the authors for clarifying the context in which they are expected to be used.
Additional comments: I would like to thank the authors for the revisions and the detailed response. I am happy to recommend the acceptance of the paper.


Instruction: You are one of the reviewers, your task is to write a review for the article. You will be given the title of the article, the number of the round in which the article is located, and your order among the reviewers.

Input:
Title: COMICS: A COMMUNITY PROPERTY-BASED TRIANGLE MOTIF CLUSTERING SCHEME
Review round: 1
Reviewer: 1

Output:
Basic reporting: --The outline of the article is clear in general. However, there are quite a few issues in the presentation that make careful readers fail to understand the content thoroughly. I list some while there are a lot left:
*In the intro of the roadmap of this article, all the reference for section numbers are missing. It is the same in other places all throughout the article.
*In Figure 2, the subfigures in the second line are exactly the same except for some literal explanations. The subfigures themselves have no usage.
*In Algorithm 1, what is the definition of G'? Is this G' same as that the problem statement? It is also hard to understand the inner loop of Algorithm 1 and its corresponding explanation in lines from 197-200.
* What is the definition of R0 in algorithm 2? R is changing due to the steps 4, 5. Is R0 changing as well? If R0 does not change, does it mean that it is at most two subgraphs that may merge? If so, it does not seem to be rational?
* The complexity analysis is kind of a mess. For Alg1, is searching for the smallest degree O(1)? For Alg2, again, the authors seem to assume that at most two subgraphs may merge. For Alg 3, if only the second smallest eigenvalue is computed, the complexity is just O(n+m), and Benson's Science paper shows that there will be more efficient methods to search for triangles (better than O(n^3)).
* The baseline algorithms including K-means and co-authorship algorithms are explained unclearly. For example, what are the features used in K-means...
* The x-axis in Figure 5 is incorrect.
* Line 324, why w (the core) is the node with the maximal degree?

-- References: some important references are missing.
1. Investigation of triangle motifs in clustering over social networks was also discussed in quite a few literatures such as
"Motif Clustering and Overlapping Clustering for Social Network Analysis" by Pan Li et al. (INFOCOM17)
2. Conductance for motifs is essentially conductance for hypergraphs. Related important works are in
"Hypergraph Markov Operators, Eigenvalues and Approximation Algorithms" by Anand Louis (FOCS15)
"Submodular Hypergraphs: p-Laplacians, Cheeger Inequalities and Spectral Clustering" by Pan Li et al. (ICML18)
3. Alg3 on motif-based clustering essentially comes from learning on hypergraphs.
"Learning with Hypergraphs: Clustering, Classification, and Embedding" by Zhou et al. (NIPS07)

--Other typos:
equation 3: G_t -> G_k
line 188: Algorithm in ?
line 379: Fig.6b -> Fig 6a and Fig 6b
line 381: the smaller gap of team member performance? Grammarly incorrect!
Experimental design: This work is to propose a clustering algorithm for social networks and co-authorship networks and investigate more on the function of triangle motifs in social networks and co-authorship networks.

Compared to Benson's Science work, the algorithmic contribution is incremental. Some tricks are used in the clustering algorithms (Alg1+Alg2), while the authors did not explain well why they work. I suggest that the authors provide more explanations from both mathematical and experimental perspectives on these tricks. The baselines used for comparing the clustering performance are poor. I suggest that the authors compare the results based on sole Alg1, Alg1+Alg2, Alg1+Alg2+Alg3, sole Alg3 to let readers understand what is happening.

The good thing about this work is to provide some new comparison that may help researchers understand the function of triangle motifs. I suggest the authors provide more comparison from this point. I also suggest the authors provide additional experiments on other social network datasets (more than FB).
Validity of the findings: I think the impact and novelty are incremental unless the authors may provide better explanations of their clustering algorithms, especially the necessity of Alg1 and Alg2, and the authors may provide more evidence on the functions of triangle motifs.
Additional comments: No additional comments


Instruction: You are one of the reviewers, your task is to write a review for the article. You will be given the title of the article, the number of the round in which the article is located, and your order among the reviewers.

Input:
Title: COMICS: A COMMUNITY PROPERTY-BASED TRIANGLE MOTIF CLUSTERING SCHEME
Review round: 1
Reviewer: 2

Output:
Basic reporting: The paper is interesting and has novelty. It is well structured and well written in general. However it needs some extensions improvements and clarifications.

I suggest authors explicitly highlight their motivation to conduct the present work and also what is the exact motivation of the work.

How do the findings of the study contribute to the body of knowledge? Please present in better detail the functionality of the property-based triangle motif clustering scheme and how it would be assistive to the related research community.

The literature needs a better reviewing since there are many related quality works that authors could examine. Please examine additional works on the community detection which utilize also user characteristics.

Please point out the novelty of the work.

The results need a better discussion and the impact of the work needs to be highlighted.
Experimental design: In the experimental study please substantiated the choose of the three large scale networks that were utilized.

The results need a deeper discussion.

What are the reasons behind the performance of authors approach? Please explain in detail the performance results presented in figures 5&6.
Validity of the findings: The findings are interesting. However, the results need a better and deeper discussion.
Additional comments: The paper is interesting and has novelty. It is well structured and well written in general. However it needs some extensions improvements and clarifications.

I suggest authors explicitly highlight their motivation to conduct the present work and also what is the exact motivation of the work.

How do the findings of the study contribute to the body of knowledge? Please present in better detail the functionality of the property-based triangle motif clustering scheme and how it would be assistive to the related research community.

The literature needs a better reviewing since there are many related quality works that authors could examine. Please examine additional works on the community detection which utilize also user characteristics.

Please point out the novelty of the work.

The results need a better discussion and the impact of the work needs to be highlighted.

In the experimental study please substantiated the choose of the three large scale networks that were utilized.

The results need a deeper discussion.

What are the reasons behind the performance of authors approach? Please explain in detail the performance results presented in figures 5&6.

The findings are interesting. However, the results need a better and deeper discussion.


Instruction: You are one of the reviewers, your task is to write a review for the article. You will be given the title of the article, the number of the round in which the article is located, and your order among the reviewers.

Input:
Title: COMICS: A COMMUNITY PROPERTY-BASED TRIANGLE MOTIF CLUSTERING SCHEME
Review round: 2
Reviewer: 1

Output:
Basic reporting: The manuscript is improved a lot after authors' first-round revision overall. Some minor issues are left and list as the follows.
* For the added references to hypergraphs, such as Zhou et al. NIPS07, I suggest the authors may explain well the relation that graphs + motif-based structures essentially give hypergraphs. To understand this point, the authors can check Section S1.12 in the supplementary material of Benson's work in Science cited in this paper or the Introduction of "inhomogeneous hypergraph clustering with applications" by Pan Li et al. NIPS'17.
* Tri_2 in Figure 1: The set A should be {1, 3} according to the format of B_2 which shows node 2 is connected with node 1 and node 3. Please make them consistent.
* The authors should also revise their conclusion accordingly as they add new datasets. Please go through the manuscript to make all statements consistent with the new revisions.

For typos, the authors should carefully proofread the manuscript several times to make sure there are not typos. I found a lot even in the new added parts:
* line 195, "there is only graph are added" is not corrected in grammar.
* Algorithm 1, G_ij -> G_ij'
* line 222, do es -> does
Experimental design: The added explanation of Alg 1 and Alg 2 is greatly helpful. I feel pleased to see that.
Validity of the findings: As the explanations of Alg 1 and Alg 2 give insight, now I think the manuscript has novelty and impact.
Additional comments: No additional comments


Instruction: You are one of the reviewers, your task is to write a review for the article. You will be given the title of the article, the number of the round in which the article is located, and your order among the reviewers.

Input:
Title: COMICS: A COMMUNITY PROPERTY-BASED TRIANGLE MOTIF CLUSTERING SCHEME
Review round: 2
Reviewer: 2

Output:
Basic reporting: no comment
Experimental design: no comment
Validity of the findings: no comment
Additional comments: Authors have addressed my previous comments.


Instruction: You are one of the reviewers, your task is to write a review for the article. You will be given the title of the article, the number of the round in which the article is located, and your order among the reviewers.

Input:
Title: A SERENDIPITY-BIASED DEEPWALK FOR COLLABORATORS RECOMMENDATION
Review round: 1
Reviewer: 1

Output:
Basic reporting: Although there are some works in serendipitous recommendation, the problem of recommending serendipitous scientific collaborators is interesting. The innovation and novelty of this research are the definition and strategy of serendipity-biased network embedding. Their method balances the serendipity and accuracy of recommendation list effectively.

The paper proposes a method for identifying serendipitous collaborators. It defines serendipity in co-author network based on relevance, unexpectedness and value. These indices are well defined and works together to define serendipity. They improve the DeepWalk to learn the serendipity-biased vector representation of each vertex based on the linear combination of three indices. The proportion of relevance is smaller than unexpectedness and value according to their definition. I suggest authors to describe how to divide the research areas of scholars by LDA in the subsection of unexpectedness score.
Experimental design: To evaluate their work, they not only adopt the existing accuracy-based metrics, but also refer to the serendipity-based metrics in literature to measure their recommendation results. Their method outperforms other baseline methods in terms of the serendipity measurement, and maintains adequate accuracy at the same time.
Validity of the findings: The research is well motivated. The authors aim to design method for overcoming the defects of the traditional recommender system. The paper is also heuristic and applicable in scientific collaboration, and it may guide and inspire future efforts on the research of serendipity.
Additional comments: Minor comments:
1.	Line 218 in page 5: value score-> value scores.
2.	Line 384 in page 11: by evaluating the serendipity-based metrics -> by evaluating from the serendipity-based metrics.


Instruction: You are one of the reviewers, your task is to write a review for the article. You will be given the title of the article, the number of the round in which the article is located, and your order among the reviewers.

Input:
Title: A SERENDIPITY-BIASED DEEPWALK FOR COLLABORATORS RECOMMENDATION
Review round: 1
Reviewer: 2

Output:
Basic reporting: Please see 'General comments for the author'
Experimental design: Please see General comments for the author
Validity of the findings: Please see General comments for the author
Additional comments: The topic of this paper is fresh and interesting as it explores the concept of serendipitous scientific collaborators and designs a serendipity-biased recommender system. The components of relevance, unexpectedness and value are well quantified to define the serendipitous collaborators in the context of collaborators recommendation. The authors combine three quantified scores linearly to guide a serendipity-biased DeepWalk for leaning the vector representation of author node.

This research is novel and innovative, it tackles a fundamental problem in recommendation: the trade off between serendipity and accuracy. The scientific production scenario is perfect to analyze this problem given that diversity is a key point in the creativity process. I can see the expected characteristics of a serendipitous collaborator is: connection in the research network with the scholar target, influence in the research network and diversity in topic researches.

The paper introduces the motivation, goals and contributions of the presented research in the first section. The analysis of the literature in related work are clear and define well the scope of the paper. The authors take some recommendation methods from the literature cited as baseline methods for comparison, and refer to some serendipity-based metrics for measuring their recommendation results.

Their framework is improved from the traditional DeepWalk, where the sentence sequences are generated by performing a serendipity-biased random walk. The walker jumps to the next neighbor node with the proportional probability of edge weight in co-author network, where the edge weight is determined by the linear combination of three scores in definition. Finally, they compute the cosine similarity between vectors and extract the Top-N scholars who are serendipitous for target scholars. The experimental results validate the effectiveness of their method via the performance comparison between different baseline methods. Multiple parameter sets are also tested and analyzed in experiments for selecting the optimal set.

The paper reports a complete cycle from motivation to concept and technique development, and experimental observation.

Minor comments:
1.	Line 200 in page 4: it across  it crosses.
2.	Line 221 in page 5: relevance, unexpectedness and value score  relevance, unexpectedness and value scores.
3.	Line 384 in page 11: by evaluating the serendipity-based metrics  by evaluating from the serendipity-based metrics.


Instruction: You are one of the reviewers, your task is to write a review for the article. You will be given the title of the article, the number of the round in which the article is located, and your order among the reviewers.

Input:
Title: A SERENDIPITY-BIASED DEEPWALK FOR COLLABORATORS RECOMMENDATION
Review round: 2
Reviewer: 1

Output:
Basic reporting: no comment
Experimental design: no comment
Validity of the findings: no comment
Additional comments: All my concerns have been issued well.


Instruction: You are one of the reviewers, your task is to write a review for the article. You will be given the title of the article, the number of the round in which the article is located, and your order among the reviewers.

Input:
Title: A SERENDIPITY-BIASED DEEPWALK FOR COLLABORATORS RECOMMENDATION
Review round: 2
Reviewer: 2

Output:
Basic reporting: no comment.
Experimental design: no comment
Validity of the findings: no comment
Additional comments: The revisions are satisfactory.


Instruction: You are one of the reviewers, your task is to write a review for the article. You will be given the title of the article, the number of the round in which the article is located, and your order among the reviewers.

Input:
Title: A SERENDIPITY-BIASED DEEPWALK FOR COLLABORATORS RECOMMENDATION
Review round: 2
Reviewer: 3

Output:
Basic reporting: All the reviewers' suggestions were addressed and the manuscript was improved for what concerns the clarity of the presented methodology and results, as well as, the text correctedness.
Experimental design: I have just one minor request. The plots of figures 3, 4 and 5 should be improved regarding its arrangement in the manuscript. Also, the plots of theses figures should be larger. I think it is better, for instance, to split the plots of precision, recall, and so on, in more rows per figure. This way the readability of those figures will be improved. The way they are shown now, the reader struggles to visualize the information. The plot of Figure 6 should also be improved.
Validity of the findings: The main topic of the paper, the serendipitous scientific collaborators, is very interesting and is well approached by the authors. The experimental setup is effective and the evaluation methods are statistically sound in order to validate the obtained results.
Additional comments: No additional comments


Instruction: You are one of the reviewers, your task is to write a review for the article. You will be given the title of the article, the number of the round in which the article is located, and your order among the reviewers.

Input:
Title: A SERENDIPITY-BIASED DEEPWALK FOR COLLABORATORS RECOMMENDATION
Review round: 2
Reviewer: 4

Output:
Basic reporting: - Professional English, clear, and coherent.  Only small corrections need to be made in some text segments at lines 44 and 46.
- It would be important to review the abstract of the article again.
Experimental design: no comment
Validity of the findings: no comment
Additional comments: - The proposal of the article in approaching the problem of scientific collaboration recommendation is interesting, innovative and novel. The authors proposed the implementation of a recommendation system to generate ideal or appropriate scientific collaborators among scholars. In addition, representative vectors or embeddings were generated for each author or scholar (each node of a co-authoring network) proposing the Seren2vec methodology, which was compared with different algorithms for vector representation of nodes such as Node2vec, DeepWalk, among others.  A series of experiments were performed with a significant and important database such as DBLP. In addition, these experiments were evaluated using different existing metrics in the literature for the validation of recommendation systems. The results showed that the proposed method surpasses other baseline methods that also maintain a correct accuracy.
- The article does not have many things to correct, because it is written with a clear, coherent and professional English. In addition, the article is well structured, organized and the proposal and results are clearly defined.  The text segments that need to be corrected are the following:
    * Line 44: quantified by us as -> quantified as
    * Line 46: the text  “research topics from that of their target scholars” is not clear, it needs to be improved.
- The analysis of the literature is quite good, well ordered and structured since it clearly divides the main methods that address the task of recommendation systems. It is also good that some works about serendipity in recommender systems are presented.
- Regarding the abstract, with the aim of making it clearer,  it would be convenient to make a small modification regarding the use and definition of the term Serendipity, since it is a little-known term. At the beginning you name several terms that include the word Serendipity (serendipitous scientific collaborators, serendipity-biased vector representation or serendipitous encounters) and after you give an explanation about that term. It would be ideal that you first try to define the term Serendipity with some related examples and then you use the terms that include the word Serendipity (serendipitous scientific collaborators, serendipity-biased vector representation or serendipitous encounters).


Instruction: You are one of the reviewers, your task is to write a review for the article. You will be given the title of the article, the number of the round in which the article is located, and your order among the reviewers.

Input:
Title: A SERENDIPITY-BIASED DEEPWALK FOR COLLABORATORS RECOMMENDATION
Review round: 3
Reviewer: 1

Output:
Basic reporting: no comments
Experimental design: no comments
Validity of the findings: no comments
Additional comments: All the requested changes were made by the authors.


Instruction: You are one of the reviewers, your task is to write a review for the article. You will be given the title of the article, the number of the round in which the article is located, and your order among the reviewers.

Input:
Title: A SERENDIPITY-BIASED DEEPWALK FOR COLLABORATORS RECOMMENDATION
Review round: 3
Reviewer: 2

Output:
Basic reporting: no comment
Experimental design: no comment
Validity of the findings: no comment
Additional comments: The revisions are satisfactory.


Instruction: You are one of the reviewers, your task is to write a review for the article. You will be given the title of the article, the number of the round in which the article is located, and your order among the reviewers.

Input:
Title: INTEGRATING THE RFID IDENTIFICATION SYSTEM FOR CHAROLAISE BREEDING BULLS WITH 3D IMAGING FOR VIRTUAL ARCHIVE CREATION
Review round: 1
Reviewer: 1

Output:
Basic reporting: Paper presents a proposal of integration between electronic identification (EID) and 3D Digital Image of animal. Authors developed a SWOT analysis of actual bull morphological and functional evaluation (2D).  Then they carried out a 3D model of the living animal, which presents to be adherent to live animial morphology.
Experimental design: The development of a SWOT analysis requires a deep understanding of the subject. Although authors presents a good literature review, the SWOT lacks of a methodological approach. It is important to describe how the proposed statements of the SWOT analysis were collected and validated.

Section 2.6 (calculation, data analysis and statistics) presents the performance of the proposed technique. Authors could describe more about how such metrics were calculated and what is a 'match' between virtual measures and real measures.  It is also important to provide an analysis based on TPR and TNR. Is there any recall issues in the results?. Present a table summarizing the results can be a good way for clarification.
Validity of the findings: Results are presented at the paper, but authors could improve the discussion, presenting detailed statistic.

Conclusion are well stated and linked to original research question.
Additional comments: The paper is well written but you could improve the details of the applied method and presents a more descriptive result analysis.  It is recommended to present the user's needs prior to the S.W.O.T analysis results. You could also describe (even with the help of a picture) how you conducted the series of measurements.


Instruction: You are one of the reviewers, your task is to write a review for the article. You will be given the title of the article, the number of the round in which the article is located, and your order among the reviewers.

Input:
Title: INTEGRATING THE RFID IDENTIFICATION SYSTEM FOR CHAROLAISE BREEDING BULLS WITH 3D IMAGING FOR VIRTUAL ARCHIVE CREATION
Review round: 1
Reviewer: 2

Output:
Basic reporting: The article shows interesting technical to measure morphology in beef cattle.
Introduction and backgrounds are clear to show the context. Tables and figures are relevant and necessary  to show the experiment and the results.
I think that material could be better explained and also, the analysis. The results  could be better explored, including more tables with detailed calculations for PPV, TPR, TNP and acc.
Experimental design: The authors could explain better the SWOT matrix and why it was used in the context of the article, because it is not clear.

The authors should explain better the analysis of variance and in detail, what were the series of measurements. Did you use two bulls and how many measures by each one?

I consider that there was a small number of animals used in the experiment (only two bulls) a to test the 3-D against the standard method. Could you explain better this point of view?
Validity of the findings: The authors should explore better the results of data analysis, mainly the calculations of PPV, TPR, TNP and ACC, showing these results in a table.

I have doubts about the necessary number of animals to validate such methodology. Only two bulls are enough? P

The implementation of 3-D images has many advantages, but only in an automated system of evaluation. Mainly to the animal breeding application, that is necessary to measure in many animals.
Additional comments: The authors presented an interesting methodology to show alternative ways of measuring animals. 3-D images could be implemented in some official system to evaluate animals, as long as its implementation becomes easy, cheap and automated.


Instruction: You are one of the reviewers, your task is to write a review for the article. You will be given the title of the article, the number of the round in which the article is located, and your order among the reviewers.

Input:
Title: INTEGRATING THE RFID IDENTIFICATION SYSTEM FOR CHAROLAISE BREEDING BULLS WITH 3D IMAGING FOR VIRTUAL ARCHIVE CREATION
Review round: 1
Reviewer: 3

Output:
Basic reporting: *** The literary references are not appropriate to the context. Lines 60-67 report "recent research efforts ongoing worldwide" on "sophisticated genomic techniques", but the most recent reference is from 2007.

*** The results should be better presented (section 3).
Experimental design: *** I am not sure if I could be replicate the experiment. Methods do not describe with sufficient detail and information to replicate. The methodology description (section 2) is short and show few details.
Validity of the findings: *** The discussions (section 4) need to be enriched, since they have little basis and some opinions derived from empirical observations. For this reason, the conclusion (section 5) is not appropriate because it does not provide solid elements of the discussion of the results.
Additional comments: 1) The improvement obtained was not mentioned in the abstract. The abstract should make mention (at least) of the improvement obtained;
2) The authors used clear, technical and unambiguous writing;
3) The manuscript presents professional structure and auxiliary elements (figures, tables etc.).
4) The necessary definitions have been included and they are clear;
5) The research problem is aligned with the scope of the journal.
6) The research question is not "a question" because it is wide. The question represents a improvement in the solution of a problem, which is relevant and meaningful. The propositions introduce how the study contributes to filling the gaps of problem;
7) Data are controlled and the statistics used are clear and effective, though simple.


Instruction: You are one of the reviewers, your task is to write a review for the article. You will be given the title of the article, the number of the round in which the article is located, and your order among the reviewers.

Input:
Title: SKILL RANKING OF RESEARCHERS VIA HYPERGRAPH
Review round: 1
Reviewer: 1

Output:
Basic reporting: In this manuscript, the authors use a methodology for analyzing scientific skills and employ a method for evaluation. The authors represent the relationship between researchers, fields, and skills through a tripartite graph, which is converted into a hypergraph.

1 - The quality of the figures is not appropriate regarding the number of DPI. So, it is difficult to read the numbers. In particular, I believe that the graphical quality of Figure 2b shall be improved.

2 - The overall English shall be improved. Some examples are presented as follows:
- When you use the word "heterogenous", I think you meant "heterogeneous". Additionally, you used “data set” instead of “dataset”.

- In the abstract, the phrase “Finally, we analyze the pattern how...”, a preposition is missing after the word “pattern”;

-In introduction:
    - instead of "Scientific cross-disciplinary collaboration brings positive effect to...".  you should write "Scientific cross-disciplinary collaboration brings positive effects to...";
    - "A dozen of state-of-art works have been proposed..." should be "A dozen state-of-art works has been proposed...";
    - "The index has been proved to be an simple..." should be "The index has been proved to be a simple...".
    - "In recent work, methods of ranking authors in heterogeneous network..." can be replaced by "In recent work, methods of ranking authors in an heterogeneous network..."
    - " They can not reflect..." should be " They cannot reflect..."

In Feature description:
    - “Facing the high competitive...” should be “Facing the highly competitive...”
    - “… and researcher’s skill ...” should be “and the researcher’s skill ”

Note that I depicted here only some cases of grammar issues. So, there are many others, and I believe that the writing shall be better revised.

3 - b) You used the letters $\gamma$ and $W$ in equations (1), (2), (3), and (4). It can confuse the reader. In the case of equations 1, 2, and 3, Iou could use $W_1$, $W_3$, and $W_3$, respectively, as adopted in equation (5).

4 - I believe that you could better explain about the two versions of ranks ($\gamma$).

5 - In line 220, you mentioned a section called " Feature", but the section name is "Feature description".

6 - In equation 7, information regarding the variables $n_1$ and $n_2$ are missing.

7 – The measurement presented in Figure 5 is not clear.

8 - When you mentioned that "the transformation of the field brings researchers more inspiration and creativity, contributing to the success of their scientific researches", I think that you cannot make this affirmation because you do not have enought elements. To fix this problem, you can say that "the transformation of the field can bring...".
Experimental design: 1- I'm not sure about the result presented in Figure 4. Because of the log scale in the y-axis, the curves seem to follow the same behavior. However, if you compare the differences in skill rankings between the points in which h-indexes are 1 and 19, the difference is in the order of magnitude of 10^4. In other words, the slopes of the curves are much different.

2- In Hypothesis test section, I could not understand why to validate your method you considered that "there are significant difference between the paper citation
counts and the average skill rank of its authors". This information is critical in this section and shall be better explained.
Validity of the findings: 1 - You affirmed that
“It has been found that collaborations cross disciplines are more competitive and more likely to gain high quality outputs. First, the development of the collaborative relationships between multi-disciplines makes researchers realize the advantages of resource sharing, which can make a better data and knowledge utilization either within a team or in the whole society.”
However, you cannot say that the articles that are most frequent have more quality or are more competitive.

2 - I did not understand the following text: "We start our analysis by calculating the change rate of researchers’ skill ranking in every two years. We have calculated the skill ranking of researchers every year, as referred in Network Construction section. We calculate the change rate of skill ranking between every two adjacent year i and i − 1, that is..."
Because of this part of the text, I could not comprehend the analysis regarding Figure 5.
Additional comments: The subject and the proposed model are valuable and can be applied in other related problems.  So, I believe that the article is suitable to be published in "PeerJ - Computer Science journal" after a major revision.


Instruction: You are one of the reviewers, your task is to write a review for the article. You will be given the title of the article, the number of the round in which the article is located, and your order among the reviewers.

Input:
Title: SKILL RANKING OF RESEARCHERS VIA HYPERGRAPH
Review round: 1
Reviewer: 2

Output:
Basic reporting: no comment
Experimental design: no comment
Validity of the findings: no comment
Additional comments: The authors present a new model on factors related to researchers' skill.
The tool considered in the model is a hypergraph (a heterogeneous network).
All experiments were carried out considering the PloS One bibliographic dataset
(more than 164000 papers coauthored by more than 684000 researchers).

In general, the treatment of the problem is technically sound. However, there
are some questions regarding the presentation and results which require
attention:

* 1. The conclusion stated in the Abstract seems to me obvious: "Our studies
show that many researchers are playing different roles in the different stages
of their academic career." Please explain in more detail the contribution of the
work.

* 2. The description and the mathematical formulation to construct the SRModel
framework models were well introduced in this work.  Would it be possible to
present a summary of all the steps in the form of an algorithm?  About the
arithmetic complexity of the considered steps, how are the computing time and
memory usage?

* 3. Figure 2 lacks details. Fig. 1(a): Please explain what are the 11 hybrid vertices in the graph. Fig.2(b): I did not understand (properly) how are built the hyper edges.

* 4. What was the data source considered for the h-indices? Please justify
the choice of data source, and indicate the period of data collection.

* 5. The review of prior literature is reasonable. The reference list should be carefully reviewed. For instance:
- Rahman, M. T., Mac Regenstein, J., Kassim, N. L. A., and Haque, N. (2017). The need to quantify
authorsŕelative intellectual contributions in a multi-author paper. Journal of Informetrics, 11(1):275-81


Instruction: You are one of the reviewers, your task is to write a review for the article. You will be given the title of the article, the number of the round in which the article is located, and your order among the reviewers.

Input:
Title: SKILL RANKING OF RESEARCHERS VIA HYPERGRAPH
Review round: 1
Reviewer: 3

Output:
Basic reporting: The manuscript entitled "Skill Ranking of Researchers via Hypergraph" presents a study about the ranking of researchers using hypergraph. I have the following comments

- I strongly recommend that the manuscript be reviewed by an expert in the English language for both the composition of the sentences and the grammar review. There are many wrong expressions in many parts of the text. Although the main topic is very interesting to the readers of this journal, the lack of text correctedness affects the quality of the manuscript.

- The related works are well structured and the main background of the manuscript is provided. The raw data has also been provided by the authors.

- The caption of Figure 1 is wrong. The correct form could be "The distribution of number of fields for ALL the papers".

- The drawing of Figure 2-b) needs to be improved.

- The caption of Figure 3 is confusing. I would remove this figure because the information that is shown in the pie chart is clearly discussed in the text and doesn't need illustration.

- The sections of the manuscript are well structured, however the name of the second section ("Feature description") is very generic and does not reflect its content. This section looks more like a background/conceptualization section.

- The section "Analysis Methods" (which includes the definitions of Correlation Coefficient and Hypothesis Test) is not necessary because it brings definitions that are very well known.

- The name of the subsection "Hypothesis test" should be changed for something that describes the purpose of the section.
Experimental design: The main topic of the manuscript is within the scope of this journal and the research questions are well defined.

- In the section "Dataset Construction" the authors mention that there are no standard rules for naming contributions in PLoS One journal. However, nowadays the journal adopts the CRediT Taxonomy to describe the author's individual contributions. This should be clarified in the text.

- It is not clear in the section "SRModel" how the ranks are defined in equations 1, 2 and 3, which are then used to define the weight of the edges.
Validity of the findings: The authors present an interesting method which performs skill ranking analysis of researchers. I think the main contribution is the analysis of the changing pattern of skill ranking. But I would like to highlight the following issues with the proposed method.

What are the challenges of using data from other journals? As the authors wrote, PLoS One cannot include all the papers published by each researcher. Using data only from this journal can be a source of bias. Can the same procedure be employed to construct a similar dataset from other journals? What would be the limitations of such approach?

Since the distributions of skill ranking and h-index are very similar and they present a high correlation, what is the novelty of the proposed approach concerning its applicability? A discussion on what are practical problems on which the skill raking can be applied is still missing in text.

One of the contributions listed by the authors is "Data retrieving". What do they mean in this case? The creation of a dataset? In this case, it should be detailed in the text how this dataset can be accessed and manipulated by the readers.
Additional comments: No additional comments


Instruction: You are one of the reviewers, your task is to write a review for the article. You will be given the title of the article, the number of the round in which the article is located, and your order among the reviewers.

Input:
Title: SKILL RANKING OF RESEARCHERS VIA HYPERGRAPH
Review round: 2
Reviewer: 1

Output:
Basic reporting: The majority of the pointed issues were fixed.
Experimental design: no comment
Validity of the findings: no comment
Additional comments: 1 - I believe that the writing shall be improved. There are still many grammar mistakes.

2 - Regarding the figures, you could consider using a vectorial format (e.g., pdf or eps).

3 - In the case of Fig. 6, you could include the axis labels.


Instruction: You are one of the reviewers, your task is to write a review for the article. You will be given the title of the article, the number of the round in which the article is located, and your order among the reviewers.

Input:
Title: SKILL RANKING OF RESEARCHERS VIA HYPERGRAPH
Review round: 2
Reviewer: 2

Output:
Basic reporting: no comments
Experimental design: no comments
Validity of the findings: no comments
Additional comments: All the reviewers' suggestions were addressed and the manuscript was improved for what concerns the clarity of the presented methodology and results, as well as, the text correctedness.


Instruction: You are one of the reviewers, your task is to write a review for the article. You will be given the title of the article, the number of the round in which the article is located, and your order among the reviewers.

Input:
Title: SKILL RANKING OF RESEARCHERS VIA HYPERGRAPH
Review round: 3
Reviewer: 1

Output:
Basic reporting: The text is much better than in the previous version. Besides, almost all the reviewers' suggestions were addressed. So, the manuscript is suitable to be published. The single remark is that the images are still not in vector image format because when you zoom the images it is possible to see the pixels (the single figure that is in vector format is Fig. 5).
Experimental design: no comment
Validity of the findings: no comment
Additional comments: no comment


Instruction: You are one of the reviewers, your task is to write a review for the article. You will be given the title of the article, the number of the round in which the article is located, and your order among the reviewers.

Input:
Title: SKILL RANKING OF RESEARCHERS VIA HYPERGRAPH
Review round: 3
Reviewer: 2

Output:
Basic reporting: no comment
Experimental design: no comment
Validity of the findings: no comment
Additional comments: The paper has been revised accordingly.

The authors thoroughly clarified all the points I raised in my previous review. I have no further objections against publication.


Instruction: You are one of the reviewers, your task is to write a review for the article. You will be given the title of the article, the number of the round in which the article is located, and your order among the reviewers.

Input:
Title: SNARE-CNN: A 2D CONVOLUTIONAL NEURAL NETWORK ARCHITECTURE TO IDENTIFY SNARE PROTEINS FROM HIGH-THROUGHPUT SEQUENCING DATA
Review round: 1
Reviewer: 1

Output:
Basic reporting: See below
Experimental design: See below
Validity of the findings: See below
Additional comments: In this paper, the authors proposed a model to identifying SNARE proteins. SNARE proteins — "SNAP" (Soluble NSF Attachment Protein) REceptor" — are a large protein complex consisting of at least 24 members in yeasts and more than 60 members in mammalian cells. The primary role of SNARE proteins is to mediate vesicle fusion, that is, the fusion of vesicles with their target membrane bound compartments (such as a lysosome). SNAREs are the targets of the bacterial neurotoxins responsible for botulism and tetanus. The authors’ motivation is correct and their efforts should be encouraged. In view of this, the paper holds potential for publication. But to meet the increasingly high quality standard of PeerJ, a careful revision is needed as detailed below.

(1) To make this paper logically more clear, operatively more transparent, and practically more useful, the authors should in the end of the Introduction (or right before the beginning of describing their own method) add a prelude, such as: “As shown in a series of recent publications [1-8], to develop a really useful statistical predictor for a biological  system, one should observe the guidelines of  Chou’s 5-step rule [9] to make the following five steps very clear: (i) how to construct or select a valid benchmark dataset to train and test the predictor; (ii) how to formulate the statistical samples with an effective mathematical expression that can truly reflect their intrinsic correlation with the target to be predicted; (iii) how to introduce or develop a powerful algorithm (or engine) to operate the prediction; (iv) how to properly perform cross-validation tests to objectively evaluate the anticipated accuracy of the predictor; (v) how to establish a user-friendly web-server for the predictor that is accessible to the public. Below, we are to describe how to deal with these steps one-by-one.”  With such a prelude, the outline of this paper and its goal would be crystal clear; its reported results easier for others to repeat. And its attraction to the readership and impact to science would be much higher as well.

(2) It is a big plus to use the intuitive set of metrics (Eqs.5-8) to replace the traditional ones copied from math books as done by many in literature.  However, in changing fundamental things, such as metrics or rules, the authors should give more description, such as: “Based on the Chou’s symbols introduced for studying protein signal peptides [10], a set of four intuitive metrics were derived [11, 12], as given in Eq.14 of [11] or in Eq.19 of  [12].”  Also, the authors should cite as many credible publications [11, 13-23] as possible to justify their decision.  Furthermore, it is instructive for the authors to add the following in-depth discussion as saying “Either the set of traditional metrics copied from math books or the intuitive metrics derived from the Chou’s symbols (Eqs.5-8) are valid only for the single-label systems (where each sample only belongs to one class). For the multi-label systems (where a sample may simultaneously belong to several classes), whose existence has become more frequent in system biology [24-27], system medicine [28, 29] and biomedicine [30], a completely different set of metrics as defined in [31] is absolutely needed.”

(3) It is one more big plus that the authors have provided a web-server for their prediction model. To further stress such an advantage, the authors should in the relevant context add a discussion: “User-friendly and publicly accessible web-servers represent the current trend for developing various computational methods [1-8]. Actually they have significantly enhance the impacts of computational biology on medical science [32], driving medicinal chemistry into an unprecedented revolution [33], here we also provide a web-server at https://github.com/khanhlee/snare-cnn for the new method reported in this paper.”

REFERENCES

[1] W. Chen, H. Ding, X. Zhou, H. Lin, iRNA(m6A)-PseDNC: Identifying N6-methyladenosine sites using pseudo dinucleotide composition. Analytical Biochemistry 561-562 (2018) 59-65.
[2] X. Cheng, X. Xiao, pLoc-mEuk: Predict subcellular localization of multi-label eukaryotic proteins by extracting the key GO information into general PseAAC. Genomics 110 (2018) 50-58.
[3] P. Feng, H. Yang, H. Ding, H. Lin, W. Chen, iDNA6mA-PseKNC: Identifying DNA N6-methyladenosine sites by incorporating nucleotide physicochemical properties into PseKNC. Genomics doi:10.1016/j.ygeno.2018.01.005 (2018).
[4] Y.D. Khan, N. Rasool, W. Hussain, S.A. Khan, iPhosT-PseAAC: Identify phosphothreonine sites by incorporating sequence statistical moments into PseAAC. Analytical Biochemistry 550 (2018) 109-116.
[5] X. Xiao, Z.C. Xu, W.R. Qiu, P. Wang, H.T. Ge, iPSW(2L)-PseKNC: A two-layer predictor for identifying promoters and their strength by hybrid features via pseudo K-tuple nucleotide composition. Genomics doi:10.1016/j.ygeno.2018.12.001 (2018).
[6] X. Cheng, X. Xiao, pLoc-mGneg: Predict subcellular localization of Gram-negative bacterial proteins by deep gene ontology learning via general PseAAC. Genomics 110 (2018) 231-239.
[7] K.C. Chou, X. Cheng, X. Xiao, pLoc_bal-mHum: predict subcellular localization of human proteins by PseAAC and quasi-balancing training dataset Genomics doi:10.1016/j.ygeno.2018.08.007 (2018).
[8] J. Jia, X. Li, W. Qiu, X. Xiao, iPPI-PseAAC(CGR): Identify protein-protein interactions by incorporating chaos game representation into PseAAC. Journal of Theoretical  Biology 460 (2019) 195-203.
[9] K.C. Chou, Some remarks on protein attribute prediction and pseudo amino acid composition (50th Anniversary Year Review). Journal of Theoretical Biology 273 (2011) 236-247.
[10] K.C. Chou, Prediction of signal peptides using scaled window. Peptides 22 (2001) 1973-1979.
[11] W. Chen, P.M. Feng, H. Lin, iRSpot-PseDNC: identify recombination spots with pseudo dinucleotide composition Nucleic Acids Research   41 (2013) e68.
[12] Y. Xu, X.J. Shao, L.Y. Wu, N.Y. Deng, iSNO-AAPair: incorporating amino acid pairwise coupling into PseAAC for predicting cysteine S-nitrosylation sites in proteins. PeerJ 1 (2013) e171.
[13] H. Lin, E.Z. Deng, H. Ding, W. Chen, iPro54-PseKNC: a sequence-based predictor for identifying sigma-54 promoters in prokaryote with pseudo k-tuple nucleotide composition. Nucleic Acids Research 42 (2014) 12961-12972.
[14] F. Ali, M. Hayat, Classification of membrane protein types using Voting Feature Interval in combination with Chou's Pseudo Amino Acid Composition. J Theor Biol 384 (2015) 78-83.
[15] W. Chen, P. Feng, H. Ding, H. Lin, iRNA-Methyl: Identifying N6-methyladenosine sites using pseudo nucleotide composition. Analytical Biochemistry 490 (2015) 26-33.
[16] J. Jia, Z. Liu, X. Xiao, iPPI-Esml: an ensemble classifier for identifying the interactions of proteins by incorporating their physicochemical properties and wavelet transforms into PseAAC. J Theor Biol 377 (2015) 47-56.
[17] B. Liu, L. Fang, R. Long, X. Lan, iEnhancer-2L: a two-layer predictor for identifying enhancers and their strength by pseudo k-tuple nucleotide composition. Bioinformatics 32 (2016) 362-369.
[18] W. Chen, P. Feng, H. Yang, H. Ding, H. Lin, iRNA-AI: identifying the adenosine to inosine editing sites in RNA sequences. Oncotarget 8 (2017) 4208-4217.
[19] P.K. Meher, T.K. Sahu, V. Saini, A.R. Rao, Predicting antimicrobial peptides with improved accuracy by incorporating the compositional, physico-chemical and structural features into Chou's general PseAAC. Sci Rep 7 (2017) 42362.
[20] P. Feng, H. Ding, H. Yang, W. Chen, H. Lin, iRNA-PseColl: Identifying the occurrence sites of different RNA modifications by incorporating collective effects of nucleotides into PseKNC. Molecular Therapy - Nucleic Acids 7 (2017) 155-163.
[21] W.R. Qiu, B.Q. Sun, X. Xiao, Z.C. Xu, J.H. Jia, iKcr-PseEns: Identify lysine crotonylation sites in histone proteins with pseudo components and ensemble classifier. Genomics 110 (2018) 239-246.
[22] M. Arif, M. Hayat, Z. Jan, iMem-2LSAAC: A two-level model for discrimination of membrane proteins and their types by extending the notion of SAAC into Chou's pseudo amino acid composition. J Theor Biol 442 (2018) 11-21.
[23] M.S. Krishnan, Using Chou's general PseAAC to analyze the evolutionary relationship of receptor associated proteins (RAP) with various folding patterns of protein domains. J Theor Biol 445 (2018) 62-74.
[24] X. Cheng, X. Xiao, pLoc-mPlant: predict subcellular localization of multi-location plant proteins via incorporating the optimal GO information into general PseAAC. Molecular BioSystems 13 (2017) 1722-1727.
[25] X. Cheng, X. Xiao, pLoc-mVirus: predict subcellular localization of multi-location virus proteins via incorporating the optimal GO information into general PseAAC. Gene (Erratum: ibid., 2018, Vol.644, 156-156) 628 (2017) 315-321.
[26] X. Xiao, X. Cheng, G. Chen, Q. Mao, pLoc_bal-mGpos: predict subcellular localization of Gram-positive bacterial proteins by quasi-balancing training dataset and PseAAC. Genomics doi:10.1016/j.ygeno.2018.05.017 (2018).
[27] X. Cheng, S.G. Zhao, W.Z. Lin, X. Xiao, pLoc-mAnimal: predict subcellular localization of animal proteins with both single and multiple sites. Bioinformatics 33 (2017) 3524-3531.
[28] X. Cheng, S.G. Zhao, X. Xiao, iATC-mISF: a multi-label classifier for predicting the classes of anatomical therapeutic chemicals. Bioinformatics  (Corrigendum, ibid., 2017, Vol.33, 2610) 33 (2017) 341-346.
[29] X. Cheng, S.G. Zhao, X. Xiao, iATC-mHyb: a hybrid multi-label classifier for predicting the classification of anatomical therapeutic chemicals. Oncotarget 8 (2017) 58494-58503.
[30] W.R. Qiu, B.Q. Sun, X. Xiao, Z.C. Xu, iPTM-mLys: identifying multiple lysine PTM sites and their different types. Bioinformatics 32 (2016) 3116-3123.
[31] K.C. Chou, Some remarks on predicting multi-label attributes in molecular biosystems. Molecular Biosystems 9 (2013) 1092-1100.
[32] K.C. Chou, Impacts of bioinformatics to medicinal chemistry. Medicinal Chemistry 11 (2015) 218-234.
[33] K.C. Chou, An unprecedented revolution in medicinal chemistry driven by the progress of biological science. Current Topics in Medicinal Chemistry 17 (2017) 2337-2358.


Instruction: You are one of the reviewers, your task is to write a review for the article. You will be given the title of the article, the number of the round in which the article is located, and your order among the reviewers.

Input:
Title: SNARE-CNN: A 2D CONVOLUTIONAL NEURAL NETWORK ARCHITECTURE TO IDENTIFY SNARE PROTEINS FROM HIGH-THROUGHPUT SEQUENCING DATA
Review round: 1
Reviewer: 2

Output:
Basic reporting: The paper has some typos and difficult to follow sections. I’ve tried to highlight some of the more confusing sections in the following comments.

Minor Points:

Typo in abstract (76.6%%, should be 76.6%)
Experimental design: It isn’t clear how cross-validation and independent data sets were produced. As far as I can tell, Uniprot was searched for all SNARE proteins and then proteins that shared over 30% similarity were removed, leaving 245 SNARE proteins. Figure 1, seems to indicate that there are 682 SNAREs used to build and test the model. Where were the extra SNARE proteins found? Is the 682 number found without any similarity filtering?

Assuming I’ve read the dataset section correctly, all of the non-SNARE proteins tested are classified as vesicular transport proteins. I’m concerned that the model is thus trained to only differentiate between SNARE proteins and VT proteins. If you put a random collection of protein sequences through the network, does the model always predict non-SNARE?

I’m also confused as to how PSI-BLAST was used to produce the PSSM profiles. It appears that a PSSM is built for each protein used to train and test the model. If you used the command line version of PSI-BLAST, can you provide the script or command used to produce the profiles? Also, I don’t understand the part of Figure 1, where all of the A rows are summed up. What is represented in the rows in that image?
Validity of the findings: No comments, beyond the previous sections.
Additional comments: I think the most interesting part of this paper is adapting the convolutional NN type for PSSM profiles. Treating the PSSMs as essentially grayscale images is an interesting if unexpected attempt to force this data type into something easier for the neural network to deal with.

Given that one of the goals of the study is to build a model to hopefully find new SNARE proteins (presumably in newly sequenced organisms), I’m a bit surprised that you didn’t make the comparison to simply BLASTing the SNARE and non-SNARE sequences.  The output could then be assessing as to whether the first non-identical match was a SNARE/non-SNARE protein. I understand if the true goal was to only assess modern classification methods, but given that PSI-BLAST is already in the pipeline for building the PSSMs, this would seem to be a relevant comparison.