Instruction: You are one of the reviewers, your task is to write a review for the article. You will be given the title of the article, the number of the round in which the article is located, and your order among the reviewers.

Input:
Title: QUANTIFYING THE IMPACT OF NON-STATIONARITY IN REINFORCEMENT LEARNING-BASED TRAFFIC SIGNAL CONTROL
Review round: 1
Reviewer: 1

Output:
Basic reporting: The manuscript is written and organised well. Apart from a few typos and grammar issues, readability is very good. Please correct situations such as:

"We them analyze..." -> "We then analyze..."?
"the impact that each one has in the learning process..." -> "on the learning process"?
"policy π^∗The rewards" -> a full stop and space are perhaps missing.

Authors also provide sufficient background underlying the preliminaries of their approach and discuss the related work to emphasise on their contributions. References seem to be appropriate.

Figures/graphics/plots are also readable and easily understandable. And results are representative of and in accordance with the authors' methodological approach and discussion.
Experimental design: Authors are interested in analysing the effect of causes of nonstationarity in their model-free RL approach to adaptive traffic signal control. Causes of nonstationarity, as listed by the authors in their study, include traffic flow pattern and the partial-observability ability of traffic signal agents. With this goal in mind, authors present different simulation setups to test their RL traffic signal agents in a grid-like traffic network modelled under the SUMO microscopic simulator.

In this perspective, some concepts might be better discussed, though. Whereas traffic flow in urban settings is potentially nonstationary, it is also true that some patterns (although stochastic) are easily observed, such as the daily 24-hour flow profile, in which off-peak and peak hours are identified. Stochasticity not necessarily implies nonstationarity, as some systems may well work under dynamic stability. Certainly, such dynamic stability may well be disturbed by unexpected events, such as accidents, or even by priority vehicles like ambulances demanding other vehicles to give the right of way. From what this reviewer could understand, it seems that the "nonstationary traffic flow" is rather yielded by a random OD demand imposed by authors in their different simulation scenarios. On the other hand, limitations in agents' observability capacity do not seem to cause itself nonstationary regimes upon the environment as might do agents' actions. Rather than a cause, it seems to be related to how sometimes such limitation might well influence rationality even in (dynamic) stationary environments making them seem rather nonstationary to the sensor-deficient agents, affecting this way the soundness of their decision making.
Validity of the findings: Results are promising (although rather intuitive) and are discussed quite well. Whether being able to observe (sense) "densities", perhaps this might also be influenced by how often vehicles' speed drops below 0.1 m/s on a link and by the distance to be traversed. The best-case scenario is when a vehicle traverses a link on free-flow speed and does not need to stop at the junction; performance degrades when flow approaches saturation, affecting densities. Isn't this behaviour of the link performance implicitly embedded in the queue signal perceived by the agents?

Also, authors have selected Total Waiting Time as the main metric to analyse. Perhaps considering total/average travel time for the whole network would allow for evaluating the system's performance as a whole (collectively). That could be an interesting analysis too.
Additional comments: Generally, the work is quite relevant and makes a good appreciation of related efforts to emphasise the authors' contributions. Some aspects, especially conceptual ones, could improve benefiting from a better discussion and deeper reflection.


Instruction: You are one of the reviewers, your task is to write a review for the article. You will be given the title of the article, the number of the round in which the article is located, and your order among the reviewers.

Input:
Title: QUANTIFYING THE IMPACT OF NON-STATIONARITY IN REINFORCEMENT LEARNING-BASED TRAFFIC SIGNAL CONTROL
Review round: 1
Reviewer: 2

Output:
Basic reporting: no comment
Experimental design: no comment
Validity of the findings: no comment
Additional comments: This paper applies a multiagent independent Q-learning method to study the traffic signal control, especially for non-stationarity and partial observability. It is an interesting job but it is not clear for the key of end condition in non-stationarity and partial observability. As is known the reinforcement learning is a class of iterative learning of episodes based on rewards and terminated at a training convergence. Authors declare “This means that original convergence properties for single-agent algorithms no longer  hold due to the fact that the best policy for an agent changes as other agents’ policies change” (line 166).  What is a performance guarantee mechanism instead of convergence during the non-stationarity iterative process? And how to provide? I notice the experiment takes use of times steps as end condition. Obviously it is not universal.


Instruction: You are one of the reviewers, your task is to write a review for the article. You will be given the title of the article, the number of the round in which the article is located, and your order among the reviewers.

Input:
Title: QUANTIFYING THE IMPACT OF NON-STATIONARITY IN REINFORCEMENT LEARNING-BASED TRAFFIC SIGNAL CONTROL
Review round: 2
Reviewer: 1

Output:
Basic reporting: no comment
Experimental design: no comment
Validity of the findings: no comment
Additional comments: I am sorry for my previous unclear statement. The RL is used to find an optimal action that matches the current states and the stationarity or non-stationarity environment in a long run. The RL is trained by episodes to pursue rewards (actions value), which is classified as an online mode and an offline mode. For online the reward of each episode can be obtained from the Q table (descrete system) or the critic network (continuous system) based on observations after taking an action. But no one knows the effect of this action in training process, which will lead to a fatal fault (traffic confusion). A feasible way is offline training. But it needs a model of actions on the environment to get the action value in all kinds of traffic cases. This model (missing in this paper) is indispensable for adjusting the Q table or the weights of critic network. Both training processes will suffer from an early fluctuation to the end stability (this is called to be convergent). For a stationarity, it will spend a very long time to get the optimal action. For a non-stationarity, maybe it will keep changing and cannot get optimal action. So I ask for how to provide a performance guarantee mechanism. By the way the agent will get the optimization (be convergent) if given sufficient time seems not to meet the reality.


Instruction: You are one of the reviewers, your task is to write a review for the article. You will be given the title of the article, the number of the round in which the article is located, and your order among the reviewers.

Input:
Title: QUANTIFYING THE IMPACT OF NON-STATIONARITY IN REINFORCEMENT LEARNING-BASED TRAFFIC SIGNAL CONTROL
Review round: 3
Reviewer: 1

Output:
Basic reporting: no comment
Experimental design: no comment
Validity of the findings: no comment
Additional comments: The concerns are responded. Thanks.


Instruction: You are one of the reviewers, your task is to write a review for the article. You will be given the title of the article, the number of the round in which the article is located, and your order among the reviewers.

Input:
Title: IMPROVING CLASSIFICATION BASED ON PHYSICAL SURFACE TENSION-NEURAL NET FOR THE PREDICTION OF PSYCHOSOCIAL-RISK LEVEL IN PUBLIC SCHOOL TEACHERS
Review round: 1
Reviewer: 1

Output:
Basic reporting: the introduction is excessively long, scatter, and irregular. The authors should avoid additional descriptions and studies.
Experimental design: The paper does not mention review of this research protocol by an ethics review board before the research was conducted. Obtaining consent is not the same as having an ethics review board vet the study before it is done.
Validity of the findings: no idea.
Additional comments: In this article, the authors have used "the physical surface tension-neural net", which is a nice methodological choice. As well as this is an interesting topic that present novel data. However, I can mark the present work as a major revision as there are a lot of scattering and irregularity during the work. Therefore, the authors are requested to answer and deal with the comments one-by-one and very closely.

Abstract:
1. I think that the abstract is very long.
2. I think that in the abstract authors should avoid using acronyms.
3. the conclusion of the abstract is vague and unrelated and it should be rewritten.

Introduction:
1. the introduction is excessively long, scatter, and irregular. The authors should avoid additional descriptions and studies.
2. It should be noted that from time to time there are some English grammatical errors in the whole manuscript. The manuscript needs to be revised by a native English editor. Of course, although the text benefits from suitable and sophisticated information presented in the sentences, some sentences have not been written correctly, which make them somehow understandable for possible readers and researchers.
3. In introduction the authors made some statements without the use of appropriate references. For example, " Psychosocial risks are stress factors that can alter and unbalance the resources and abilities of the person to manage and respond to the flow of the activity derived from work, negatively affecting physical and psychological health".  " Liquid surface tension is defined as the amount of energy necessary to increase surface area of a liquid per unit of area. Surface tension (a manifestation of liquid intermolecular forces) is the force that tangentially acts, per unit of longitude, on the border of a free surface of a liquid in equilibrium, and which tends to contract said surface.", ….
4.You have too many abbreviations in the study; doesn't this make the article difficult to understand by readers and researchers?
5. It's better for authors to express the importance of artificial neural networks compared to classical regression methods:
- Farhadian M, Aliabadi M, Darvishi E. Empirical estimation of the grades of hearing impairment among industrial workers based on new artificial neural networks and classical regression methods. Indian J Occup Environ Med. 2015 May-Aug; 19(2): 84–89. doi: 10.4103/0019-5278.165337.

Material and Methods
1. The method part is not satisfactory. Subsections must be modified as below:
- Description of the samples (teachers)
- Description of model variables.
- Database and Data Pre-processing
- Description of Model construction process:
¬ - Step 1: …
 -  Step 2 …
 -  Step 3 …
 -  Step 4
- Comparison techniques

2. The variables considered has been collected based on what standard or reference? For example, the musculoskeletal symptoms have been collected using what tools? Moreover, there is no reference to the tools and methods of measuring physiological parameters.

3. The authors should add references that support the defined methodology.
4. The paper does not mention review of this research protocol by an ethics review board before the research was conducted. Obtaining consent is not the same as having an ethics review board vet the study before it is done.
5. There could be a more organized flow throughout the manuscript and greater organization of the methods.

Results:
1.	The number of tables is very large. Authors can merge a number of tables. It will be very boring for the reader.
2.	2. The authors can add one table and submit the predicted risk level by the PST-NN model in the 1-3 cases of new samples.

Discussion:
1. Overall, discussion should be improved. The authors should avoid to describe the results again. They should only explain, with appropriate support, why the results were obtained. Additional explanations are missing.
2. What do the authors think with ANN and other models? I invite the authors to read, among others, the following article:
Aliabadi M, Farhadian M, Darvishi E. Prediction of hearing loss among the noise-exposed workers in a steel factory using artificial intelligence approach. International Archives of Occupational and Environmental Health volume 88, pages779–787(2015).

Conclusion:
I advise that the conclusion was rewritten based on followed:
“The proposed hybrid method in this investigation, PST-NN, obtained better results than other models for the prediction of the psychosocial risk level in Colombian public-school teachers. The percentage of classification was 97.37% for the PST-NN approach. Generally, the input parameters were selected in accordance with researchers' experience, but in this model, the Physical Surface Tension method performed better in the resolution of parameter optimization, which transformed and reduced from 20 to 2 physical variables, and reduced ANN deficiencies. The results obtained in the prediction demonstrate that the proposed prediction model is applicable for the identification of the psychosocial risk level in Colombian public-school teachers with high levels of accuracy and contribute to the guidelines with the public health plans defined by the government.”


Instruction: You are one of the reviewers, your task is to write a review for the article. You will be given the title of the article, the number of the round in which the article is located, and your order among the reviewers.

Input:
Title: IMPROVING CLASSIFICATION BASED ON PHYSICAL SURFACE TENSION-NEURAL NET FOR THE PREDICTION OF PSYCHOSOCIAL-RISK LEVEL IN PUBLIC SCHOOL TEACHERS
Review round: 1
Reviewer: 2

Output:
Basic reporting: The use of English is unclear and ambiguous. There are some grammatical errors and typos such as:
- ...  is to develop a model that support machine learning model to ...
- (Collie, Shapka & Perry, 2012) in a previous study shown how teachers’ perception of their work  environment influenced in the levels of: ...
- Prediction of psychosocial risk levels in public school teachers have to fundamental importance, owing to its impact on teacher health.
- According this, the goal were develop a model be able to mathematically relate the psychosocial factors ...
- The review of literature show that models with different applications have been proposed for classification purposes, however, don´t anything ...
- Therefore, it can be argued that such a development is a novel approach that adds to the scientific of literature in this field.
- ...
The authors should re-check and revise carefully. It is necessary to be proofread by a native speaker or editing service.

Literature review is long and some of them are not related to the study. The authors should optimize and re-organize the literature review.
Experimental design: What is "sensibility level"?

The authors should re-organize the methodology section, now it is not easy to follow. Also, there are some missing sections, i.e., there is a section "2.2.1" but not "2.2.2"? Now it is difficult to replicate the methods with this presentation.

Measurement metrics (such as sensitivity, specificity, accuracy) have been used in previously published works such as PMID: 28643394, PMID: 32942564, and PMID: 32613242. Therefore, the authors should refer more works in this description to attract broader readership.

"Classification error" is an useless metric, since we can easily get it by (100 - accuracy).

Why did the authors run the tests 6 times with different ratios of train/val/test?

It is important to have ROC curve and AUC analysis
Validity of the findings: The authors proposed a lot of things in methodology section, but very few information in results section. It did not show clearly the findings and impacts of the results.

Also, the authors should add more discussions.

In Table 6, why did the authors only report the training performance? Where is validation and testing performance?

According to Fig. 6, the model achieved the best performance at about 100 iterations, why did the authors not stop their training process at this cut-off level? It the authors continue to train, the model will lie to overfitting problem.

How did the authors deal with overfitting problem, since the performance between training and validation/testing are different?
Additional comments: No comment


Instruction: You are one of the reviewers, your task is to write a review for the article. You will be given the title of the article, the number of the round in which the article is located, and your order among the reviewers.

Input:
Title: IMPROVING CLASSIFICATION BASED ON PHYSICAL SURFACE TENSION-NEURAL NET FOR THE PREDICTION OF PSYCHOSOCIAL-RISK LEVEL IN PUBLIC SCHOOL TEACHERS
Review round: 2
Reviewer: 1

Output:
Basic reporting: nothing to add
Experimental design: nothing to add
Validity of the findings: nothing to add
Additional comments: In my opinion, many of the comments have remained unchanged.
particularly in method and material, the authors should be  modified the subsections the order as below:
- Description of the samples (teachers)
- Description of model variables.
- Database and Data Pre-processing
- Description of Model construction process:
¬ - Step 1: …
- Step 2 …
- Step 3 …
- Step 4
- Comparison techniques


Instruction: You are one of the reviewers, your task is to write a review for the article. You will be given the title of the article, the number of the round in which the article is located, and your order among the reviewers.

Input:
Title: IMPROVING CLASSIFICATION BASED ON PHYSICAL SURFACE TENSION-NEURAL NET FOR THE PREDICTION OF PSYCHOSOCIAL-RISK LEVEL IN PUBLIC SCHOOL TEACHERS
Review round: 2
Reviewer: 2

Output:
Basic reporting: No comment
Experimental design: No comment
Validity of the findings: No comment
Additional comments: Thanks for addressing my previous comments. Although some efforts have been done, the authors have not addressed well in most of my comments. Therefore, I do not suggest to accept this manuscript for publication. Some comments are still not answered well as follows:

1. Literature review is long and some of them are not related to the study. The authors should optimize and re-organize the literature review.

2. Why did the authors not have section 2, but they have sub-sections 2.1, 2.2., ...?

3. Measurement metrics (such as sensitivity, specificity, accuracy) have been used in previously published works such as PMID: 28643394, PMID: 32942564, and PMID: 32613242. Therefore, the authors should refer more works in this description to attract broader readership.

4. Why did the authors run the tests 6 times with different ratios of train/val/test? ==> Table 7 did not show any reason that the authors used different ratios of different test times.

5. The authors proposed a lot of things in methodology section, but very few information in results section. It did not show clearly the findings and impacts of the results. ==> I even don't understand the answer from the authors in this question. (We believe that the evaluator confuses the methods of previous studies with this one and the results.
The results of the proposed method shown are those sufficient and necessary to evauate a new classification method, regarding both stability and overfitting. However, as the evaluator suggested, the ROC metric was added to guarantee the adequate per-formance of the algorithm.
)

6. In Fig. 6, the best performance might come at epoch of 100, why did the authors need to run to 1000 epochs?

7. The authors mentioned "To prevent overfitting, the preprocessed dataset was split in 5-fold cross validation", but I think it is not a solution to prevent overfitting. This is a normal evaluation method for all machine learning problems. I'd like to ask whether the authors applied some other techniques to avoid overfitting in their models. According to the results, I think the models contained a lot of overfitting.


Instruction: You are one of the reviewers, your task is to write a review for the article. You will be given the title of the article, the number of the round in which the article is located, and your order among the reviewers.

Input:
Title: IMPROVING CLASSIFICATION BASED ON PHYSICAL SURFACE TENSION-NEURAL NET FOR THE PREDICTION OF PSYCHOSOCIAL-RISK LEVEL IN PUBLIC SCHOOL TEACHERS
Review round: 3
Reviewer: 1

Output:
Basic reporting: There is nothing special in this area.
Experimental design: There is nothing special in this area.
Validity of the findings: There is nothing special in this area.
Additional comments: There is nothing special in this area.


Instruction: You are one of the reviewers, your task is to write a review for the article. You will be given the title of the article, the number of the round in which the article is located, and your order among the reviewers.

Input:
Title: IMPROVING CLASSIFICATION BASED ON PHYSICAL SURFACE TENSION-NEURAL NET FOR THE PREDICTION OF PSYCHOSOCIAL-RISK LEVEL IN PUBLIC SCHOOL TEACHERS
Review round: 3
Reviewer: 2

Output:
Basic reporting: No comment.
Experimental design: No comment.
Validity of the findings: No comment.
Additional comments: No comment.


Instruction: You are one of the reviewers, your task is to write a review for the article. You will be given the title of the article, the number of the round in which the article is located, and your order among the reviewers.

Input:
Title: DXPR: AN R PACKAGE FOR GENERATING ANALYSIS-READY DATA FROM ELECTRONIC HEALTH RECORDS—DIAGNOSES AND PROCEDURES
Review round: 1
Reviewer: 1

Output:
Basic reporting: See General Comments
Experimental design: See General Comments
Validity of the findings: See General Comments
Additional comments: This well-written though improperly formatted and presented article proposes a common data model (CDM) R algorithm, dxpr, that would address analysis “bottlenecks” inherent in clinical data sets due to “heterogeneity and sparseness” through pre-processing to output analysis-ready data sets. I confess that, for the most part, this is not a challenge that I have to deal with as I am fortunate to work for an agency (DoD) that has consistently formatted medical data. However, I have often dealt with those and other issues when synthesizing data from diverse sources.

I do not work with R, but exclusively with SAS, so I make no pretense of critiquing the coding per se. The application, as described, appears to be flexible, i.e. it provides the option for CDM categorizations (CCS, Elixhauser, Charlson comorbidities, etc.) as well as user-defined.  The standardization, in terms of formatting of ICD dx and procedure codes is a welcome essential feature.

I would not use this product--partly because I don’t work with R although I see no reason why my raw data couldn’t be pre-processed in dxpr and then analyzed using SAS. I also would not make use of the point-and-click menu analysis features in SAS Enterprise Guide (I generally despise SAS EG). The fact is having an intimate understanding of the raw data is essential for executing any competent statistical analysis. I would not consider delegating that responsibility, or rather, that opportunity. Occasionally, a well-meaning potential customer, who cannot access raw data, will offer to perform their own analysis. “All I need you to do is clean, manipulate, and synthesize the data.” Just that. They’re happy to do the high level work of pressing the regression button. If there is a perceived analysis bottleneck I suspect it is among practitioners who perform analysis as a secondary function.

I am instinctively uneasy with the notion of producing “analysis-ready” data. That said, we employ metrics, imperfect though they all are, to find common ground for comparisons. Knowing that other investigative groups operating under different health care systems (perhaps in different countries) employed the same algorithm to clean and prepare their data would seem to provide a strong starting point for comparison. Nevertheless, analysis bottlenecks are not entirely a bad thing.  Making complex clinical data more accessible sounds like a good idea but lowering the requirements for entry may permit access to naïve analysts.  Ultimately though, the onus is on the investigator(s) to competently execute their analysis. Dxpr seems like a useful product—if used responsibly.  But that is always the caveat.

It’s not clear why this dxpr package description was presented as an analysis—which it isn’t. I don’t care for the arbitrary case-control “analysis” described in the results. (Selecting controls from a sample of sick people is not best practice and at any rate, if dxpr is about creating analysis ready data a contrived example of an analysis is not necessary.) There was no mention of it in the Methods.  If the aim is to describe the dxpr package then please don’t pretend to do something else.  One would expect the editorial staff to address these issues before sending the article out for review.

Recommendation: Major revision/re-formatting

I have no interest in reviewing a second time.


Instruction: You are one of the reviewers, your task is to write a review for the article. You will be given the title of the article, the number of the round in which the article is located, and your order among the reviewers.

Input:
Title: DXPR: AN R PACKAGE FOR GENERATING ANALYSIS-READY DATA FROM ELECTRONIC HEALTH RECORDS—DIAGNOSES AND PROCEDURES
Review round: 1
Reviewer: 2

Output:
Basic reporting: The paper is well-written and easy to read and comprehend
Experimental design: no comment
Validity of the findings: The paper introduces an R packages that seems to do what is supposed to do, but this is relatively hard to asses in detail. That being said, I wold have expected more unit tests (now there seems to be just one very general such test)
Additional comments: I found the paper well written and I think the package is a useful addition to the computational ecosystem.
However, I do have some comments, questions and suggestions:

abstract: "combat" -> "address"?
p9, 100-107: this text seems duplicated from the intro and does not really seem to be about methods...
p10 112-113: please give a better ref to the devtools package
p10 123: so, the date format cannot be changed? why?
p16 305-307: the index date must be pre-computed, as it seems to be a fixed date -- is this correct? I do see why this choice but I was wondering if a different approach might be useful as well (say, x days since the first event) or so?
condition eras: the gapDate is fixed for all patients and conditions, right? why?

The code: I looked at the GitHub repo and I do have some general questions:
1. (touched above): I feel that a good set of unit tests are essential, especially as the package will continue to be developed
2. help: I found that the help could be improved and that, in some cases, not info is given about the expected parameters (e.g., "dxDataFile" is actually *not* a file, but a data.frame-like object but this is not specified)
3. the I/O seems limited to in-memory only (and I fully understand that) but it would be useful to see a discussion (maybe with some code) about how to use the package with, say, and SQL database?


Instruction: You are one of the reviewers, your task is to write a review for the article. You will be given the title of the article, the number of the round in which the article is located, and your order among the reviewers.

Input:
Title: DXPR: AN R PACKAGE FOR GENERATING ANALYSIS-READY DATA FROM ELECTRONIC HEALTH RECORDS—DIAGNOSES AND PROCEDURES
Review round: 2
Reviewer: 1

Output:
Basic reporting: no comment
Experimental design: no comment
Validity of the findings: no comment
Additional comments: Thanks for revising the paper and taking my comments seriously. However, there are a few (minor ones) that still stand and that would improve the paper. I will re-mention them here:

- please give a better ref to the devtools package: what I meant is to give a proper reference to the paper in the academic sense. Running `citation("devtools")`
gives:
  @Manual{,
    title = {devtools: Tools to Make Developing R Packages Easier},
    author = {Hadley Wickham and Jim Hester and Winston Chang},
    year = {2020},
    note = {R package version 2.3.2},
    url = {https://CRAN.R-project.org/package=devtools},
  }
which should be used instead of the current "(Wickham H)"

- so, the date format cannot be changed? why?:  now the documentation seems to specify that "As for date column, the data format should be YYYY/MM/DD or YYYY-MM-DD" -- I still wonder why not letting the user specify the date format (see, for example, the `lubridate` package)

- gapDate: ok, thanks for the clarification; while I agree that it makes sense in most cases to fix it per condition, I am wondering if there might be particular cases where you still want to let vary *also* between patients?

- unit tests: what I meant is the stuff that resides in the `dxpr/tests/testthat` folder, which now is just a skeleton `test-icdToCCS.R`; it is always a good idea to have a battery of unit tests to make sure that evolutions/bug fixes do not break things (e.g., https://towardsdatascience.com/unit-testing-in-r-68ab9cc8d211)

Otherwise, thanks for updating the documentation and the vignettes, including with accessing SQL data.


Instruction: You are one of the reviewers, your task is to write a review for the article. You will be given the title of the article, the number of the round in which the article is located, and your order among the reviewers.

Input:
Title: DXPR: AN R PACKAGE FOR GENERATING ANALYSIS-READY DATA FROM ELECTRONIC HEALTH RECORDS—DIAGNOSES AND PROCEDURES
Review round: 3
Reviewer: 1

Output:
Basic reporting: no comment
Experimental design: no comment
Validity of the findings: no comment
Additional comments: Thanks for addressing all my previous comments!


Instruction: You are one of the reviewers, your task is to write a review for the article. You will be given the title of the article, the number of the round in which the article is located, and your order among the reviewers.

Input:
Title: ECOVNET: A HIGHLY EFFECTIVE ENSEMBLE BASED DEEP LEARNING MODEL FOR DETECTING COVID-19
Review round: 1
Reviewer: 1

Output:
Basic reporting: It's well written and easy to understand.
Experimental design: The method is clear and straightforward.
Validity of the findings: I have a major concern regarding the results part. The authors created a (small) balanced subset for testing and claimed better performance on the balanced testing set. I am not convinced by the logic behind this. Balanced testing set is unnatural and has little practical value. We all know that machine learning loves balanced dataset. But reporting a good number using an unnatural dataset is a little bit misleading. It also makes it hard to judge the contribution of this work because the numbers reported by other methods do not use this strategy. I would suggest the authors remove everything related to the "balanced dataset" and only report their results on the original dataset.
Additional comments: 1. Some related works are not mentioned, for example: https://doi.org/10.1038/s41467-020-18685-1
2. It's a little misleading to use an unnatural balanced testing dataset. It also makes it hard to compare with other methods unless they are using the same strategy. So I would suggest the authors to remove everything related to the balanced dataset and report results on the original dataset.


Instruction: You are one of the reviewers, your task is to write a review for the article. You will be given the title of the article, the number of the round in which the article is located, and your order among the reviewers.

Input:
Title: ECOVNET: A HIGHLY EFFECTIVE ENSEMBLE BASED DEEP LEARNING MODEL FOR DETECTING COVID-19
Review round: 1
Reviewer: 2

Output:
Basic reporting: The paper is well written and organized.

The literature references about work related to Covid are sufficient, but there are no references for the vast amount of deep learning work in other closely related areas. There are also many references about well known concepts like ensemble methods.

The article is well structured, and the data is publicly available.

The paper contains excessively lengthly explanations of well known concepts like ensemble learning and class activation maps.

The results sections is satisfactory.
Experimental design: The research cannot be said to be original. There is no identified knowledge gap being filled. Applying ensemble methods is not sufficient to meet this requirement.

The method applies a stock EfficientNet with the final layers changed from 1000 classes (ImageNet) to 3 classes. Two fully connected layers are added, but there is no comparison included to show that these contribute to performance.

The size of the validation and test sets is too small with 90% of the data used for training. There are only a very small number of positive Covid cases in the validation and test sets.

Methods are well described and replicable.
Validity of the findings: The data is provided and the results could be replicated.

The work does not answer any research question, as it applies stock methods with little to no modification to a very small dataset. The paper claims a very minor modification to a stock EfficientNet as a novel CNN architecture. As such, it is not clear what conclusions can be made from the presented research.
Additional comments: No additional comments


Instruction: You are one of the reviewers, your task is to write a review for the article. You will be given the title of the article, the number of the round in which the article is located, and your order among the reviewers.

Input:
Title: ECOVNET: A HIGHLY EFFECTIVE ENSEMBLE BASED DEEP LEARNING MODEL FOR DETECTING COVID-19
Review round: 2
Reviewer: 1

Output:
Basic reporting: no comment
Experimental design: no comment
Validity of the findings: no comment
Additional comments: I would say that the revised version is much improved. I still have two major concerns regarding table 8 which should be addressed before publishing.

1. Table 8 compared the proposed method with other existing methods so its accuracy is very important. I haven't checked all numbers but at least for DeepCOVIDExplainer the number in Table 8 does not match the one I found in either https://ieeexplore.ieee.org/document/9313304 or https://arxiv.org/pdf/2004.04582.pdf. Perhaps the authors of DeepCOVIDExplainer have updated their numbers, but it is still better for the current work to cite numbers from the peer-reviewed version (the IEEE version) or at least the latest pre-print version instead of the old pre-print version. Please check the numbers of other methods too to make sure all numbers are current and accurate.

2. It is not explained in the text why COVID-Net, EfficientNet-B3, DeepCOVIDExplainer, and the proposed method all use the COVIDx dataset but have different dataset size in Table 8. It would be much better if the authors could compare the proposed method with other methods on the same dataset, at least for methods that use COVIDx, if possible.


Instruction: You are one of the reviewers, your task is to write a review for the article. You will be given the title of the article, the number of the round in which the article is located, and your order among the reviewers.

Input:
Title: ECOVNET: A HIGHLY EFFECTIVE ENSEMBLE BASED DEEP LEARNING MODEL FOR DETECTING COVID-19
Review round: 3
Reviewer: 1

Output:
Basic reporting: no comment
Experimental design: no comment
Validity of the findings: no comment
Additional comments: All my concerns have been successfully addressed by the authors.


Instruction: You are one of the reviewers, your task is to write a review for the article. You will be given the title of the article, the number of the round in which the article is located, and your order among the reviewers.

Input:
Title: OVERVIEW OF CURRENT STATE OF RESEARCH ON THE APPLICATION OF ARTIFICIAL INTELLIGENCE TECHNIQUES FOR COVID-19
Review round: 1
Reviewer: 1

Output:
Basic reporting: The article presents an overview of recent papers which use artificial intelligence methods for prediction and diagnosis of COVID-19. Since there has been many such papers published in recent months, an overview is timely and reflects the need to summarize the existing research. The presented overview is broad and comprehensive, citing 142 literature sources. Such overview papers are highly needed by the researchers trying to orient themselves in the deluge of results reported on COVID-19. Clear English is used throughout. Professional article structure, figures and tables are provided.
Experimental design: The description of the methodology of survey should be improved to allow for replication. Specifically, indicate the query string. Were any papers found excluded from further analysis and why? Why you analyze only Preprints rather than complete publications registered on scientific bibliography databases such as Web of Science or Scopus?
Validity of the findings: The findings are valid, however, they must bus discussed in more-detail as suggested in my Comments for the author.
Additional comments: I suggest the following minor comments to be addressed before the paper could be accepted for publication:
1.	Present a more extensive bibliographic analysis of literature sources on COVID-19, including the most popular venues of publications and leading research teams around the world. You can use, for exampling, the guidelines presented in Conducting systematic literature reviews and bibliometric analyses, Australian Journal of Management 45(2):175-194, DOI: 10.1177/0312896219877678 on how to visualize the results of literature reviews.
2.	I suggest to present a summary for each type of system or application discussed (such as including information on the number of instances and attributes for biomedical datasets; or what are the most popular deep learning architectures used). This would provide a deeper insight on the analyzed research domain.
3.	Add a critical discussion section and discuss the current limitations of such systems as well as the successes.
4.	Present a more extensive and in-depth conclusions of where the COVID-19 related research is heading.


Instruction: You are one of the reviewers, your task is to write a review for the article. You will be given the title of the article, the number of the round in which the article is located, and your order among the reviewers.

Input:
Title: OVERVIEW OF CURRENT STATE OF RESEARCH ON THE APPLICATION OF ARTIFICIAL INTELLIGENCE TECHNIQUES FOR COVID-19
Review round: 1
Reviewer: 2

Output:
Basic reporting: This paper surveys contributions of AI to COVID-19. It groups the surveyed articles into five categories: AI systems that predict the COVID-19 disease, diagnose the disease, recommend treatment, design new drugs, and predict social and economic impact. Therefore, the title of the paper is somewhat misleading as it spans more than prediction and diagnosis. It is therefore recommended to change the title.
The survey is definitely an interesting idea and could be of interest to many readers.
In the data gathering part, the grouping of different data gathering and prediction systems is interesting. It is recommended to refer to “surveillance” systems and to epidemiology for those forecasting the spread of the disease.
Regarding the form, there are many spelling and grammatical errors in the paper. It is recommended to have the paper proofread by a native speaker.
Experimental design: The methodology of querying only pre-print papers may not be sufficient to cope with the extent of the literature in this domain, so that it is recommended to broaden the base of papers studied. However, the information collected from the body of literature is already interesting so that the extension to a broader literature could wait until a next paper. This should be discussed in future directions.
Validity of the findings: One general impression reading about the different AI systems presented is that we do not have much data regarding their effectiveness. They definitely aim at addressing diverse challenges related to COVID-19, but do not have measurements showing that they do so. This is expected due to the novelty of the disease, but it should be better explained in the paper. You are proposing ideas for addressing the challenge, but these ideas need to be tested to be shown effective.
It is recommended to better highlight the systems that being used or are going to be used in medicine or in the field, by contrast to the systems only used by computer systems on existing datasets. The reader would like to know, among these systems, which ones actually advance the diagnosis, treatment, prediction etc. of COVID-19 on real persons, not only on existing datasets.
Additional comments: The paper is interesting to read and covers a broad range of AI applications in the fight against COVID-19. The language should be improved before publication, and the recommendations listed above should be considered.


Instruction: You are one of the reviewers, your task is to write a review for the article. You will be given the title of the article, the number of the round in which the article is located, and your order among the reviewers.

Input:
Title: OVERVIEW OF CURRENT STATE OF RESEARCH ON THE APPLICATION OF ARTIFICIAL INTELLIGENCE TECHNIQUES FOR COVID-19
Review round: 1
Reviewer: 3

Output:
Basic reporting: •	The review is riddled with typographical and grammatical errors that are too numerous to detail here. Furthermore, the writing is not professional enough to meet standards for an international audience. While I am sympathetic towards the barriers that non-native English speakers have to overcome, these language issues actually hampered my ability to follow the paper. I strongly advise having a native English speaker proofread the review.
•	Introduction and background are adequate but suffer from logical gaps that weaken the section. For instance, in line 58, outcome risk is not clearly defined. Instead a confusing sentence is presented, “If the treatment is not effective towards infected person/group then there is less possibility to recover or die.” I am assuming that there is a lower possibility of recovering rather than dying.
•	The structure of the paper largely conforms to PeerJ standards and is as expected for an interdisciplinary review.
•	The review is timely, of broad and cross-disciplinary interest and within the scope of the journal.
•	However, the field has been extensively reviewed (a quick PubMed search revealed several such papers: https://pubmed.ncbi.nlm.nih.gov/?term=artificial%20intelligence%20covid-19%20review). It is unclear what distinguishes this review from existing ones. This would be a good point to emphasize in the abstract and the introduction.
•	The Introduction does not explicitly make it clear which audience the review is addressing. However, the implication is that this is a review meant to initiate AI researchers towards COVID-19 work.
Experimental design: •	The content is within the aims and scope of the journal.
•	While I cannot comment on the ethical standard of the work done for this review, I think on the technical side, the review could be carried our more systematically (see below).
•	There is a great lack of detail in the description of the methods and I highlight two major points here. First, it is unclear what search terms and their combinations were used to query the preprint servers. Were any synonyms such as “machine learning” used? Second, what were the inclusion/exclusion criteria for the manual selection of papers?
•	By using preprint servers as a primary resource to search for papers, I am concerned that the Survey Methodology is biased towards work that is not peer-reviewed or is undergoing peer review. This suggests a wide range of maturity of these articles and it is not clear whether this is accounted for when choosing and highlighting papers. The survey largely ignores the body of work published in peer-reviewed journals. At the very least, the survey would be more comprehensive if open access journal articles were included in this review. Alternatively, providing clear statistics on the fates of these articles and the process to eliminate works-in-progress can be considered.
•	Sources are adequately cited.
•	While the review is coherently organized into sections and sub-sections, the content in each of these is inconsistent with the title. For example, Section 7 is titled “Treatment of COVID-19” but its sub-sections largely focus on radiology images, which has more to do with diagnosis. On a related note, there is a much larger issue pervasive throughout the paper that I discuss below in my general comments.
Validity of the findings: •	Since this is a review, there are no real conclusions to draw or goals to be met but the paper does a modest job in providing a broad overview of this topic.
•	The review does a reasonable job in outlining future directions. However, each sub-section is sparse on details and does not come across as a well thought out roadmap for the field. I am also not convinced by the use of drones to combat COVID-19 as issues such as trust and safety will need to be addressed. A better approach would be to adopt a more speculative tone and condense this section to 2-3 broad sections that talk about ongoing work. This will help demonstrate feasibility.
Additional comments: •	The most important concern that I have is that there is a recurring abuse of terminology when it comes to the COVID-19 aspects of the paper. For instance, prediction is a broader term that spans can be made at the patient level (the progression of the disease) or the population level (the spread of the disease). On the other hand, diagnosis specifically focuses on the patient and involves either assigning a COVID-positive or COVID-negative status or more deeper classification (e.g., severity). In this paper, these terms are used interchangeably and are confusing when trying to follow the organization of the review. Similarly, in line 175, it is debatable whether apps that collect responses to questionnaires are actually “predicting” COVID-19 status.
•	The lack of mastery of terminology, particularly in molecular biology, comes through and adversely affects the quality of the review. For instance, in line 243, the RCSB Protein Data Bank contains protein structures (and not “gene structures” as mentioned here). Another example, is the following phrase in line 610: “cell-lines of existing medicines”. What does this mean?
•	Are Tables 1 and 2 directly extracted from existing papers or were they generated for this review? If it is the latter, what data were used to populate these tables and how were these constructed?
•	The paper is very inconsistent in when it chooses to highlight the performance of tools and when it merely reports the existence of a particular tool. What was the rationale for diving deeper for some papers?


Instruction: You are one of the reviewers, your task is to write a review for the article. You will be given the title of the article, the number of the round in which the article is located, and your order among the reviewers.

Input:
Title: OVERVIEW OF CURRENT STATE OF RESEARCH ON THE APPLICATION OF ARTIFICIAL INTELLIGENCE TECHNIQUES FOR COVID-19
Review round: 2
Reviewer: 1

Output:
Basic reporting: It meets the standards of a scientific journal.
Experimental design: It falls within the scope of the journal.
Validity of the findings: The methodology was chosen correctly, the results are valid.
Additional comments: All my comments have been taken into account, I recommend acceptance of this work.


Instruction: You are one of the reviewers, your task is to write a review for the article. You will be given the title of the article, the number of the round in which the article is located, and your order among the reviewers.

Input:
Title: LEARNER QUESTION’S CORRECTNESS ASSESSMENT AND A GUIDED CORRECTION METHOD: ENHANCING THE USER EXPERIENCE IN AN INTERACTIVE ONLINE LEARNING SYSTEM
Review round: 1
Reviewer: 1

Output:
Basic reporting: The first thing I noticed in the paper is its main objective, which is not specific and so
general to study. As well, the researcher does not determine the scope of the study. For
example: In this paper, propose a novel method to assess the correctness of the user query, in terms of syntax and semantics. The question is: “Where is this paper conducting?”.

Secondly, the researcher did not mention any of the references at the end of the paragraph in the first chapter (introduction), and it is so important to mention the name followed with the year of the reference, which these information are taken from it. For example: Online learning systems (OLSs) have brought great advantages to all kind of formal and informal
learning modes. Over the years, OLSs have evolved from simple static information
delivery systems to interactive, intelligent, and context-aware learning systems, virtually
incorporating real-life teaching and learning experience. In today's OLSs, much of the
emphasis is given on designing and delivering learner-centric learning, in terms of the
learning style, learning approaches, and progress of a particular learner.
Experimental design: 1-A shortage of information and data regarding the participated sample conducted in this
study including its sample number or any specific information related to them. I did not
find that in the methodology section. And in the abstract section, the researcher only briefly mentions it. For example: A trigram language model is built and trained for assessing the correctness of learners' queries on Java.
2-The researcher does not explain the web model (client or server) well, as this model is used in the experimental procedure in this paper. For example, the proposed framework for the correct question is implemented as a web model (client/server) for experimentation and performance analysis. And this paper also does not explain the reasons behind choosing this model for this paper.
3-Moreover, There are concepts in this paper that do not have a specific definition or explanation that makes this paper hard to understand and appears too complex to read. Such as the Softcosine technique. This technique does not have enough information about how it works, and how to benefit from it to achieve the paper objective.
Validity of the findings: 1-A lot of significant information mentioned in this paper but without a clear and specific
sequence, which makes the paper hard to understand and the information are not found
effectively. And that means there is no benefits from most of the results of this paper.
2-The research questions, objectives do not clearly formulate in this paper.
3-No discussion section for the results found and comparing to what was discovered in the
previous studies regarding the paper issue.
Additional comments: The main aim of the paper is strong and recent and has a strong significance in this field. It will add value to the educational system. The experimental analysis used is fully explained and effectively evaluated the results of the study. In addition, having a lot of tables and charts that explained the results in a clearer and detailed way.
But, you have to fix the paper according to the following comments:
1-The whole paper should be rearranged with specific main titles and sub-titles. And each title must completely explain all the significant information that may benefit in this section.
2- The research question, objective, and sub-objectives and (hypothesis of the research paper if needed) should formulate.
3- More in-depth information about the introduction and background of the research issue in the first chapter should be mentioned, to give the reader more information and data about the research issue.
4- The sample method and model used in this research paper should discuss and explain in what way can be beneficial in achieving the research paper objectives.
5- It should mention all the models used in this paper in the methodology section and the framework of this research paper should be explained in more detail by determining the dependent and in-depended variables for the research paper.
6- The researcher must discuss the results with the previous outcomes found in past related studies.
7- It is so important to determine the scope of the study to make the research more beneficial.
8- All references must be cited in the paper, if there is a piece of information it must be cited for easy checkout.
9- It must formulate some recommendations before the conclusion section, to contribute in improving the educational system using the results found in this paper.

Long sentences:
•	Page 6 line 52: “Over the years, OLSs have evolved from simple static information delivery systems to interactive, intelligent, and context-aware learning systems, virtually incorporating real-life teaching and learning experience.”

•	Page 6 line 61: “To achieve advanced learning skills like analyzing, evaluating, creating, and applying, a higher level of interactions like discussion, hands-on experiments, exchanging views with experts, etc. are required.”

•	Page 16 line 399: “Further analysis of these false-negative questions reveals that most of them after preprocessing and stopword removal the question length is reduced to less than three.”

•	Page 16 line 417: “Similarly decreasing N leads to word sequence pattern search at lower order, and this restricts the probability of correctness of the word sequences at higher orders. This typically lowers accuracy.”

•	Page 18 line 459: “Failing to determine which parts of the question and thus their semantics are aligned to learner's intent lead to ambiguity in identifying the parts of a sentence to be taken correctly for similarity matching.”

Better words:
•	Page 6 line 57: “one key aspect of an OLS is interactivity”
o	One key aspect of an OLS is interaction.

•	Page 6 line 58: “But, despite the advantages, due to high-cost factor and complexity, contents developed for OLSs have limited or no interactivity.”
o	Limited or no interaction.

•	Page 6 line 66: “The best option is to opt for a question-answer based OLS”
o	The best option is to choose a question-answer based OLS.

•	Page 7 line 73: Fundamentally, these systems process the input query to parse its structure and semantics to understand the intention of the query.
o	Fundamentally, these systems process the input query to determine its structure and semantics to understand the intention of the query.

•	Page 7 line 82: “For instance, more often than not, the non-native English-speaking people having poor knowledge of English find it difficult to compose queries in English.”
o	Most often.

•	Page 9 line172: Error checking in the text is a problem since long back.
o	Error testing the text has been a problem for a long time ago.

•	Page 14 line 345: “The correctness of a question is estimated based on its syntactical and semantic aspects and accordingly is classified as correct or wrong.”
o	As right or wrong.

World abbreviation:
•	Page 9 line 158: what’s POS refer to?

Citation:
•	It need more citation, there is no citation on most of the paper.

•	Page 9 line 196: “Pattern recognition is one of the successful ways of detecting errors in the textual sentence. In [3], a learning model is generated from labelled sequential patterns of both correct and wrong sentences.”
o	better citation.

•	Page 10 line 205: “To detect the word usage error bidirectional long short-term memory (LSTM) model is proposed in [5].”
o	better citation.

•	Page 17 line 448: “Cosine and Jaccard similarity techniques are the two text-based similarity approach which has been widely incorporated for finding similar text.”
o	Need citation.


Instruction: You are one of the reviewers, your task is to write a review for the article. You will be given the title of the article, the number of the round in which the article is located, and your order among the reviewers.

Input:
Title: LEARNER QUESTION’S CORRECTNESS ASSESSMENT AND A GUIDED CORRECTION METHOD: ENHANCING THE USER EXPERIENCE IN AN INTERACTIVE ONLINE LEARNING SYSTEM
Review round: 1
Reviewer: 2

Output:
Basic reporting: Improve English:

1) Who is "her" in this paragraph?
"The proposed model has exhibited 92% accuracy while assessed on the test data. Furthermore, in case the query is not correct, we also propose an approach to guide the user leading to a correct question complying her intent."

2) Not recommended to use "more often than not" in a scientific paper.

3) In the context of this paper: "interactivity" is not clear and fuzzy. Provide a coherent definition.  "Like every learning process, one key aspect of an OLS is interactivity,"

4) "Insufficient domain knowledge also leads to frame a wrong question" INSTEAD OF  "Insufficient domain knowledge also leads to wrong framing of a question."

5) The title of this section should be rephrased "1.2 Plausible Way Outs and their Limitations". It is not suitable for a scientific paper.
Experimental design: Add references:

1)  This section "1.2 Plausible Way Outs and their Limitations" is describing some limitations in NLP and Pattern matching. However, no references that support these limitations.

2) "1.3 Proposed Solution" needs to be designed based on facts supported with references.

3) It would be interesting to add references in 2020

4) Is "Table 1" a research result of another research paper? If yes , then add the reference. If no, then explain how this table was found. Example: did the author conduct a content analysis? what is the methodology so that the results in this table can be supported.
Validity of the findings: no comments
Additional comments: no comments


Instruction: You are one of the reviewers, your task is to write a review for the article. You will be given the title of the article, the number of the round in which the article is located, and your order among the reviewers.

Input:
Title: LEARNER QUESTION’S CORRECTNESS ASSESSMENT AND A GUIDED CORRECTION METHOD: ENHANCING THE USER EXPERIENCE IN AN INTERACTIVE ONLINE LEARNING SYSTEM
Review round: 1
Reviewer: 3

Output:
Basic reporting: -in the the abstract there is a confusion of terms like method, approach, or framework.
-In the abstract there is an ambiguity in the word "Java".
-abstract does not include the source of dataset used in the paper.
-the problem statement and the objective is not defined clearly.
- the missing of the phase model training of the proposed model in the proposed method.
- the measure of the correctness of the query is not enough and not clear.
Experimental design: -in line 299 the sentence "we corrected a total number of 2533 questions of core-Java from
300 different websites and books" ; it is not clear how the questions are corrected.
- in line 351, the phrase "backof model"; there is no such model in the litrature!.
-in the section 4.3 line 525 to 533 the justification of the using of soft-cosine similarity is unclear.
-the authors must refer to the following reference "Soft similarity and soft cosine measure: Similarity of features in vector space model"
-section 4.4 : there is no justification of using thresholds 0.5, 0.2, 0.2 for soft-cosine , Jaccard and Cosine respectively.
- table 8, line 658 ; need more justifications and discussions on how the passes changes the values of the diversity and evenness measures. and the meaning of these values.
Validity of the findings: no comment
Additional comments: No additional comments


Instruction: You are one of the reviewers, your task is to write a review for the article. You will be given the title of the article, the number of the round in which the article is located, and your order among the reviewers.

Input:
Title: LEARNER QUESTION’S CORRECTNESS ASSESSMENT AND A GUIDED CORRECTION METHOD: ENHANCING THE USER EXPERIENCE IN AN INTERACTIVE ONLINE LEARNING SYSTEM
Review round: 2
Reviewer: 1

Output:
Basic reporting: No comment
Experimental design: No comment
Validity of the findings: No comment
Additional comments: Paper format:
- Text alignment seems to be off.
- The references citation doesn't appear in order, it seems like the ones missing were just injected in the middle without considering the sequence although it is considered in the rest of the paper.
- In the appendix, table 4 caption says "Table 1. Performance measures of the proposed approach".
- Sub-sections title font like the one in line 122 (1.3.1) seems to be too small and unrecognizable which makes it hard to follow through and identify the beginning of the section.

Content:
- I still believe the paper lacks consideration for future work to improve the proposed framework or other further work recommended for e-learning experiment enhancement.

Language:
- Minor grammatical mistakes like missing articles such as in line 333 (an error instead of error), 343 (should be a/the user instead of user), line 906 ( the same instead of same), 905 (the corpus instead of corpus), and many others.
- Grammatical mistakes like subject/verb disagreement like in line363 ( should be the system gives instead of giving) Also (this puts instead of put.
- Line 966 says "Questions or questions", I believe that is meant to be "questions and answers". Also, line 968 says question or question, not sure if that's intended but it doesn't make sense to me.
- Wrong usage of [pre[osions such as in line 883 (should be along with instead of along to).


Instruction: You are one of the reviewers, your task is to write a review for the article. You will be given the title of the article, the number of the round in which the article is located, and your order among the reviewers.

Input:
Title: ENSEMV3X: A NOVEL ENSEMBLED DEEP LEARNING ARCHITECTURE FOR MULTI-LABEL SCENE CLASSIFICATION
Review round: 1
Reviewer: 1

Output:
Basic reporting: This paper is about deep learning classification.
1- It is well written and well structured.
2- The literature review and the references are sufficient and updated.
3- The tables and the figures are understandable.
Experimental design: 1- The novelty of this paper is not clear. The proposed structure is based on ImageNet and VGG. The authors should explain more about their collaborations.
2- The authors should explain more on the proposed structure. Based on Figure 7, the only difference between the proposed structure and ImageNet is a fully-connected NN. The authors should explain more about how Figure 8 is related to Figure 7.
Validity of the findings: 1- The result and the discussion sections should be more clarified based on Figures 14-19.
Additional comments: No additional comments


Instruction: You are one of the reviewers, your task is to write a review for the article. You will be given the title of the article, the number of the round in which the article is located, and your order among the reviewers.

Input:
Title: ENSEMV3X: A NOVEL ENSEMBLED DEEP LEARNING ARCHITECTURE FOR MULTI-LABEL SCENE CLASSIFICATION
Review round: 1
Reviewer: 2

Output:
Basic reporting: The authors are advised to revise the manuscript thoroughly and carefully to avoid any possible technical and grammatical errors.
Experimental design: The experimental part of the article lacks the experimental design, which is a serious drawback of the experiment. Besides, the results cover a very narrow area. Meanwhile,no statistical analysis of the results.
Validity of the findings: no comment
Additional comments: Reviewer comments:
This paper presented a novel Ensembled Deep Learning Architecture for multi-label scene
classification. However, the logical relationship is confused and the core content is not prominent of the manuscript. The other issues of the manuscript are as follows:
1.The content of the abstract is not complete, and the purpose of scientific work is not highlighted.
2.In LITERATURE REVIEW, "Lack of indoor scene datasets makes it difficult to get satisfying results by fine-tuning a pre-trained CNN". Why it is difficult to get satisfying results? The satisfying results are not explained clearly.
3.The authors are suggested to provide a comparative study between the proposed work and earlier published works. It will help for future researchers.
4.What are the major advantages of proposed new model over the other available techniques, such as WideResNets, AlexNet, VGG, Inception, ResNets and others?
5.The experimental part of the article lacks the experimental design, which is a serious drawback of the experiment. Besides, the results cover a very narrow area. Meanwhile,no statistical analysis of the results.
6.The authors are suggested to improve the conclusions.
7.The authors are advised to revise the manuscript thoroughly and carefully to avoid any possible technical and grammatical errors.
8.The format of references is not uniform.
My overall opinion then, I don't believe the paper can be recommended for publication.


Instruction: You are one of the reviewers, your task is to write a review for the article. You will be given the title of the article, the number of the round in which the article is located, and your order among the reviewers.

Input:
Title: ENSEMV3X: A NOVEL ENSEMBLED DEEP LEARNING ARCHITECTURE FOR MULTI-LABEL SCENE CLASSIFICATION
Review round: 2
Reviewer: 1

Output:
Basic reporting: Interesting article and you can see many changes. Besides, the article is generally correctly written with well-chosen bibliography.
Experimental design: In this paper, it aims to show multi-label scene classification using five architectures namely VGG16, VGG19, ResNet50, InceptionV3, and Xception using imagenet weights available in the Keras library. The experiments have shown a performance comparison of the different architectures further in the paper.
Validity of the findings: The proposed a new model, EnsemV3X with reduced number of parameters giving an accuracy of 91% better than the best performing models Inception and Xception.
Additional comments: Interesting article and you can see many changes. All the comments have been addressed clearly, so I have no other comments.


Instruction: You are one of the reviewers, your task is to write a review for the article. You will be given the title of the article, the number of the round in which the article is located, and your order among the reviewers.

Input:
Title: ENSEMV3X: A NOVEL ENSEMBLED DEEP LEARNING ARCHITECTURE FOR MULTI-LABEL SCENE CLASSIFICATION
Review round: 2
Reviewer: 2

Output:
Basic reporting: no comment
Experimental design: no comment
Validity of the findings: no comment
Additional comments: Authors proposed EnsemV3X by a way of dividing the feature extraction with weight files and feature recognition with using five classical architectures. It is an interesting idea. There are some concerns:
1.	Different convolution kernels will lead to different effects of feature extraction. How to choose the appropriate weights as a general feature because the weights files are best for the default models with special kernels, but there is no guarantee that they will be universal?
2.	Authors make an ensemble feature recognition with fully-connection ways based on the outputs of five models. It still needs the weights of connection. How to determine these weights in order to get the best results?
3.	The definition of loss should be shown in order to be clear.
4.	Fig.9 to 11 are not clear. For example it seems the training accuracy has not be stable at 70 epochs. What is the end condition? Do the accuracy and the loss have the same coordinates? What is the relation of them? More explanations are necessary. The similar problems are for figure 13 to 18.


Instruction: You are one of the reviewers, your task is to write a review for the article. You will be given the title of the article, the number of the round in which the article is located, and your order among the reviewers.

Input:
Title: DEEP LEARNING PREDICTION OF MILD COGNITIVE IMPAIRMENT CONVERSION TO ALZHEIMER’S DISEASE AT 3 YEARS AFTER DIAGNOSIS USING LONGITUDINAL AND WHOLE-BRAIN 3D MRI
Review round: 1
Reviewer: 1

Output:
Basic reporting: Authors says "Future studies should investigate MCI to AD conversion at 1, 2 and 5 years". Recently, regarding this topic, deep survival analysis has been developed and applied to MRI data. Authors can add discussion about this.
Experimental design: no comment
Validity of the findings: Authors used 4-Fold cross validation. Does this method need to be repeated more than once to avoid subject selection bias?

Many studies revealed default mode network and hippocampus contributes development of AD and MCI. Why doesn't heatmap in this study show their contributions?  Please add discussion on this if possible.
Additional comments: In abstract, what does "#D" means?

I strongly recommend authors to upload the script on web.


Instruction: You are one of the reviewers, your task is to write a review for the article. You will be given the title of the article, the number of the round in which the article is located, and your order among the reviewers.

Input:
Title: DEEP LEARNING PREDICTION OF MILD COGNITIVE IMPAIRMENT CONVERSION TO ALZHEIMER’S DISEASE AT 3 YEARS AFTER DIAGNOSIS USING LONGITUDINAL AND WHOLE-BRAIN 3D MRI
Review round: 1
Reviewer: 2

Output:
Basic reporting: In the present study, the authors investigate the performance of several deep learning models using structural MRI for the prediction of MCI conversion to AD dementia within 3 years. Furthermore, the authors explored the added value of including a second structural MRI performed 12 months after baseline; the authors thus had to exclude MCI participants who converted to AD dementia within 1 year. This implies that their prediction model was in fact a prediction model over 2 years in MCI subjects that remained stable over 1 year. The paper is well written and easy to follow.
Experimental design: Methods are clearly explained. The authors should have followed the TRIPOD guidelines for predictive models and provide the corresponding checklist (https://www.equator-network.org/reporting-guidelines/tripod-statement/). The authors should explore how changing random partitions for testing influences their results.
Validity of the findings: Their main novel finding was that including longitudinal MRI improved the prediction of conversion from MCI to AD dementia. However, this improvement was very small (0.02 in BA) and it is unlikely to be clinically relevant as the strategy of performing two structural MRI scans over a year does not seem to be sufficiently cost-effective. Moreover, the authors have not even demonstrated that their novel model actually outperforms standard regional volumetric measures such as hippocampus volume or medial temporal thickness that can be rapidly measured with other deep learning approaches such as e.g. FastSurfer. Therefore, with the data presented here, I cannot see how this model actually represents an advance towards better prediction of AD dementia in MCI patients.
Additional comments: No additional comments


Instruction: You are one of the reviewers, your task is to write a review for the article. You will be given the title of the article, the number of the round in which the article is located, and your order among the reviewers.

Input:
Title: DEEP LEARNING PREDICTION OF MILD COGNITIVE IMPAIRMENT CONVERSION TO ALZHEIMER’S DISEASE AT 3 YEARS AFTER DIAGNOSIS USING LONGITUDINAL AND WHOLE-BRAIN 3D MRI
Review round: 1
Reviewer: 3

Output:
Basic reporting: The writing is fluid, and the text is reasonably easy to follow.
In general, there is sufficient background, and the reporting is sound.
For paper improvement, there are some remarks that need to be addressed, described below. Some details are missing, and some clarification is needed.

- Minor writing problems, for instance, "T1-weighted #D MRI" (line 27), "some used predetermination 3D patches" (line 90). Please review.

- "improve or slow the rate of decline of symptoms" (line 64). Even though the meaning could be inferred, this exact sentence is confusing. Please rewrite.

- "In contrast to traditional analysis methods such as logistic regression, ML does not require relationships between different input variables and the outcomes to be explicitly specified a priori" (lines 69-71). Actually, logistic regression is also a form of ML. Please use another example.

- "To our knowledge, there are no published studies to date on deep learning to predict MCI to AD conversion using longitudinal and whole-brain 3D MRI." (lines 100-101). In the cited paper "Convolutional Neural Networks for Classification of Alzheimer’s Disease: Overview and Reproducible Evaluation", there are some references to the combination of 3D subject-level CNN (meaning whole-brain), Longitudinal, and sMCI vs pMCI, for instance, in Tables 5 and 6. Please clarify how the proposed method is different from this paper.

- "main binary classifier metric" (line 161). Given that BA only works at a determined threshold, it would be also interesting to report AUC, as it provides a general performance of the model across a range of thresholds.

- "The neural network models [...]" (lines 173-). The network models are simply stated. I feel there is a need to add a discussion on the reasoning for choosing these models.

- "After initial analysis of networks with varying heads, global maximum pooling followed directly by a single final dense prediction layer was selected as the optimal fully connected layer architecture." (lines 194-196). Given this description, "heads" in Figures 2 and 4 might actually be misleading. Please adjust accordingly.

- "Resulting images had a resolution of 91x109x91 voxels." (lines 148-149). Considering down-sampling was performed in halves (stride=2 in Figure 3), how did authors deal with odd dimensions throughout the network? Additionally, given down-sampling was symmetrical, how did 91x109x91 voxels end up as 2x2x2 voxels or less? Please clarify.

- "L2 regularization as also added in the convolutional layers in all networks." (line 211). Please include the regularization parameter.

- "All training was performed using Tensorflow2/Keras python library, on Google Compute Platform virtual instances with Tesla V-100 GPU acceleration." (lines 211-213). It would be interesting to include some information on training and inference times.

- "Discussion" (lines 259-); "We examined three different convolutional architectures [...]" (lines 314-). I feel that, at some point in the discussion, an analysis about the performances from the different networks is missing, for instance, why an architecture would be better than the others. Please complement.
Experimental design: This research is relevant and meaningful.
For paper improvement, there are some points in the text that need better descriptions and clarification. For a rigorous investigation, a few additional experiments are necessary.

- "(3) a twin network (identical channels with weights independently optimized)." (lines 180-181). The siamese networks were experimented with two merging options. It seems that the twin network was experimented with only one merging option. Which one? Why? Please clarify. For completeness, it would be interesting to experiment with both options using the twin network as well.

- "Data were split into 75% and 25% for training and testing, respectively, with 4-fold cross validation." (lines 34-35)
- "Data for training and testing were randomly split up as 75% and 25%, with the training set composed of 415 patients (249 sMCI and 166 pMCI), and the testing set composed of 139 patients (84 sMCI and 55 pMCI). Training employed 4-fold cross validation." (lines 129-132)
- "Data for training and testing were randomly split up as 70% and 30%. The training set consisted of 387 patients (160 AD and 227 NC). Training employed 4-fold cross-validation." (lines 136-138)
- "Images were split into testing, validation, and testing sets at the patient level in order to avoid data leakage. [...]" (lines 154-159)
- "We also carefully prevented data leakage [...]" (lines 336-)
I appreciate that the authors were concerned with data leakage, as this is very important. However, the text is not completely clear, and, more importantly, I'm afraid that the provided code does not represent what is described in the text.
In each data split, it seems that cross-validation was performed on the "training" set, which makes it a combined training and validation set. If this is correct, a possible suggestion would be in the lines of: "Data were split into 75% and 25% for training/validation and testing, respectively. Then, we optimized the networks using a 4-fold cross-validation on the training/validation set, resulting in 56.25% for training and 18.75% for validation from the complete data in each fold split. In the end, we had four trained models for each experiment configuration, and reported the mean and SD BA."
Additionally, in file data_split.py, there is a single call to train_test_split when splitting training and validations sets. It seems that there is no cross-validation. Please double check.

- "During training, data augmentation was performed on the training set by rotating each MRI by up to 5% in any direction" (lines 149-150). Given that registration includes rotation, I wonder how much this rotation augmentation might actually improve (or even deteriorate) results. For completeness, I suggest the authors perform an additional experimentation without this rotation augmentation.

- "For the single timepoint experiments, we used the 12-month images because they performed better than the initial baseline images during the classification task." (lines 245-247)
- "The dual timepoint network performed better regardless of whether the initial or the follow up image was used for the single timepoint." (lines 274-275)
These are very important statements. If the 12-month images were used in the classification task, the results shouldn't change, as both baseline and 12-month images represent either AD or CN. However, if the 12-month images were used in the prediction task, then it changes the interpretation of the experiments and results, as it would mean a prediction at 2 years, instead of 3 years.
Nevertheless, using the follow up image for the single timepoint is an interesting experiment and should be reported throughout the paper, in Methods, Results, and Discussion.
Note that, in Figure 2, there are some differences between the architectures. First, as previously stated, there is the difference between baseline and follow up images, which should be compared in the single timepoint experiment. Secondly, the siamese architecture has twice the amount of input data. Finally, the twin architecture has twice the amount of input data and nearly twice the amount of parameters. I see this as an ablation study going from the "baseline" model to the "final/best" model.
For completeness, please report both single timepoint experiments.
Validity of the findings: It is not possible to completely assess the validity of the findings at this moment due to questions raised in "2. Experimental design".
Additional comments: No additional comments


Instruction: You are one of the reviewers, your task is to write a review for the article. You will be given the title of the article, the number of the round in which the article is located, and your order among the reviewers.

Input:
Title: DEEP LEARNING PREDICTION OF MILD COGNITIVE IMPAIRMENT CONVERSION TO ALZHEIMER’S DISEASE AT 3 YEARS AFTER DIAGNOSIS USING LONGITUDINAL AND WHOLE-BRAIN 3D MRI
Review round: 2
Reviewer: 1

Output:
Basic reporting: I appreciate the authors incorporating the observations made in the reviews.
There are only a few questions/remarks left.

"slow the rate of cognitive decline" (line 22)
"to improve symptoms, or at least slow their rate of decline" (line 64)
Please note that English is not my native language, so take this with a grain of salt.
The first sentence makes sense to me, while the second one does not.
In the first one, you are trying to slow the rate of cognitive decline. In the second one, you are trying to slow the rate of symptoms decline.
In my understanding, a symptom is something bad, such as bad memory. When a symptom declines, I infer that the patient is getting better. If you slow this rate, then the patient will no longer get better. That's why I suggested rewriting this sentence.

Table 2 and Table 3
It is not common to see test performance better than validation performance. Do the authors have an opinion on why this happened? It might be interesting to include some discussion about this.

"The neural network models [...]" (lines 189-)
I appreciate the authors adding a discussion about siamese networks and merging options.
I feel that the same discussion could be included regarding the types of convolutional blocks ("sequential, residual with bottleneck, and wide residual").

"After initial analysis of networks with varying heads, global maximum pooling followed directly by a single final dense prediction layer was selected as the optimal fully connected layer architecture." (lines 224-226)
I apologize for not being clear.
My understanding from this sentence is that, after the conv or merge blocks, there is a global maximum pooling layer, and a single final dense prediction layer (with 1 hidden unit, aka, neuron). There is a mismatch between this understanding and what is represented by "head" in Figures 2 and 4.
Figure 2 shows three FC layers. Figure 4A shows two FC layers. I believe that they should match.

"When a convolutional layer processes an input whose size is odd-numbered in any of its dimensions (2n-1 for any integer n) the resulting output of a stride 2 convolution will be of size n for the corresponding dimension." (lines 214-216)
I appreciate the clarification. I believe that this behavior really depends on the implementation. So I understand that the implementation used here adds a zero-padding before the convolution, so no information is lost.
Experimental design: "Using either the single timepoint with the 1-year image or both timepoints together longitudinally represents, for patients who have had MCI for one year and not yet progressed to AD, a prediction of whether they will eventually convert to AD within two more years." (lines 203-206)
Given this, maybe the authors should adjust the title (and respective text in the paper) to 2 years.
Validity of the findings: Findings are sound and grounded.
Additional comments: No additional comments


Instruction: You are one of the reviewers, your task is to write a review for the article. You will be given the title of the article, the number of the round in which the article is located, and your order among the reviewers.

Input:
Title: AUGMENTING THE TECHNOLOGY ACCEPTANCE MODEL WITH TRUST MODEL FOR THE INITIAL ADOPTION OF A BLOCKCHAIN-BASED SYSTEM
Review round: 1
Reviewer: 1

Output:
Basic reporting: In the manuscript, the authors report findings of an extended technology acceptance model (TAM) applied to analysis of blockchain use cases.

The authors provide detailed background and context in the paper covering both TAM, blockchain, and research methodology.  Motivation of the work is clear.

The paper includes well formulated hypotheses to be tested through survey studies of targeted users.

The extended TAM for blockchain based systems focuses on constructs, security, privacy, and trust. Because these concepts are closely intertwined, I feel that additional elaboration of their interpretation in context of blockchain based systems is useful to understand the structural model.
Experimental design: The survey study is designed to validate specific hypotheses derived from the new TAM for blockchain based systems. There are two targeted use cases (one consumer oriented – shopping, and the other one enterprise oriented – data sharing). The results show that TAM based predictors and constructs are affected by the use case differences.

The authors conduct a thorough analysis of the data using established structural model scheme. I feel that some of the findings are informative. Although studies using similar methodology have been performed in the past for other ICT applications, there is a gap of similar analysis to blockchain based systems, which is tackled by the authors’ work. The analysis seems comprehensive.

I wonder if the path weighting structural modeling approach has been well validated. How to separate correlational vs causal relationships between the TAM constructs? Some clarification may be helpful.
Validity of the findings: The authors conduct a comprehensive analysis of the data. The results are provided at the end of the paper.

There are certain limitations on how the results could be generalized, which is also discussed by the authors.
Additional comments: Overall, the work focuses on validating a decision model for adopting blockchain based systems. The model incorporates constructs important to blockchain based systems such as security, privacy, and trust as predictors. It applies well established methodology in technology adoption studies. Some of the reported findings are informative and potentially useful for designing new blockchain based systems.

A limitation is that the surveyed users are overall well educated and have better knowledge in ICT than the general public. Likely literacy levels for both computer and blockchain play important roles in the decision making process of adoption. It is not clear how the TAM could model such effect.

Additional justification of applying path structural modeling scheme to validate decision model of technology adoption could be useful to readers who are not experts in this area.


Instruction: You are one of the reviewers, your task is to write a review for the article. You will be given the title of the article, the number of the round in which the article is located, and your order among the reviewers.

Input:
Title: AUGMENTING THE TECHNOLOGY ACCEPTANCE MODEL WITH TRUST MODEL FOR THE INITIAL ADOPTION OF A BLOCKCHAIN-BASED SYSTEM
Review round: 1
Reviewer: 2

Output:
Basic reporting: The paper reports an empirical study investigating how various latent variables impact users’ attitudes and intentions of using a blockchain-based system. The study has used the primary Technology Acceptance Model (TAM)-based predictors. Data from more than 60 respondents, mostly Master level IT students, are collected and analyzed.
Positive aspects of the study and the manuscript
-	Comprehensive related studies
-	Detailed comparison of this study’s results with related work
-	Structured presentation of the research design
-	A detailed report of the data analysis results
Experimental design: The experimental design is explained in detail. However, I could not find a mapping between the constructs and the questions in each questionnaire. From the questions I can see in the questionnaires, constructs of two latent variables need to be explained better.
-	Quality of the system. In the software engineering domain, security and privacy are regarded as part of Quality of the system. It seems that the questions in the questionnaire related to this latent variable are about dependability. In this case, it would be better to use reliability/dependability rather than using the Quality of the system, which is a broad concept.
-	Attitude: I could not find myself which constructs measure attitude in the questionnaire. The term attitude also needs to be defined more precisely in the paper.
Validity of the findings: In the “participants and demographics” chapter, the authors mentioned that some data are cleaned. Please explain the criteria applied to include/exclude data.

The main limitation of the study is the limited number of respondents. The respondents are also people who know IT technologies well. The study needs to be replicated to users of
different profiles. The authors have discussed these limitations. This is acceptable.

Another limitation the authors need to discuss is the results related to DSS. DSS is more related to companies than individual users. How much can we trust the results from students, who usually do not have industrial experience, to answer company-related questions? This needs to the reflected and discussed in the paper.
Additional comments: The discussion chapters can be structured better. Most content of the discussion chapter is about comparison with related work. The authors could discuss more the implications of the results to academia and industry.
In the discussion chapter, some conclusions are too strong. For example, line 870, it is because … Without follow-up interviews to confirm the assumptions, the authors shall write “it may be because …”


Instruction: You are one of the reviewers, your task is to write a review for the article. You will be given the title of the article, the number of the round in which the article is located, and your order among the reviewers.

Input:
Title: AUGMENTING THE TECHNOLOGY ACCEPTANCE MODEL WITH TRUST MODEL FOR THE INITIAL ADOPTION OF A BLOCKCHAIN-BASED SYSTEM
Review round: 1
Reviewer: 3

Output:
Basic reporting: 1) They constructed the survey questions after a pilot study with 14 researchers, so the questionnaire is supposed to be good enough for the study.

2) The authors evaluated ease of use, perceived usefulness, quality of the system, intention to use the system, security, trust, and other factors that seem promising.
Experimental design: It is review/study type manuscript rather than technical paper.  The experimental design are not focused which looks fine considering its nature.
Validity of the findings: 1) 14 hypotheses are constructed and discussed their decision with the required illustrations.
Additional comments: 1) Authors have merged the related works within the introduction, which looks verbose but lacks focus on the problems, challenges targeted and specific contributions claimed.

2) It is recommended to separate/merge the referred literature within the later 'Related Work subsection and keep the introduction focused on the targeted problems and challenges. In addition, it is better to have a list of contributions Authors claim after addressing the issues.

3) The contents style as in line 125, 127, 146, etc., preceded with '..our/current study..' looks bringing an impression that Authors have struggled to fit the survey type literature in the Blockchain-based technical contexts (Model for the initial adoption of a blockchain-based
system) as mentioned in the Abstract.

4) Writing needs improvement in terms of professionalism and broader readers' attention.

5) Too many mathematical notations and values seem confusing average readers and conflict and moving its focus from review-nature.

6) Language need to improve, should check grammatical errors and typos, formatting.

7) Its highly recommended to improve the organization of the manuscript.


Instruction: You are one of the reviewers, your task is to write a review for the article. You will be given the title of the article, the number of the round in which the article is located, and your order among the reviewers.

Input:
Title: AUGMENTING THE TECHNOLOGY ACCEPTANCE MODEL WITH TRUST MODEL FOR THE INITIAL ADOPTION OF A BLOCKCHAIN-BASED SYSTEM
Review round: 2
Reviewer: 1

Output:
Basic reporting: The updated manuscript has addressed the comments from the reviewers. I do not have further comments. I think the paper is acceptable.
Experimental design: The experimental design is reasonable.
Validity of the findings: The findings are valid and the limitations are discussed properly.
Additional comments: The paper can be accepted from my point of view.


Instruction: You are one of the reviewers, your task is to write a review for the article. You will be given the title of the article, the number of the round in which the article is located, and your order among the reviewers.

Input:
Title: LINKPRED: A HIGH PERFORMANCE LIBRARY FOR LINK PREDICTION IN COMPLEX NETWORKS
Review round: 1
Reviewer: 1

Output:
Basic reporting: This paper proposed a library with ease of use interfaces for link prediction. It reported and reviewed briefly the current existing related works' limitations that lead to LinkPred development. It gave a detail explanation on how the LinkPred was produced and deployed. The report clearly explained how the development of the LinkPred and showed the result of its time execution performance on eight different networks as depicted in Table 1. However some questions arise - From Figure 1, how 'user-defined' can be deployed together with the LinkPred library? How different is between 'user defined' in link prediction algorithm with performance measures? How the work prove ease of use, extensibility, and efficiency as claimed in line 37? Why Djikstra's algo is chosen? ULPredictor must be explained further as what method has been used to predict? why it has been chosen? How accurate the prediction result is?
Experimental design: The experimental design started by showing the architectural design in general explanation followed by the development of the library. Research questions are not provided with that may lead to many questions on the development of the library. However, the elaboration of the development is very useful for researchers to understand how the library works. A section on how users apply the library in their scenario will be very beneficial.
Validity of the findings: Thorough experiments to prove the library's ease of use, extensibility and efficiency is needed to improve the paper further, despite its clear explanation on the library development. How the LinkPred library can be used in different programming languages that affect effectiveness? It is believed the conclusion will be well stated when most of these questions answered.
Additional comments: The work proposed a library for link prediction which is very beneficial and relevant in the current network situation. The library will benefit practitioners and researchers to fasten up their work process as the network is scaled every day. However, the proposed library needs more work for improvements as questions arise especially in term of its reliability.


Instruction: You are one of the reviewers, your task is to write a review for the article. You will be given the title of the article, the number of the round in which the article is located, and your order among the reviewers.

Input:
Title: LINKPRED: A HIGH PERFORMANCE LIBRARY FOR LINK PREDICTION IN COMPLEX NETWORKS
Review round: 1
Reviewer: 2

Output:
Basic reporting: References to key libraries doing link prediction is missing including SNAP (http://snap.stanford.edu/snap/description.html) and GEM (https://github.com/palash1992/GEM).

The authors have not considered graph embedding/representation learning based approaches for link prediction which have been very popular recently.

The documentation in the library is very limited.
Experimental design: The approaches included in the library are not comprehensive missing several state-of-the-art link prediction methods including those based on graph representation learning which are included in libraries such as SNAP an GEM.
Validity of the findings: The methods implemented have been duly cited and well defined.
Additional comments: No additional comments


Instruction: You are one of the reviewers, your task is to write a review for the article. You will be given the title of the article, the number of the round in which the article is located, and your order among the reviewers.

Input:
Title: LINKPRED: A HIGH PERFORMANCE LIBRARY FOR LINK PREDICTION IN COMPLEX NETWORKS
Review round: 2
Reviewer: 1

Output:
Basic reporting: No comment.
Experimental design: No comment
Validity of the findings: No comment
Additional comments: The latest version of the paper has very significant improvement which clearly answered the previous questions with good explanation by additional data and experiments. Important and relevant works of literature are included where necessary in each section. The research methodology is clear. The library construction is clear and justified. The issue of the library's reliability has been answered well.


Instruction: You are one of the reviewers, your task is to write a review for the article. You will be given the title of the article, the number of the round in which the article is located, and your order among the reviewers.

Input:
Title: LINKPRED: A HIGH PERFORMANCE LIBRARY FOR LINK PREDICTION IN COMPLEX NETWORKS
Review round: 2
Reviewer: 2

Output:
Basic reporting: The authors have addressed all my concerns.
Experimental design: The authors have addressed all my concerns.
Validity of the findings: The authors have addressed all my concerns.
Additional comments: Thanks for addressing all the concerns. The manuscript and code repository look good now.