Instruction: You are one of the reviewers, your task is to write a review for the article. You will be given the title of the article, the number of the round in which the article is located, and your order among the reviewers.

Input:
Title: DETECTION OF HIERARCHICAL CROWD ACTIVITY STRUCTURES IN GEOGRAPHIC POINT DATA
Review round: 1
Reviewer: 1

Output:
Basic reporting: This paper presents a synthetic data generator that reproduces structures commonly found on geographical event data sets and describes clustering and hierarchical scale extraction algorithms to be evaluated. Basically, the topic is interesting and the manuscript is well-written but I have some comments to be improved.

Major comments:
-	The authors mentioned that the second contribution is to propose an improvement of the DBSCAN algorithm, but existing clustering algorithms were introduced and used. Please clarify improvement points of the algorithm. The title should also be modified to fit the content because the current one seems to be the proposal of an algorithm.

Minor comments:
-	Figure 1 should be explained more detail such as the meaning of colors. Also, the quality of the Figure 1(b) should be improved.
-	The authors discussed the results in Section 7, so the section name should be Results and discussions.
-	There are some typos as follows.
Line 20: “even data” would be “event data”
Line 166: “as shown in 1” would be “as shown in Figure 1”
Line 431: There is a space missing after the comma.
Line 492: “This results” should be “These results”
Experimental design: no comment
Validity of the findings: no comment
Additional comments: No additional comments


Instruction: You are one of the reviewers, your task is to write a review for the article. You will be given the title of the article, the number of the round in which the article is located, and your order among the reviewers.

Input:
Title: DETECTION OF HIERARCHICAL CROWD ACTIVITY STRUCTURES IN GEOGRAPHIC POINT DATA
Review round: 1
Reviewer: 2

Output:
Basic reporting: 1.	There are several places (Lines 131-137, Lines 186-191, Lines 494-496) where it is argued that the expected differences between different clustering algorithms (e.g., HDBSCAN and OPTICS vs. DBSCAN and its adaptive version). It would be beneficial to use a toy example dataset to demonstrate the differences argued here.
2.	Some figures (e.g., Figure 1) need to be of higher resolution to be legible (check all other figures for this issue). In Figure 1 caption, please describe how (a) corresponds with (b). For example, is a node in (b) a point in (a)?
3.	Lines 160-161: The reviewer did not quite understand what “three iterations of head-tail breaks” entails. Please elaborate.
4.	Line 249: What important factors need to be considered when manually setting parameters?
5.	Line 518:   Does this "new method" refer to applying clustering algorithms recursively to discover structure within structure in general, or specifically to applying the adaptive DBSCAN recursively? Need to be clear.
Experimental design: 1.	Line 404: “statistically significant evaluations”. What does this entail exactly? Later in the results section, only box plots are used to show the differing performance of the algorithms. I would expect some sorts of statistical significance test on the differences (or comparison of confidence intervals?).
2.	Lines 530-531: The reviewer thinks it is necessary to do so in this paper. Apply the adaptive DBSCAN a real-world dataset recursively to discover structures within structures, and see if the results make sense (probably with only qualitative evaluations).
Validity of the findings: 1.	Line 404: “statistically significant evaluations”. What does this entail exactly? Later in the results section, only box plots are used to show the differing performance of the algorithms. I would expect some sorts of statistical significance test on the differences (or comparison of confidence intervals?).
2.	Lines 530-531: The reviewer thinks it is necessary to do so in this paper. Apply the adaptive DBSCAN a real-world dataset recursively to discover structures within structures, and see if the results make sense (probably with only qualitative evaluations).
Additional comments: See the annotated pdf file attached for other minor comments.


Instruction: You are one of the reviewers, your task is to write a review for the article. You will be given the title of the article, the number of the round in which the article is located, and your order among the reviewers.

Input:
Title: DETECTION OF HIERARCHICAL CROWD ACTIVITY STRUCTURES IN GEOGRAPHIC POINT DATA
Review round: 2
Reviewer: 1

Output:
Basic reporting: no comment
Experimental design: no comment
Validity of the findings: no comment
Additional comments: Thank you for your revisions. I checked the revised manuscript and confirmed that the authors have addressed my major and minor comments. I think the paper is accepted for publication after minor revision below.

1. Figure 1: Instead of arranging figures vertically, how about arranging them horizontally?
2. The caption of Figure 14: hours should be removed.


Instruction: You are one of the reviewers, your task is to write a review for the article. You will be given the title of the article, the number of the round in which the article is located, and your order among the reviewers.

Input:
Title: DETECTION OF HIERARCHICAL CROWD ACTIVITY STRUCTURES IN GEOGRAPHIC POINT DATA
Review round: 2
Reviewer: 2

Output:
Basic reporting: The revised manuscript has sufficiently addressed my comments. I now recommend acceptance for publication.
Experimental design: The revised manuscript has sufficiently addressed my comments. I now recommend acceptance for publication.
Validity of the findings: The revised manuscript has sufficiently addressed my comments. I now recommend acceptance for publication.
Additional comments: The revised manuscript has sufficiently addressed my comments. I now recommend acceptance for publication.


Instruction: You are one of the reviewers, your task is to write a review for the article. You will be given the title of the article, the number of the round in which the article is located, and your order among the reviewers.

Input:
Title: AN IMPROVED TWO-STAGE LABEL PROPAGATION ALGORITHM BASED ON LEADERRANK
Review round: 1
Reviewer: 1

Output:
Basic reporting: Line 87: The "Q" should be italicized.
Line 147: "definition similarity": Did you mean "definition of similarity"?
Line 216: Suggest to remove the word "real".
Experimental design: Line 297: Is the alpha optimized for the networks in this research work, or it is an optimized value that can be used in any other network? Kindly clarify.
Validity of the findings: Table 5 and 6: Is there a reason why the LPA-MNI is included in Table 6 but not in Table 5? As it is also one of the latest LPA variants.
Line 380: In most cases, LPA and its variants will start to deteriorate past a certain threshold of mixing parameter. I wonder how well the proposed method handles higher values of mixing parameter. For example, mu = 0.6 and 0.7.
The proposed method produces high-quality community division in terms of NMI and Q for the LFR networks. It outperformed the other methods in terms of Q (more than double in some cases). Thus, it will be interesting to see what is the number of communities that are detected by the proposed method as compared to the actual number of communities in the LFR networks.
Additional comments: No additional comments


Instruction: You are one of the reviewers, your task is to write a review for the article. You will be given the title of the article, the number of the round in which the article is located, and your order among the reviewers.

Input:
Title: AN IMPROVED TWO-STAGE LABEL PROPAGATION ALGORITHM BASED ON LEADERRANK
Review round: 1
Reviewer: 2

Output:
Basic reporting: This paper proposes a new community detection method based on LeaderRank and LPA, the problem has significant importance, and the proposed method seems somewhat interesting. The related work section should include more related and recent studies to state the new contribution of the proposed method against ones. I could find a few very recent studies on the use of LPA in community detection which have not been indicated; moreover,  the new contributions and shortcomings of existing methods may also be included somewhere in the manuscript? A working example to illustrate the functioning of the proposed method could be of immense help to the readers.
Experimental design: The authors may consider incorporating the convergence, sensitivity to the starting conditions, and other relevant characteristics of the proposed algorithm (including time complexity analysis). The authors may compare their approach with the current benchmarks on community detection using relatively real-world data sets of significant size based on the actual CPU time (current datasets considered are too small).
Validity of the findings: The proposed method should be competitive in computational performance for large-scale network data sets (>30000 nodes, for example); otherwise, the utility of the proposed approach to deal with realistically large networks shall remain unknown.
Additional comments: No additional comments


Instruction: You are one of the reviewers, your task is to write a review for the article. You will be given the title of the article, the number of the round in which the article is located, and your order among the reviewers.

Input:
Title: AN IMPROVED TWO-STAGE LABEL PROPAGATION ALGORITHM BASED ON LEADERRANK
Review round: 1
Reviewer: 3

Output:
Basic reporting: no comment
Experimental design: no comment
Validity of the findings: no comment
Additional comments: In this manuscript entitled “an improved two stage label propagation algorithm based on LeaderRank”, authors propose an improved two-stage label propagation algorithm to solve the problems of poor stability and low modularity.

In the introduction, authors sum up their contributions and experimental findings. They focus their study on label propagation algorithm for community detection, they point out its advantages as low time complexity and suitability for large scale-networks as well as its limitations as randomness and instability.  Authors give then a short overview of improved algorithms that take into consideration the importance of each node in order to seed node selection. They detailed the LPA-TS algorithm and underlined the inaccurate proposed similarity measurement between nodes that aims to rank nodes for labeling.
They propose a new version of LPA-TS named LPA-ITSLR where an improvement of the similarity measurement is proposed as well as an optimised strategy for node label updating based on LeaderRank value, this will avoid the randomness of ordered labels update.
The paper is well organised in two main other parts beside the introduction and the conclusion.
 In the second part, theoretical basis with the necessary notions and indicators are exposed and defined. Likewise, the proposed algorithm is detailed.

The third part detailed experiment and analysis, experiments are carried out on six classical social network data sets where users show that LPA_ITSLR performed well for all these data sets and yielded more stable community partitioning in comparison with LPA and LPA-TS. Performance comparison of LPA_ITSLR with four other recent community detection algorithms are also carried out on the same classical data sets. Two of these algorithms are based on the importance of nodes. Experimental results show that LPA_ITSLR yielded the highest modularity and the most stable results. Other comparison results with seven other classical community detection algorithms show also good performance especially in term of modularity on the three following datasets: Karate, Dolphin and football.
Other experiment studies were carried out on artificial sets and showed the same improvement.

Here are some suggestion and remarks:
The complexity of the algorithm LPA-ITSLR is not studied neither experimented on large-scale graph. In fact, considering the new similarity measurement as well as the new proposed strategy for updating labels leads certainly to improve the randomness and the instability of the algorithm, but may, on the other hand, slow down the processing in comparison to the classical LPA.  Could you please comment this or mention it is as a perspective of your study?

On the other hand, here are some remarks related to terminology, authors use the terms “community discovery” instead of the most common used one which is “community detection”. Is there any reason to this?
 In table 3, it seems to me that d is the diameter and not the path average length. Likewise, c is called “clustering coefficient” and not the aggregating one.
At line 49 the place of the citation number 1 is inappropriate, could you please put it at the end of the sentence? At line 241, please check the sentence: “the node .. and the community .. has a large similarity”


Instruction: You are one of the reviewers, your task is to write a review for the article. You will be given the title of the article, the number of the round in which the article is located, and your order among the reviewers.

Input:
Title: AN IMPROVED TWO-STAGE LABEL PROPAGATION ALGORITHM BASED ON LEADERRANK
Review round: 2
Reviewer: 1

Output:
Basic reporting: No comment.
Experimental design: No comment.
Validity of the findings: No comment.
Additional comments: The paper is well written and I am looking forward to any related future work on this method.


Instruction: You are one of the reviewers, your task is to write a review for the article. You will be given the title of the article, the number of the round in which the article is located, and your order among the reviewers.

Input:
Title: AN IMPROVED TWO-STAGE LABEL PROPAGATION ALGORITHM BASED ON LEADERRANK
Review round: 2
Reviewer: 2

Output:
Basic reporting: Supplementary material contains .class files (unreadable) of most of the code/ material, which is not following the Peerj CS Policy where code should be made available ( Though it may be with restricted access).
Experimental design: The current utility of the proposed method to deal with realistically large networks remains unknown.
Validity of the findings: I appreciate the efforts made by the authors to address my comments in the previous round (although not all of them have been answered). Different parts of the paper have been improved. However, I still have a major concern that results from previous remarks that have not been addressed.
Specifically, my comment on the "Validity of Findings"  has not been satisfactorily answered. I reiterate my comment below.
"...Validity of the findings
The proposed method should be competitive in computational performance for large-scale network data sets (>30000 nodes, for example); otherwise, the utility of the proposed approach to deal with realistically large networks shall remain unknown....".

Based on the availability of large-scale network data set, one would expect a new community detection algorithm to be competitive in terms of computational performance. The authors should explain how the proposed method is competitive for realistically large networks?
Additional comments: No additional comments


Instruction: You are one of the reviewers, your task is to write a review for the article. You will be given the title of the article, the number of the round in which the article is located, and your order among the reviewers.

Input:
Title: AN IMPROVED TWO-STAGE LABEL PROPAGATION ALGORITHM BASED ON LEADERRANK
Review round: 3
Reviewer: 1

Output:
Basic reporting: Different parts of the paper have been significantly improved in the previous rounds of review.
Experimental design: I appreciate the author's effort to address my comments in the previous round, although not all of them have been answered. Additional experiments may be required to address my concerns which have not been addressed adequately.
Validity of the findings: I still have a major concern that has not been addressed. It is important to show the computational competitiveness of the proposed method for realistically large networks and not for LFRs (which are used by the authors to address my comments). Can the proposed method outperform any one of the benchmarks like the Fast greedy algorithm, Walktrap algorithm, or Edge Betweenness algorithm on realistically large networks? It is not easy to understand the novelty of publication without having a clear and convincing contribution, as suggested above.
Additional comments: No additional comments


Instruction: You are one of the reviewers, your task is to write a review for the article. You will be given the title of the article, the number of the round in which the article is located, and your order among the reviewers.

Input:
Title: AN IMPROVED TWO-STAGE LABEL PROPAGATION ALGORITHM BASED ON LEADERRANK
Review round: 3
Reviewer: 2

Output:
Basic reporting: no comment
Experimental design: no comment
Validity of the findings: no comment
Additional comments: In this manuscript entitled “an improved two stage label propagation algorithm based on LeaderRank”, authors propose an improved two-stage label propagation algorithm to solve the problems of poor stability and low modularity.

According to first round review suggestions, authors improved the algorithm and carry out experiments on large-scale networks. The largest data set in the original paper contains 10000 nodes. they generated simulation data sets containing 20000, 30000, 40000 and 50000 nodes respectively. From the experimental results, they draw the conclusion that, as the size of the data set increases, the time complexity of this algorithm increases, but it can still achieve a high modularity and NMI in terms of large-scale data sets. It would be preferable to give some measurement allowing to observe how the time complexity is increasing.


Instruction: You are one of the reviewers, your task is to write a review for the article. You will be given the title of the article, the number of the round in which the article is located, and your order among the reviewers.

Input:
Title: AN IMPROVED TWO-STAGE LABEL PROPAGATION ALGORITHM BASED ON LEADERRANK
Review round: 4
Reviewer: 1

Output:
Basic reporting: Seems ok
Experimental design: Seems ok
Validity of the findings: The authors have made efforts to address some of the major concerns raised in the previous rounds of review; I recommend this revised manuscript for further processing.
Additional comments: Authors should check leftover grammatical/typographical issues (like in Table 12) at their end.


Instruction: You are one of the reviewers, your task is to write a review for the article. You will be given the title of the article, the number of the round in which the article is located, and your order among the reviewers.

Input:
Title: MALWARE DETECTION FRAMEWORK BASED ON GRAPH VARIATIONAL AUTOENCODER EXTRACTED EMBEDDINGS FROM API-CALL GRAPHS
Review round: 1
Reviewer: 1

Output:
Basic reporting: Thias work presents and interesting research on applying GVAE to dimensionality reduction, followed by some classic ML models applied to the resulting features. Some comments:
- Why to choose linear SVM and LightGBM and not others.
- Highlight all assumptions and limitations of your work.
- Conclusions should provide some lessons learnt.
- Related works section does not mention recent research effors in feature characterization in NIDS using also clustering and contrastive learning, as well as, review works on VAEs.  Authors are advised to refer to the following related articles to add some discussions:  [1] Variational data generative model for intrusion detection, Knowledge and Information Systems, 2018 [2] Analysis of Autoencoders for Network Intrusion Detection, Sensors, 2021. [3] Network Intrusion Detection Based on Supervised Adversarial Variational Auto-Encoder With Regularization, IEEE Access, 2020 [4] Supervised contrastive learning over prototype-label embeddings for network intrusion detection, Information Fusion, 2022
Experimental design: The experimental design and the descriptions of experimental results is well done and sufficient.
Validity of the findings: Accuracy anf F1 has been obtained, and to assess the significance of results a Wilcoxon Signed Rank Test was applied using a 0.05 significance level. The results are correctly analyzed and are significant. To obtain some additional metrics eg recall, precision, etc,...would be fine but it is not strictly necessary. Table 9 of classification results of recent malware studies is interesting to position the results in perspective with others.
Additional comments: No additional comments


Instruction: You are one of the reviewers, your task is to write a review for the article. You will be given the title of the article, the number of the round in which the article is located, and your order among the reviewers.

Input:
Title: MALWARE DETECTION FRAMEWORK BASED ON GRAPH VARIATIONAL AUTOENCODER EXTRACTED EMBEDDINGS FROM API-CALL GRAPHS
Review round: 1
Reviewer: 2

Output:
Basic reporting: This study proposes a malware detection model based on API-call graphs and uses Graph Variational Autoencoder (GVAE), which is a variant of graph neural networks, to reduce the size of graph nodes extracted from Android apk files.
Basic reporting is good. However, literature work should be added more from the year 2020-2021
Experimental design: Authors did not mentioned why they use Recursive Feature Elimination (RFE) and Fisher Score (FS) to conduct  feature selection in comparison to other studies. Please explain why you used these methods. Furthermore, Is the analysis is sufficient enough by using SVM as a model., why not considering other well-known classifiers that classifies better than SVM?

Please explain list of feature selection in tabular format.
In table 6 it seems that increasing number of features will increase accuracy, which is normally not true. So, what will be the accuracy at 60 features?
Validity of the findings: Please explain the parameter space of LightGBM in detail.
Additional comments: No additional comments


Instruction: You are one of the reviewers, your task is to write a review for the article. You will be given the title of the article, the number of the round in which the article is located, and your order among the reviewers.

Input:
Title: MALWARE DETECTION FRAMEWORK BASED ON GRAPH VARIATIONAL AUTOENCODER EXTRACTED EMBEDDINGS FROM API-CALL GRAPHS
Review round: 1
Reviewer: 3

Output:
Basic reporting: The paper is well written and the content is well connected. In terms of basic reporting I would recommend following corrections:
1) The abstract of the paper is lengthy. Please reduce the content of the abstract and omitting detailed methodology and only report the primary results.
2) The Figure methodology diagram should be re-drawn as a block diagram. The diagram should be divided in to different connected components.
3) The contributions of the paper should be written in bullet points or with numbering.
4) The paragraph after the contributions should be re-written.
5) The API extraction from the data set is not explained. How did you extracted the API calls? Which tools did you used? This should be explained.
Experimental design: The experiments are presented clearly, however, i have concern with the explanation of dataset. Please address the following:
1) Include a section which explains the dataset. I do not see the number of benign apps being used. Is the dataset balanced or not?
2) Include a table to present/explain the dataset.
Validity of the findings: Satisfied.
Additional comments: No additional comments


Instruction: You are one of the reviewers, your task is to write a review for the article. You will be given the title of the article, the number of the round in which the article is located, and your order among the reviewers.

Input:
Title: MALWARE DETECTION FRAMEWORK BASED ON GRAPH VARIATIONAL AUTOENCODER EXTRACTED EMBEDDINGS FROM API-CALL GRAPHS
Review round: 2
Reviewer: 1

Output:
Basic reporting: It is ready for publication
Experimental design: It is ready for publication
Validity of the findings: It is ready for publication
Additional comments: It is ready for publication


Instruction: You are one of the reviewers, your task is to write a review for the article. You will be given the title of the article, the number of the round in which the article is located, and your order among the reviewers.

Input:
Title: MALWARE DETECTION FRAMEWORK BASED ON GRAPH VARIATIONAL AUTOENCODER EXTRACTED EMBEDDINGS FROM API-CALL GRAPHS
Review round: 2
Reviewer: 2

Output:
Basic reporting: Satisfied
Experimental design: Changes are made as mentioned
Validity of the findings: Satisfied
Additional comments: Suggested changes are modified.


Instruction: You are one of the reviewers, your task is to write a review for the article. You will be given the title of the article, the number of the round in which the article is located, and your order among the reviewers.

Input:
Title: AN ETHICAL VISUALIZATION OF THE NORTHCOVID-19 MODEL
Review round: 1
Reviewer: 1

Output:
Basic reporting: I’m impressed how many words can be spent for explaining and justifying the design of a simplified user interface. The paper reads very well, however, I can’t really assess its value as the videos themselves, not to speak of the system generating the videos on the fly, are not available, in contrast to the original website, https://covid.datalab.science, for which the simplified user interface has been developed for.

I suggest to add a few words explaining how the outcome of the reported work will be used/made available in the future.
Experimental design: no comment
Validity of the findings: Can't be assessed as the main result of the paper (a video-generating add-on to a given web interface) is not available.
Additional comments: This paper has nine authors; it could be helpful to see how each of the authors contributed to the reported work.

There are no raw data.

minor:
line 119: there seems to be something missing after ‘section’;


Instruction: You are one of the reviewers, your task is to write a review for the article. You will be given the title of the article, the number of the round in which the article is located, and your order among the reviewers.

Input:
Title: AN ETHICAL VISUALIZATION OF THE NORTHCOVID-19 MODEL
Review round: 1
Reviewer: 2

Output:
Basic reporting: The authors examine the outputs of an epidemic model called “NorthCOVID-19” along with its web interface to see what issues it may have when being interpreted by the general public. They identified three main issues that could cause anxiety and confusion and focused on creating a solution that would thoroughly explain the results in an ethical, easy-to-understand format. They built audio and video methods including using AI to communicate the epidemic model results.

This is a well-written paper that I recommend to be accepted.
Experimental design: The experimental design is clear with sufficient details to reproduce the experiments and results.
Validity of the findings: Conclusions are well-stated.
Additional comments: No additional comments


Instruction: You are one of the reviewers, your task is to write a review for the article. You will be given the title of the article, the number of the round in which the article is located, and your order among the reviewers.

Input:
Title: AN ETHICAL VISUALIZATION OF THE NORTHCOVID-19 MODEL
Review round: 1
Reviewer: 3

Output:
Basic reporting: Overall it looks interesting.
However, I am not too sure if the assumptions made are true.
Experimental design: Assuming that the proposed model is for public use. I don't think so. You cannot find the model anymore.
Validity of the findings: One needs to provide some facts about how much the model was used by the general public.
How independent the study can be if the first author of this paper is the second author of the published model paper?
A sample of 40 is not too bad, but not knowing who the participants are in real trouble.
Additional comments: No additional comments


Instruction: You are one of the reviewers, your task is to write a review for the article. You will be given the title of the article, the number of the round in which the article is located, and your order among the reviewers.

Input:
Title: AN ETHICAL VISUALIZATION OF THE NORTHCOVID-19 MODEL
Review round: 2
Reviewer: 1

Output:
Basic reporting: () re  individual contributions of all authors: I assume that this will be visible in the final paper; so far this information is not included.

() https://github.com/andrfish/NorthCOVID19, Video Generation Script for the COVID Crushers,
introductory comment:
“This script takes input from NorthCOVID-19 (see model here) and produces a video animation (see sample_format.pdf) of the results. The file "sample_output.csv" is an example of what the output will look like, "sample_parameters.json" is an example saved file of the parameters used, and "sample_results.json" is an example of the results output to the website from the simulation.”

-> none of the links work - ‘NorthCOVID-19’ , ‘model’;
for some of the pointers there are no links given at all, eg. sample_format.pdf, sample_results.json, sample_parameters.json.

() “The result was a video”
thanks for adding the video generation feature to the original website https://covid.datalab.science;  I suggest to move the link to this website, hidden so far in a footnote, to a more prominent position.

() “Please do not leave this page, the generation may take up to 10 minutes and the download will start automatically.”
-> having to wait up to 10’ (it actually took almost as long) is a rather long delay for getting something simplified compared to what I immediately get and comprises actually more information.

() the last slide in the generated “video” (is more a sequence of slides) needs some normalisation to make the results visible.

in summary,
I’m still amazed how many words one can spend for explaining and justifying the design of a simplified user interface. I personally would prefer the direct output as offered by the original website https://covid.datalab.science; not to say that I somehow feel offended that I - as a user-  is  considered not to be able to understand what the original diagrams are telling me.

But I’m happy to accept that I might not belong to the actual target group of the tool.
Experimental design: N/A
Validity of the findings: N/A
Additional comments: none


Instruction: You are one of the reviewers, your task is to write a review for the article. You will be given the title of the article, the number of the round in which the article is located, and your order among the reviewers.

Input:
Title: MATHEMATICAL COVID-19 MODEL WITH VACCINATION: A CASE STUDY IN SAUDI ARABIA
Review round: 1
Reviewer: 1

Output:
Basic reporting: -
Experimental design: -
Validity of the findings: -
Additional comments: Please add contribution with bullet mark in introduction (Please see and cite [1-9])
Please add lit. review table and add your research in end of table and show gap research (Please see and cite [1-9])
Please add notation list and classify to sets (indices) , parameters, decision variables (Please see and cite [1-9])
Please add Managerial insights and practical implications (Please see and cite [1-9]).
Please add results with bullet mark in conclusion (Please see and cite [1-9]).
Please suggest uncertainty form like fuzzy, robust optimization, data-driven and stochastic.
Please suggest risk form like CVaR, VaR.

[1] Lotfi, R., Kheiri, K., Sadeghi, A., & Babaee Tirkolaee, E. (2022). An extended robust mathematical model to project the course of COVID-19 epidemic in Iran. Annals of Operations Research, 1-25.
[2] Lotfi, R., Kargar, B., Gharehbaghi, A., & Weber, G. W. (2021). Viable medical waste chain network design by considering risk and robustness. Environmental Science and Pollution Research, 1-16.
[3] Lotfi, R., Mardani, N., & Weber, G. W. (2021). Robust bi-level programming for renewable energy location. International Journal of Energy Research, 45(5), 7521-7534.
[4] Lotfi, R., Kargar, B., Hoseini, S. H., Nazari, S., Safavi, S., & Weber, G. W. Resilience and sustainable supply chain network design by considering renewable energy. International Journal of Energy Research.
[5] Lotfi, R., Mehrjerdi, Y. Z., Pishvaee, M. S., Sadeghieh, A., & Weber, G. W. (2021). A robust optimization model for sustainable and resilient closed-loop supply chain network design considering conditional value at risk. Numerical Algebra, Control & Optimization, 11(2), 221.
[6] Lotfi, R., Yadegari, Z., Hosseini, S. H., Khameneh, A. H., Tirkolaee, E. B., & Weber, G. W. (2020). A robust time-cost-quality-energy-environment trade-off with resource-constrained in project management: A case study for a bridge construction project. Journal of Industrial & Management Optimization.
[7] Lotfi, R., Safavi, S., Gharehbaghi, A., Ghaboulian Zare, S., Hazrati, R., & Weber, G. W. (2021). Viable Supply Chain Network Design by considering Blockchain Technology and Cryptocurrency. Mathematical Problems in Engineering, 2021.
[8] Lotfi, R., Sheikhi, Z., Amra, M., AliBakhshi, M., & Weber, G. W. (2021). Robust optimization of risk-aware, resilient and sustainable closed-loop supply chain network design with Lagrange relaxation and fix-and-optimize. International Journal of Logistics Research and Applications, 1-41.
[9] Lotfi, R., Kargar, B., Rajabzadeh, M., Hesabi, F., & ?zceylan, E. (2022). Hybrid Fuzzy and Data-Driven Robust Optimization for Resilience and Sustainable Health Care Supply Chain with Vendor-Managed Inventory Approach. International Journal of Fuzzy Systems, 1-16.


Instruction: You are one of the reviewers, your task is to write a review for the article. You will be given the title of the article, the number of the round in which the article is located, and your order among the reviewers.

Input:
Title: MATHEMATICAL COVID-19 MODEL WITH VACCINATION: A CASE STUDY IN SAUDI ARABIA
Review round: 1
Reviewer: 2

Output:
Basic reporting: The English is poor. Authors have to revise the paper carefully. For example, "The discovery of a new form of corona-viruses in December 2019, SARS-CoV-2, commonly named covid-19, has transformed the world." the word transformed here is wrongly used. "The world has been waiting for the vaccine for almost one year." may be more than one year? Also line 261 and line 265 should be corrected.
Experimental design: The numerical experiment are given. However, wome detail illustration should be given on these results with figures.
Validity of the findings: The results in this paper deserve publication.
Additional comments: The references are not completed, such as Nonlinear Dynamics 106, 1491-1507, 2021, Nonlinear Dynamics 106, 1347-1358, 2021.


Instruction: You are one of the reviewers, your task is to write a review for the article. You will be given the title of the article, the number of the round in which the article is located, and your order among the reviewers.

Input:
Title: MATHEMATICAL COVID-19 MODEL WITH VACCINATION: A CASE STUDY IN SAUDI ARABIA
Review round: 1
Reviewer: 3

Output:
Basic reporting: The COVID-19 pandemic has already spread throughout the world and the people are aware about the diseases and they are using precautions about the pandemic. But, still the covid-19 is spreading very quickly. Formulate a mathematical model for the transmission dynamics of COVID-19. Few minor issues:

---- The abstract is a little thin and does not quite convey the vibrancy of the findings and the depth of the main conclusions. The authors should please extend this somewhat for a better first impression.
---- The manuscript lacks motivation. Author needs to include the motivation of the study.
-----To stop the spread of the diseases vaccine is needed. But, in absence of the vaccine people must have maintain the social distancing. In order to maintain the social distancing must obey the modeling rule. The introduction need to be improved by incorporating some recent references of COVID-19 pandemic.  To do so, I suggest some modeling work must be included in the references: "Modeling and forecasting the COVID-19 pandemic in India, Chaos, Soliton & Fract. 139 (2020) 110049", “Mathematical modeling of the COVID-19 outbreak with intervention strategies, Results in Physics, 2021, 104285”.
Experimental design: Need to explain properly the parameter estimation. For parameter estimation and model validation authors should include the manuscripts:

 "Forecasting the daily and cumulative number of cases for the COVID-19 pandemic in India, Chaos, 30(7) (2020) 071101", “Impact of social media advertisements on the transmission dynamics of COVID-19 pandemic in India, Journal of Applied Mathematics and Computing (2021)”
Validity of the findings: For model validation and proper writing of the manuscript authors should include the manuscripts: "Modeling the dynamics of COVID-19 pandemic with implementation of intervention strategies, The European Physical Journal Plus (2022)"; "Mathematical modeling and optimal intervention strategies of the COVID-19 outbreak, Nonlinear Dynamics, (2022)".
Additional comments: No coemmnts.


Instruction: You are one of the reviewers, your task is to write a review for the article. You will be given the title of the article, the number of the round in which the article is located, and your order among the reviewers.

Input:
Title: MATHEMATICAL COVID-19 MODEL WITH VACCINATION: A CASE STUDY IN SAUDI ARABIA
Review round: 2
Reviewer: 1

Output:
Basic reporting: It is suitable for publish
Experimental design: It is suitable for publish
Validity of the findings: It is suitable for publish
Additional comments: It is suitable for publish


Instruction: You are one of the reviewers, your task is to write a review for the article. You will be given the title of the article, the number of the round in which the article is located, and your order among the reviewers.

Input:
Title: MATHEMATICAL COVID-19 MODEL WITH VACCINATION: A CASE STUDY IN SAUDI ARABIA
Review round: 2
Reviewer: 2

Output:
Basic reporting: Improved.
Experimental design: Improved.
Validity of the findings: Improved.
Additional comments: None


Instruction: You are one of the reviewers, your task is to write a review for the article. You will be given the title of the article, the number of the round in which the article is located, and your order among the reviewers.

Input:
Title: AN APPROACH TO FILL IN MISSING DATA FROM SATELLITE IMAGERY USING DATA-INTENSIVE COMPUTING AND DINEOF
Review round: 1
Reviewer: 1

Output:
Basic reporting: I support the publication of this work as the authors present a non-easy task through a creative approach. However, I suggest they follow the below suggestions before this research can be considered for publication.

I also noted that the quality of the figures provided is not the best and needs to be improved.
Experimental design: To improve the experimental design of filling missing (infrared/missing/clouds) satellite-data I suggested following a different approach. It relates to the use of non-missing data from model outputs and adding different percentages of clouds/missing data randomly. Better yet, this new approach can be developed on multiple time steps fields, thus you fully approach the strength of your method by following the existent nature of surface features.

Thus you will be able to develop stronger statistics (figure/table) to conclude how well/bad this new approach solves the filling of missing data while ensuring that your newly created data do not create false surface features and it is consistent with the original data.

It is also important that authors provide the original data and the methodology (Python/Matlab) scripts to recreate their analysis.
Validity of the findings: I believe the validity of the findings is questionable based on the current approach. See above for suggestions on a new approach to better evaluate statistics and skipping the creation of false features, which is conditional to some interpolation approaches.
Additional comments: I suggest you invest a little bit more time in this analysis and provide a robust analysis that we all in the satellite-data world will appreciate.


Instruction: You are one of the reviewers, your task is to write a review for the article. You will be given the title of the article, the number of the round in which the article is located, and your order among the reviewers.

Input:
Title: AN APPROACH TO FILL IN MISSING DATA FROM SATELLITE IMAGERY USING DATA-INTENSIVE COMPUTING AND DINEOF
Review round: 1
Reviewer: 2

Output:
Basic reporting: The article has a clear and unambiguous English. The English level is sufficient for an overall correct understanding of the text, taking into account that is not likely the mother tongue of any of the authors. However, I have highlighted (in my "General Comments") a few terms that should be replaced by more appropriate ones

Literature references are sufficient to illustrate the scope of the research and to give adequate background.

The structure of the article is correct and the authors share all the material needed for a thorough revision.

The objectives of the research and the results are clearly stated
Experimental design: The scope of the research fits in the aims and scope of the journal.

The authors propose a methodological improvement to well established gap-filling methods for ocean colour satellite data. The experimental design is correct and the obtained processor allows to apply the DINEOF program to high-resolution data through a segmentation procedure.

This is the main advance in the developed processor, together with a refinement in the merging procedure that, however, it is not clear to me how different is from the previous method, based on pixel averages (see my comment in the "General Comments" section).

The advance in knowledge is very small, but the results are of interest to the community of satellite data processing.
Validity of the findings: The results presented are insufficient to support the conclusions of the authors.
I think this is the weakest part of the article, that needs to be improved.
In my general comments below I suggest the authors different validation exercises that would serve to improve the soundness of the conclusions. In summary:

- For the assessment of the differences in the merging method, artificially removing a certain number of valid pixels in the original images is proposed. These pixels would be then used for validation

- Instead of the visual inspection in Figure 6, some statistical analysis is asked for
Additional comments: Below, my notes with comments and amendments requests throughout the text:

Abstract:
Line 19: Replace “capture” by “acquisition”
Line 23: Replace “For proof of concept” by “As a proof of concept”
Line 26: Replace “chlorophyll level” by “chlorophyll concentration”
Throughout the text, the term “acquisition” should be used preferably, instead of “capture”.

Introduction:
Overall, the introduction to the oceanographic background of the work needs a revision, preferably by an expert in the field, to correct several misunderstandings on the biological and physical basis of the phytoplankton role in Oceans.

Line 28: Replace “plant organisms” by “photosynthetic organisms”. Besides microscopic algae, Cyanobacteria are the main constituents of phytoplankton.
The references to the impact of phytoplankton are basically incorrect and should be removed or changed. The authors confound “weather” with “climate”. Moreover, the term “weather balance” is vague and meaningless.
Line 31: replace “biochemical cycle” by “biogeochemical cycle”
It is not clear how phytoplankton could provide information to predict impact of Climate change. The whole first paragraph needs to be revised.
Line 34: The most used abbreviation for Chlorophyll-a is “Chl-a”, instead of “Chlo-a”
Line 38: Replace “interaction” with “impact”. Chl-a interacts with light. Its interaction, in turn, determines the magnitude and shape of the reflectance spectra in water bodies.
Line 39: Replace “closed” by “close”
Line 42: replace “space” by “scene”
Line 53: What does “an overall temporal record” means?
Line 74: Again the confusion between “weather” and “climate”. The sentence in line 74 could be correct, because there are many meteorological satellites used in atmospheric modelling and weather forecasting. But the three references included are related to ocean color sensors and to the relationship of phytoplankton and climate

Methodology
Lines 89-90: The sentence has no sense: “importance of chlo-a in chlorophyll studies” ??
Line 103: What does “Python is a representative” means?
Line 110: Replace “passed by” by “passed over”
The formula for filling holes in the preprocessing step consists in calculating the average of the n nearest valid pixels (with data). Which is the difference in this calculation with the so-called “classical” method based on the average?
The adjustment applied to reduce the impact of differences in capture conditions is not clearly explained

Results
In the explanation of the merging procedure, it is stated that “the ortho-mosaic with most data related to chlorophyll is chosen and tagged as base-image”. However, in the example shown in the results, the VIIRS-SNPP image was taken as the base-image despite not being the one with more valid data. Can the authors explain the reasons for that choice?

Figure 5 shows that the classical method (GPT-SNAP) overestimates values with respect to the proposed method. But, why it is assumed that the proposed method is producing the correct values?
A good way of proving this is to produce artificial holes (by removing valid pixels with a simulated pattern) and then use the valid original pixels to test the performance of the merging procedure. Authors are encouraged to do this experiment.

Line 209: Replace “fussed” by “fused”
In the comparison described lines 209 to 215 (figure 6) it is very difficult to visually observe the differences in concentration claimed by the authors. A comparison of histograms and/or statistic test on differences would be more informative than the visual inspection.


Instruction: You are one of the reviewers, your task is to write a review for the article. You will be given the title of the article, the number of the round in which the article is located, and your order among the reviewers.

Input:
Title: AN APPROACH TO FILL IN MISSING DATA FROM SATELLITE IMAGERY USING DATA-INTENSIVE COMPUTING AND DINEOF
Review round: 1
Reviewer: 3

Output:
Basic reporting: The manuscript "An approach to fill in missing data from satellite imagery using data-intensive computing" by Rivera-Caicedo et al is well structured and the scientific investigation is clearly presented, including a basic context and motivation. However, the scoping is not clear and the methodology shows little originality.

The use of the DINEOF method for gap-filling is not new, and the literature contains other titles, including MODIS-based work.

Data Interpolating Empirical Orthogonal Functions (DINEOF): a tool for geophysical data analyses (2011)
Reconstruction of MODIS total suspended matter time series maps by DINEOF and validation with autonomous platform data (2011)
Analysis of gap-free chlorophyll-a data from MODIS in Arabian Sea, reconstructed using DINEOF (2018)
Exploratory Analysis of Urban Climate Using a Gap-Filled Landsat 8 Land Surface Temperature Data Set (2020)
Experimental design: Although the authors claim they introduce "a new approach for filling in missing data from satellite imagery", the level of originality is low. The DINEOF method is already well-known, and the authors present some references. For me it is not clear what is new, and the authors do not emphasize the novelty.
Validity of the findings: The validation of the findings is a very important part is such studies. I cannot find a distinct validation section, where the results are compared either with other methods or to measurements. This is a fundamental part which makes me have serios doubts about the quality of the work. The so-called new method is divided into modules, which in principle is good, but in fact only one module refers to data-gap filling, and the rest are basic image processing.
Additional comments: I strongly recommend to justify better your results, refer to more previous work and resubmit the manuscript.


Instruction: You are one of the reviewers, your task is to write a review for the article. You will be given the title of the article, the number of the round in which the article is located, and your order among the reviewers.

Input:
Title: AN APPROACH TO FILL IN MISSING DATA FROM SATELLITE IMAGERY USING DATA-INTENSIVE COMPUTING AND DINEOF
Review round: 1
Reviewer: 4

Output:
Basic reporting: The authors present an approach to fill in missing data from satellite
imagery using data-intensive computing. The approach was divided into three main modules.  The idea is interesting; however, the authors must improve several details before accepting the paper.

The authors must improve the approach implementation explanation. There are gaps in how it was implemented. As explained, it seems that they only divided the data into smaller sets to be processed.

Also, several plots must be explained in another way. For example, in figure 5, the comparison is not understood.

the authors must justify  parameters used in the implementation ( for example 3 x 3 window)

Table 1 is missing
Experimental design: The authors present interesting experiments and results.  However, they must find a metric to validate the results; a visual comparison is not enough.

This metric will allow comparing the proposed approach with other methods.

Also, they can report processing time a compare it with the DINEOF algorithm processing time.
Validity of the findings: no comment
Additional comments: The idea is a novelty; however, it needs to be clarified to understand the real impact and originality.


Instruction: You are one of the reviewers, your task is to write a review for the article. You will be given the title of the article, the number of the round in which the article is located, and your order among the reviewers.

Input:
Title: AN APPROACH TO FILL IN MISSING DATA FROM SATELLITE IMAGERY USING DATA-INTENSIVE COMPUTING AND DINEOF
Review round: 2
Reviewer: 1

Output:
Basic reporting: The authors present an approach to filling in missing data from satellite
imagery using data-intensive computing. The approach was divided into three main modules. The idea is interesting; however, the authors must improve several details before accepting the paper.

The authors must improve the approach implementation explanation a justify what is the main novelty of their method compared to the state of the art. As explained, it seems that they only divided the data into smaller sets to be processed and this is not enough.

Also, a computational complexity analysis must be added to understand and clarify the contribution of the approach.
Experimental design: Experimental design
The authors present interesting experiments and results. However, they must find a metric to validate the results; a visual comparison is not enough. This metric will allow comparing the proposed approach with other methods reported in the literature. for example the complexity of their approach.

Also, they can report processing time a compare it with the DINEOF algorithm processing time with previous work reported.
Validity of the findings: no comment
Additional comments: the authors must clarify the real impact and originality of the proposed method. Currently, this is not enough to accept the paper.


Instruction: You are one of the reviewers, your task is to write a review for the article. You will be given the title of the article, the number of the round in which the article is located, and your order among the reviewers.

Input:
Title: QUANTUM-EFFECTIVE EXACT MULTIPLE PATTERNS MATCHING ALGORITHMS FOR BIOLOGICAL SEQUENCES
Review round: 1
Reviewer: 1

Output:
Basic reporting: Avoid using abbreviations in abstract unless it’s a time complexity.
Please provide a table for all the abbreviations.
Please merge the Prior work and Important Findings and Related work section into one section.
Experimental design: Theorem 1 and 2 are defined twice. It should be defined once only in the proposed methodology section.
Validity of the findings: The Section- ‘Simulation detail and analysis with algorithms evaluation criteria’ should be added as a subsection within Results and Discussions.
Additional comments: No additional comments


Instruction: You are one of the reviewers, your task is to write a review for the article. You will be given the title of the article, the number of the round in which the article is located, and your order among the reviewers.

Input:
Title: QUANTUM-EFFECTIVE EXACT MULTIPLE PATTERNS MATCHING ALGORITHMS FOR BIOLOGICAL SEQUENCES
Review round: 1
Reviewer: 2

Output:
Basic reporting: The article claims to proposes an efficient quantum solutions for exact multiple pattern matching to process the biological sequences. It appears that a lot of effort has been made in writing the article for presenting the solution through complexity analysis of algorithm. Pl. refer to the attachment.
Experimental design: Experimental Design is weak due to limitation of current hardware technology so cannot be basis of a publication. Further, Grover's search can overshoot if the number of solutions are not known in advance, which is going to be the case when t exact pattern matches are not known, so how are you going to handle such cases while performing simulation. Ideally there should be table for depicting numerical values of variables like N, m, t etc. along with  constants C etc. and performances in time and space rather than equations. Further Table 11 and 12 should also report on the number or percentage of patterns correctly identified, incorrectly identified and incorrectly missed etc.
Validity of the findings: The article claims to proposes an efficient quantum solutions for exact multiple pattern matching to process the biological sequences. It appears that a lot of effort has been made in writing the article for presenting the solution through complexity analysis of algorithm. The article claims to find all
Additional comments: This work should be divided into two parts viz., theoretical and application part with simulations giving details on experimental accuracy of the proposed algorithm.


Instruction: You are one of the reviewers, your task is to write a review for the article. You will be given the title of the article, the number of the round in which the article is located, and your order among the reviewers.

Input:
Title: QUANTUM-EFFECTIVE EXACT MULTIPLE PATTERNS MATCHING ALGORITHMS FOR BIOLOGICAL SEQUENCES
Review round: 2
Reviewer: 1

Output:
Basic reporting: 1.All the previous concerns have been addressed except usage of abbreviations in abstract.
2. English should be corrected.Grammatical errors like "At present, there is no effective quantum solution exists to 26 process multiple patterns." should be corrected.
3.In "Motivation and contribution of work" a point is given as follows:
 Existing
Experimental design: 1.In Proof of Theorem 1 what does 't' stand for.
Validity of the findings: In the Results Section, observations are given Table wise, instead the findings that they emphasize should be given as headings and relevant Tables should be included under them.
Additional comments: No additional comments


Instruction: You are one of the reviewers, your task is to write a review for the article. You will be given the title of the article, the number of the round in which the article is located, and your order among the reviewers.

Input:
Title: QUANTUM-EFFECTIVE EXACT MULTIPLE PATTERNS MATCHING ALGORITHMS FOR BIOLOGICAL SEQUENCES
Review round: 3
Reviewer: 1

Output:
Basic reporting: All previous concerns have been addressed.
A table containing all abbreviations should be added.
Experimental design: The corrections have been done as per requirement.
Validity of the findings: The impact and novelty has been properly presented
Additional comments: No additional comments


Instruction: You are one of the reviewers, your task is to write a review for the article. You will be given the title of the article, the number of the round in which the article is located, and your order among the reviewers.

Input:
Title: FPGA-BASED SYSTOLIC DECONVOLUTION ARCHITECTURE FOR UPSAMPLING
Review round: 1
Reviewer: 1

Output:
Basic reporting: There seems to be an issue with citation through out the text, for example in line 59  "Lu et al. Lu et al. (2017)" should be "Lu et al. (2017)". This repeated referencing should be an easy fix by fixing the format file.

In line 128 " (i) Interpolation techniques" and line 129  "(iii) Transpose Convolution need references." citations are needed.
Experimental design: No comment.
Validity of the findings: No comment.
Additional comments: No additional comments


Instruction: You are one of the reviewers, your task is to write a review for the article. You will be given the title of the article, the number of the round in which the article is located, and your order among the reviewers.

Input:
Title: FPGA-BASED SYSTOLIC DECONVOLUTION ARCHITECTURE FOR UPSAMPLING
Review round: 1
Reviewer: 2

Output:
Basic reporting: This paper presents  a deconvolution accelerator that upsample n×n input to 2n×2n output by convolving with a k ×k kernel. The proposed architecture does not insert and pad zeros and thus removes
the redundant computations to achieve high resource efficiency with reduced number of multipliers
and adders.
The paper is well-written and easy to follow.
Adequate literature reviews has been discussed and background clearly explained.
The paper structured well along with formatted plots and figures.
The result section presents a good proof of the established hypothesis.
Experimental design: The article has articulated well around the defined research question.
The results are reproducible and well presented.
Validity of the findings: The results have been evaluated against benchmarks, the dataset has been explained and conclusions are stated clearly along with highlighted contributions.
Additional comments: No additional comments


Instruction: You are one of the reviewers, your task is to write a review for the article. You will be given the title of the article, the number of the round in which the article is located, and your order among the reviewers.

Input:
Title: FPGA-BASED SYSTOLIC DECONVOLUTION ARCHITECTURE FOR UPSAMPLING
Review round: 1
Reviewer: 3

Output:
Basic reporting: Abstract:
* Please clearly specify the precision used for operations like multiply/add etc in the proposed architecture.
* Please provide SNR reported by prior works as a point of reference. Further, please specify how the SNR translate to increased accuracy compared to prior work?
* Please use IEEE double precision instead of MATLAB double precision, if applicable.
* Compared to prior work, is the performance higher or lower?

Introduction:
* Line 26-29: It is unclear why the citation [Daoud and Bayoumi, 2019] is used here.
* Line 45-98:  Please fix the citations. There are two authors cited for the same work. Example Line 48-49.
* Line 59: Please clarify what is segmentation cost and how it is different from throughput. Is the segmentation cost the energy consumption or latency or something else?
* Line 73: Yazdanbaksh et al has been cited but not explained.
* Line 71: Only suitable for small kernels — why? Is it due to arithmetic/compute complexity or memory consumption.
* Further, the metric used by the paper for evaluating the error

The reported 3.6 GOps at 200 MHz would translate to just 18 Ops/cycle. Does the proposed architecture fully utilize the resources in the FPGA?

Please comment on how the proposed architecture would support neural networks where multiple different deconvolutions layers with different kernel sizes and input/output sizes are used.
Experimental design: The experiments focus on deconvolution layers of fixed sizes (for example, 256x256 output and 3x3 kernel). It is unclear if the evaluated sizes for inputs/output/kernels/strides are employed in real-world workloads like UNets.

The numbers in Table 4 and Table 5 are generated from three sets of images - randomized, camera man, and natural. In contrast, prior works report end-to-end neural network accuracy degradation on widely used neural networks and datasets. As such, it is difficult to fully understand how good/bad the reported SNR is.
To rectify this, please evaluate the impact on accuracy when using the proposed architecture for deconvolution/upsampling layers and a traditional CPU/GPU for the rest in widely used neural networks (like UNET) and datasets.

Please clearly report the utilization of resources for the proposed architecture. The reported 3.6 GOps and 0.135 GOps/DSP would translate to just 27 DSPs. In this case, the proposed architecture does not fully utilize all the DSPs available on the FPGA zc7020.
Validity of the findings: The comparison in Table 10 is misleading since the proposed architecture only supports deconvolution layers of a neural network, while the cited prior works (Di et al, 2020) report end-to-end performance and energy/power efficiency. Please clarify in the table, perhaps by adding a row stating whether the evaluated architectures support layers other than deconvolution.
Additional comments: No additional comments


Instruction: You are one of the reviewers, your task is to write a review for the article. You will be given the title of the article, the number of the round in which the article is located, and your order among the reviewers.

Input:
Title: INVESTIGATING THE IMPACT OF VULNERABILITY DATASETS ON DEEP LEARNING-BASED VULNERABILITY DETECTORS
Review round: 1
Reviewer: 1

Output:
Basic reporting: Paper summary:
This paper studies the impact of datasets' distribution on the performance of deep learning-based vulnerability detectors, Three different aspects of vulnerability datasets were investigated, including the dataset granularity (function level or slice level), similarity (inter-class similarity and intra-class similarity), and the code features, such as AvgCyclmotic, AveEssential. In the evaluation part, several datasets and deep learning models are evaluated according to the three aspects, and associated insights were proposed.

Advantages:
+ I think this paper has very good research insights on the deep learning model datasets, which does not draw much attention in previous literature. Meanwhile, the distribution of datasets is in fact very important to deep learning models' performance.

+ The overall experiment design is reasonable and clear.

+ The datasets used in this paper as well as the deep learning models are very new.

Weakness:
I have several concerns about this paper, as summarized below:

+ Overall, the writing of this paper requires improvement. The high-level structures of this paper are relatively clear, but there are flaws:
    1) The abstract needs to be more succinct, especially on the method description. Such as "The training set is used to train the DL-based vulnerability detector". Such statements do not need to be in the abstract.
    2) In the Experimental Result Section, I think it will be better to switch the order of Table 4 and Table 5, and their introduction. It is more intuitive to present the representation similarity of the datasets first before giving the corresponding evaluation results.
    3) In Table 2, I think it will be better to add a column to show the category of the dataset, such as synthesized dataset, manually modified, and open-source software. I understand such information can be found in RELATED WORK, doing so will make the table clear and release the unnecessary burden of readers.
    4) Typos such as "Dulnerability Datasets" IN Section EXPERIMENTAL SETUP Line 186.
    5) In the INTRODUCTION section, redundant information on "what's deep learning" is given. I think this part could also be more succinct since the knowledge should be already known to the journal readers.

+ Another concern is the motivation to select the three aspects (granularity, similarity, code features) is not clearly presented in the paper. More evidence and motivation should be added to show why these three factors are important for evaluations. In addition, the similarity and code features are in fact characteristics of the raw code datasets, while granularity can be considered as a kind of preprocessing. I think it will be better to discuss them separately.

+ The details of experimental implementation are not given. For example, I was expecting the introduction to experiment platforms, the libraries used, experiment data such as running time.

+ Still in the evaluation part, the sizes of the datasets are not given. Also, the parameters (m, n) mentioned in STEP I (Line 146) are not given and not evaluated.
Experimental design: As shown above.
Validity of the findings: As shown above.
Additional comments: no comment


Instruction: You are one of the reviewers, your task is to write a review for the article. You will be given the title of the article, the number of the round in which the article is located, and your order among the reviewers.

Input:
Title: INVESTIGATING THE IMPACT OF VULNERABILITY DATASETS ON DEEP LEARNING-BASED VULNERABILITY DETECTORS
Review round: 1
Reviewer: 2

Output:
Basic reporting: This paper analyzes the inner connection between DL-based vulnerability detectors and datasets. The paper mainly focuses on the following aspects: fine-grained samples, datasets with lower inner-class similarity, and datasets with higher inter-class similarity and lower intra-class similarity. Although many other points can be explored in this kind of research, this paper grants a good view of the relationship between dataset and detector schemes. Also, this is interesting research, giving guidelines to the related research.
Experimental design: This is the most important part of the paper since the author claims their contributions focus on the evaluation. More effort should be put into this part.
Validity of the findings: The author needs to put more effort into the evaluation parts, as mentioned above. Some suggestions:
1. using recall rate and an F score is good, but ROC curve, EER, and accuracy are also necessary to evaluate various schemes. Since this part is the paper's focus, authors are suggested to provide detailed evaluation results in the paper.

2. How did you divide the dataset? What is the training-testing ratio? This will also impact the detectors' performance.

3. What was information loss when you applied PCA to the dataset? and what is the energy and time consumption when you remove the PCA?

4. Instead of using intra- or extra-class similarity, authors are suggested to evaluate the entropy difference between data samples.
Additional comments: No additional comments


Instruction: You are one of the reviewers, your task is to write a review for the article. You will be given the title of the article, the number of the round in which the article is located, and your order among the reviewers.

Input:
Title: INVESTIGATING THE IMPACT OF VULNERABILITY DATASETS ON DEEP LEARNING-BASED VULNERABILITY DETECTORS
Review round: 1
Reviewer: 3

Output:
Basic reporting: 1- This paper has not reached to the acceptable level for publication because of lacks originality and novelty.
2- The spell-checks, grammatical and writing style errors of the paper must be improved.
Experimental design: No comment
Validity of the findings: No comment
Additional comments: 1- For readers to quickly catch the contribution in this work, it would be better to highlight major difficulties and challenges, and your original achievements to overcome them, in a clearer way in abstract and introduction.
2- Some references are too old and please add at least five references within the past one year for related work section.
3-There are unsatisfactory organization and writing in the "Conclusion" section. The authors must rewrite and reorganize this section contents.
4- The highlights are not stressed in the paper, and the innovation of the paper weakness.
5- There are some mistakes in the style of English writing in the text which are to be revised/corrected carefully.


Instruction: You are one of the reviewers, your task is to write a review for the article. You will be given the title of the article, the number of the round in which the article is located, and your order among the reviewers.

Input:
Title: INVESTIGATING THE IMPACT OF VULNERABILITY DATASETS ON DEEP LEARNING-BASED VULNERABILITY DETECTORS
Review round: 2
Reviewer: 1

Output:
Basic reporting: The authors solved the issues mentioned in the review comments well. This version looks good to me.
Experimental design: See above.
Validity of the findings: See above.
Additional comments: No additional comments


Instruction: You are one of the reviewers, your task is to write a review for the article. You will be given the title of the article, the number of the round in which the article is located, and your order among the reviewers.

Input:
Title: IRIMAGE: OPEN SOURCE SOFTWARE FOR PROCESSING IMAGES FROM INFRARED THERMAL CAMERAS
Review round: 1
Reviewer: 1

Output:
Basic reporting: This contribution is about processing of thermal images.
In general, this is a complex problem, users usually want to determine the temperature of the object, but thermal cameras measure the thermal radiation of the thin outer layer of the object.
The English is ok, but I am not native speaker, and I am not relevant to evaluate English.
Literature review is sufficient, the article structure is appropriate.
As I am first time as reviewer in this journal, it is a little bit difficult for me to find additional information.
Experimental design: The research should be supported by a physical overview. The reader appreciated that.
You write: the theoretical background of the algorithm used in IRimage and how it was implemented is included in this paper as Article S1
First time I downloaded only the main text.only and I  didn’t find more information about physical overview. I don’t find in this paper Article S1 (I find it later), why it is divided on more parts? Maybe it's complicated from the publisher
This article (main text) is basic introduction to your system (software) with case projects, but it seems so inappropriate to me as a scientific paper for journal.
 Only some sentences and reference to S1.are In material and methods, and so the article looks just like a guide to the software and usage examples.
However, this research is certainly interesting, usable, and valuable.
Validity of the findings: The article presents usable software for processing thermal images. Cheap thermal cameras only output to jpg and do not give raw data, this is an example of thermal cameras on a DJI Mavic drone, for example, which complicates scientific use.
Additional comments: General comments: this article summarizes the state of the art of using low-cost thermal cameras and gives an opportunity to process data from these instruments based on scientific research. Research and information are useful for practice. I recommend the article for publication. But I have a more fundamental note to the overall text: why is the theoretical part outside the article? I would like to at least welcome the extension to essential physical information and fundaments in the Materials and methods section.


Instruction: You are one of the reviewers, your task is to write a review for the article. You will be given the title of the article, the number of the round in which the article is located, and your order among the reviewers.

Input:
Title: IRIMAGE: OPEN SOURCE SOFTWARE FOR PROCESSING IMAGES FROM INFRARED THERMAL CAMERAS
Review round: 1
Reviewer: 2

Output:
Basic reporting: This paper describes the functionality of the open source software IRimage which can be used for processing raw thermal images taken with several cameras of one brand FLIR and can potentially be used for other brand cameras. Some example results are presented on 14 thermal images obtained from a (non-verified) public source of thermal images ( Wikimedia ). In addition, an example use case of monitoring the temperature of plant leaves is presented of well and non-well hydrated plants during 24 cycles where many parameters vary in time.
The paper is clearly written in professional English language.  The introduction give a good background in support of the need and usability for the software IRimage presented.
Experimental design: The author states that in this paper the software is validated against standard software. The reviewer does not agree that the method used can be considered as scientifically validation procedure using some sample images of a non-verified source and presenting the results in graphs at large scales (e.g. figure 2) were a difference of even 5 degree cannot be distinguished. In the example use case, too many variables have an influence on the measured/calculated temperature to prove the accuracy of the software. The details from the case presented distract from the purpose of this paper to present the IRimage software and could be a paper by itself.

For a true validation process, the thermal images should be taken with different cameras of the same object under controlled conditions. Not having all these camera available, the reviewer understand that this is a difficult task. However, with only 3 different cameras, the author would already present the validity of the software.

As example case, it would be more convincing to have only 1 or 2 variables for the temperature changes in time compared to a temperature reference e.g. a black body source temperature controlled by a current (temperature phantom). The case with the leaf temperature presented is nice as example for the many processing possibilities but cannot be claimed as a confirmation for accuracy.
Validity of the findings: If the author would adapt the text by stating some examples are presented of processing some random thermal images using IRimage it would be acceptable. Still the temperature comparison graphs like in figure 2 need to be presented zoomed in on the temperature scale of e.g. 10 degree so the accuracy within 0.1 degree can be distinguished.
Additional comments: The reviewer is convinced the software IRimage will be well received by the scientific community as a useful tool to process raw data from thermal cameras as long as this data is accessible. It would be great if this would also be used for small thermal imagers like the FLIR-One. This paper gives a good introduction in the potentials of the software. However, a scientific validation process is needed in which other researcher could contribute. The paper should be adapted not claiming the software has been validated by the examples presented. The example case could be presented with far less detail or replaced by a more simple case as suggested.


Instruction: You are one of the reviewers, your task is to write a review for the article. You will be given the title of the article, the number of the round in which the article is located, and your order among the reviewers.

Input:
Title: IRIMAGE: OPEN SOURCE SOFTWARE FOR PROCESSING IMAGES FROM INFRARED THERMAL CAMERAS
Review round: 2
Reviewer: 1

Output:
Basic reporting: The reply of the author and revisions in the manuscript are adequate and meet the expectations of the reviewer. The 'claim' that the software is validated have been removed and replaced by 'compared to existing software'.  It would still be preferred to have the software validated in a controlled setting with various thermal cameras (in the future paper). Now there is no claim of validation, the case presented is a nice illustration but still a more simple case would be preferred with less variables that can influence the temperature.
Experimental design: In this revision more theoretical background is given on the algorithm and parameters used to determine the object temperature. This greatly improves the scientific basis for the paper.
No additional comments compared to the first version
Validity of the findings: The presentation of the thermal images in figure 2 extended with a detailed section,  as suggested by the reviewer,  enables a better comparison (and convincing) between IRimage and FLIR Tools down to 0.1 degrees.
The example use case is presented in the results extensively and could be shortened e.g. leaving figure 6 out.
Additional comments: No additional comments


Instruction: You are one of the reviewers, your task is to write a review for the article. You will be given the title of the article, the number of the round in which the article is located, and your order among the reviewers.

Input:
Title: EVALUATION OF FEATURE PROJECTION TECHNIQUES IN OBJECT GRASP CLASSIFICATION USING ELECTROMYOGRAM SIGNALS FROM DIFFERENT LIMB POSITIONS
Review round: 1
Reviewer: 1

Output:
Basic reporting: The paper is concerned with feature projection methods applied to the domain of EMG signal processing. In particular, data from several sensors  is collected in order to classify grasps. Since the number of computed features may be large in such tasks, it is desired to reduce the dimensions before the application of classification algorithms. The authors test several methods for dimension reduction, some linear and some non-linear, with an emphasis on the non-linear technique named spectral regression extreme learning machine. This method was shown to have the best performance.

The level of English is good. The background and Introduction are adequate. Figures are of good quality.
Experimental design: The experimental designs is explained in detail, the presented figures are helpful. Some comments regarding the algorithmic implementation are detailed bellow.
Validity of the findings: Experimental results are adequate, and the findings are solid.
Additional comments: Overall, the topic is of interest, and the paper is solid, but I would say that the contribution mainly focuses on the question of comparison between dimension reduction techniques and a focus on the dataset. The fact that limb position changes in the dataset, is more challenging compared to some previous studies (still this seems like a not-very-hard classification task).

Comments:

Page 3, line 126 – This is more challenge  more challenging

Feature projection section:
•	It is mentioned that the number of coordinates in the projected space is the number of grasp types minus one. It may be the case that more dimensions are needed for a better separation. Did you try other numbers?

Page 6, line 191 and on:
•	The train-test separation was not clear to me. I assumed that the classification is not multi-subject, rather it is single subject (train-test belongs to one subject). Is that so?
•	Does the train data contain all of the recorded positions? Or are some 'new' positions left out for the test data?
•	It would be interesting to see the performance of the methods as the number of training samples is reduced.
•	Please explain in more detail how the projected coordinates in each method are extended to the test data. In PCA, new points are normalized and projected to the train model. How about tSNE? It is not trivial to extend these coordinates. Is the model built over with the test data?
•	Same question for SRELM.

Page 6 line 217: SRELM
•	Since this method is new and probably most readers are not familiar with it, it would be useful to add a more detailed explanations of its construction, maybe in a pseudo code manner (instead of sending the reader to past papers).

Results section
•	Please add an opening sentence before the sentence in line 250 (Page 8) that starts with "In Table 2…"


Instruction: You are one of the reviewers, your task is to write a review for the article. You will be given the title of the article, the number of the round in which the article is located, and your order among the reviewers.

Input:
Title: EVALUATION OF FEATURE PROJECTION TECHNIQUES IN OBJECT GRASP CLASSIFICATION USING ELECTROMYOGRAM SIGNALS FROM DIFFERENT LIMB POSITIONS
Review round: 1
Reviewer: 2

Output:
Basic reporting: 1.	Subheadings must be bold, followed by a period, and start a new paragraph e.g. 	Background. The background section text goes here...
2.	Figures are of poor resolution and clarity. They need to be revised with high resolution
Experimental design: 1. I suggest that you improve the description about % split of train and test set at lines 191- 192. Also give system specifications.
2. Feature extraction methods should be improved with more elaboration and illustrations.
3.	 More clarity on the Classifier parameter selection should be included
Validity of the findings: 1. No new hybrid algorithm/enhancement based on feature projection method is proposed.
2.	 No simulation/Pseudo-codes or algorithm is given.
3.	Novelty of work should be strongly emphasized by authors.
Additional comments: 1.	Intro & background to show context are well explained. Problem statement is well defined
2.	Literature well referenced & relevant
3. Rigorous investigation performed to a high technical & ethical standard.
4. Conclusions are well stated, linked to original research question & limited to supporting results


Instruction: You are one of the reviewers, your task is to write a review for the article. You will be given the title of the article, the number of the round in which the article is located, and your order among the reviewers.

Input:
Title: EVALUATION OF FEATURE PROJECTION TECHNIQUES IN OBJECT GRASP CLASSIFICATION USING ELECTROMYOGRAM SIGNALS FROM DIFFERENT LIMB POSITIONS
Review round: 1
Reviewer: 3

Output:
Basic reporting: No Comment
Experimental design: NO Comment
Validity of the findings: The authors have mentioned in their paper "In this study, the number of hidden nodes 1000 and alpha 1
223 were used. Both were determined by trial and error."
It would be better if the authors gave a range or reason for picking up a certain range and then coming to the experimental value.
Additional comments: 1. The paper is written well and results show a varied range of Classification accuracy.
 It would be interesting to comment on the reason for sometimes very low accuracy /high accuracy. Correlating the results with data distribution can help in this regard.
2. work with subjects needing prosthesis can also be a step for future work
3. The authors need to develop their own techniques / modify the existing for novelty in the paper.


Instruction: You are one of the reviewers, your task is to write a review for the article. You will be given the title of the article, the number of the round in which the article is located, and your order among the reviewers.

Input:
Title: EVALUATION OF FEATURE PROJECTION TECHNIQUES IN OBJECT GRASP CLASSIFICATION USING ELECTROMYOGRAM SIGNALS FROM DIFFERENT LIMB POSITIONS
Review round: 2
Reviewer: 1

Output:
Basic reporting: The paper was majorly improved.
All the point I raised were answered and addressed.
More experiments were preformed, especially these emphasize the influence of the train set size.
Additional text was added to explain various points that were not clear in the initial manuscript.
Experimental design: Adequate
Validity of the findings: Adequate
Additional comments: No additional comments


Instruction: You are one of the reviewers, your task is to write a review for the article. You will be given the title of the article, the number of the round in which the article is located, and your order among the reviewers.

Input:
Title: EVALUATION OF FEATURE PROJECTION TECHNIQUES IN OBJECT GRASP CLASSIFICATION USING ELECTROMYOGRAM SIGNALS FROM DIFFERENT LIMB POSITIONS
Review round: 2
Reviewer: 2

Output:
Basic reporting: No comment
Experimental design: No comment
Validity of the findings: No comment
Additional comments: Authors have revised the manuscript as per guidelines. The proposed algorithms are added in the paper. The result section is very well written


Instruction: You are one of the reviewers, your task is to write a review for the article. You will be given the title of the article, the number of the round in which the article is located, and your order among the reviewers.

Input:
Title: EVALUATION OF FEATURE PROJECTION TECHNIQUES IN OBJECT GRASP CLASSIFICATION USING ELECTROMYOGRAM SIGNALS FROM DIFFERENT LIMB POSITIONS
Review round: 2
Reviewer: 3

Output:
Basic reporting: no comment
Experimental design: no comment
Validity of the findings: no  comment
Additional comments: The paper includes all the suggestions given earlier. Clarity and Novelty aspects are covered in depth.


Instruction: You are one of the reviewers, your task is to write a review for the article. You will be given the title of the article, the number of the round in which the article is located, and your order among the reviewers.

Input:
Title: A SURVEY OF THE STATE OF THE PRACTICE FOR RESEARCH SOFTWARE IN THE UNITED STATES
Review round: 1
Reviewer: 1

Output:
Basic reporting: The text is well written, but contains a few syntax and grammar errors (lines 169, 178, 510, 661). In line 316 the sentence structure seems to be off for me. While I (as a non native speaker) cannot determine if it is wrong, it could be improved for better understanding.

The paper is very well structured and figures and tables are included in appropriate places to ease the perceiving of the data, but unfortunately the figures have some drawbacks.

Figure 1 is missing an explanation for the two sets of columns. Also, it is rather unreadable due to the subfigures being too small. Removing at least the vertical dotted lines might further increase readability.

Nearly all figures show absolutes. Especially with the difference regarding the absolute number of answers, I would prefer a presentation as relatives (percentages) to allow a better comparison of the answers in figures itself (i.e. figure 5 and 6) and between figures. My suggestion is to state the absolutes in the figure caption or the x-axis captions since it is an important information. In the text the authors already use mainly the percentages, referring to the absolutes in brackets.
Experimental design: In line 388 it is stated that the data was sanitized, but not explained in what way. A short description of this process would be helpful.
Validity of the findings: The results are US and UK centric, which is due to the nature of the survey distribution and focus and acknowledged by the authors. While I see no issue with this and most findings might be applicable for other regions, I would recommend to mention it earlier in the paper, perhaps even the abstract.
Additional comments: No additional comments


Instruction: You are one of the reviewers, your task is to write a review for the article. You will be given the title of the article, the number of the round in which the article is located, and your order among the reviewers.

Input:
Title: A SURVEY OF THE STATE OF THE PRACTICE FOR RESEARCH SOFTWARE IN THE UNITED STATES
Review round: 1
Reviewer: 2

Output:
Basic reporting: The basic reporting is done well.  I indicated on the attached pdf a few spots where the sentences are phrased awkwardly, or where there are some typos, but for the most part the English is quite good.

The literature review is particularly well done.

The figures were nicely done, except for a few points that I highlighted on the marked up pdf.

The paper is self-contained.  The results are given for the research questions.

The section with the title Discussion should probably have the section title changed to Conclusions, since the Discussion is really what came in the previous section.  Also, there isn't currently a Conclusions section.
Experimental design: The research questions are well defined, except for one where the wording is awkward.  The one with the awkward wording is highlighted in the marked up version.

The investigation is well performed and the ethical aspect is explained.  It is also nice to see that the full questions and data are released as open-source.  It is good to see authors practicing what they preach (or eating their own dog food).  :-)

Enough information is given to replicate the experimental design.
Validity of the findings: For the most part the findings seem reasonable.  However, there are a few questionable points, as highlighted in the marked up version.  A few specific comments include:

1. A bigger deal seems to be made of the differences in Figure 1 than the data justifies.  The ideal time and the spent time look similar in all cases.  For the cases where the paper claims the difference is noteworthy, it would be good to quantify the relative difference between the two measures.

2. A section on threats to validity is missing.  One threat to validity is the respondents not knowing the meaning of the terminology, such as regression testing, requirements, software carpentry, maintaining and sustaining, design, ....  Another threat is respondents self-reporting on their activities, such as what documentation they write.

3. There is no section for future work.  For instance, the respondents self-report on what documentation they write, but this documentation could be measured directly by mining existing software repositories.  Also, extended interviews could be held with some researchers/developers to get more qualitative data.

4. At line 643 it is stated that the most difficult aspects of software development are not technical.  However, this is misinterpreting the study question.  The respondents weren't asked what they found most difficult, but what aspects of software development are more difficult than they should be.  The respondents might feel like the technical aspects are difficult, but not more difficult than they should be.
Additional comments: The paper shows that domain data was collected, but this data does not appear to be used.  Did the responses differ between the different domains?  If this was not investigated, possibly because of insufficient data, this should be explicitly stated.


Instruction: You are one of the reviewers, your task is to write a review for the article. You will be given the title of the article, the number of the round in which the article is located, and your order among the reviewers.

Input:
Title: A SURVEY OF THE STATE OF THE PRACTICE FOR RESEARCH SOFTWARE IN THE UNITED STATES
Review round: 1
Reviewer: 3

Output:
Basic reporting: The paper is very well-written. The language used throughout is very clear and unambiguous and the research is very well motivated. The introduction is really great and motivates the problem well. The tables and figures are also clear and easy to understand. The background is extensive, however the section is a little unorganized. I appreciate the list of prior surveys and Table I is very helpful, however it would be useful to have more specific details about how these prior studies differ from the current work. Additionally, the research questions (more on this later) do not seem to fit in this section. They have some background from the prior work, but are not well motivated for the overall work. There should be more background included to motivate each RQ and explain why it's included or they should be placed in a separate section (i.e. with the Methods). Overall this research is exciting and provides a valuable contribution to the field.
Experimental design: The research is within the scope of the journal. The investigation and methods are rigorous and meet a technical and ethical standard. The main limitation are the research questions. There is a wide breadth between the RQs and, while the researchers show how these topics fit into the previous surveys, in my opinion the authors fail to motivate why these topics are important for this study and show how each RQ relates to each other and contributes to the overall goal of the work. Each RQ on its own seems very meaningful and relevant, however it is unclear how these connect to each other. To improve upon this, I would suggest adding to each RQ to explain why it is necessary, how it was selected, and how it fits into the current work. I think also combining the RQs with the specific parts of the survey mentioned in the Methods section would also be helpful. Additionally, the RQs lack background for some of specifics studied, especially for the SE-related questions. For example, requirements, design, and testing as software engineering processes, however other potential processes considered part of SE such as implementation, deployment, maintenance, etc. are missing in RQ1. Also the *best* practices (continuous integration, coding standards, arch/design, requirements, peer code review) have some missing options (i.e. pair programming, static analysis tools,) and would be interesting to know the details on how these were selected.
Validity of the findings: The work is novel and impactful for improving how research software is developed. The findings are valid and sound based on the methods and RQs. It would be beneficial to see statistical tests used to further analyze the results and provide their significance. The organization of the results could also be improved by providing a clear answer (i.e. highlighted) to each of the research questions based on the findings. The paper is also missing a threats to validity section to explain limitations of the work (i.e. the huge discrepancy between the respondent types in the results) and how the researchers mitigated these threats. In the conclusion, I would prefer more discussion about implications of these problems and solutions for researchers and developers to improve the state of research software development for each RQ topic, especially for improving the software engineering practices. Additionally, it would be interesting to note future opportunities for this work, such as mining GitHub repos to programmatically determine the tools and practices used for research software. Overall this is a very interesting research paper with implications for improving research software, development practices, and research overall. Looking forward to seeing what's next!
Additional comments: No additional comments