Instruction: You are one of the reviewers, your task is to write a review for the article. You will be given the title of the article, the number of the round in which the article is located, and your order among the reviewers.

Input:
Title: IMPROVING PATIENT REHABILITATION PERFORMANCE IN EXERCISE GAMES USING COLLABORATIVE FILTERING APPROACH
Review round: 1
Reviewer: 2

Output:
Basic reporting: Language needs proof-reading. A professional, native English-speaking proof-reader is required. There are too many language issues to single out, but this proof-reading is needed.

Conceptual clarity needs improving: consistently refer to either patients or users. For example, "to suggest the most appropriate setting for patients to enhance the users' performance" ==> is the user the same as patient?
Experimental design: The explanation for the Bacterial Foraging Optimisation Algorithm needs to be clarified. While the other two algorithms are commonly used for recommenders, this algorithm is not.

Specifically,
(a) why was it selected?
(b) how was it applied --- the current explanation is describing micro-biology, but the use case here was exergame. The explanation needs to reflect the use case.

The authors allude to these points in later sections of the paper, but they need to be first explained when introducing the algorithm.

The number assignment of clusters needs to be clarified:
"MIRA data are tested in a set of k clusters ranging between 5 and 10 clusters"
>> why this test range?

The same for kNN:
"In this experimental work, the k of neighbours is determined based on a set of k (25 neighbours, 50 neighbours, 75 neighbours, 100 neighbours)."
>> why this range?

The inability of normalization to help overfitting needs to be explained:
"prediction values are over-fitting; hence they need to be normalised to fit the prediction values. Nonetheless, the normalisation method is inapt for this process."
>> why inapt?
>> is there a reference applying BFOA in a similar task before? If so, please include.
Validity of the findings: The application of RMSE needs clarification - the authors state, correctly, that the metric is applied "for calculating the differences between the predicted values and the values observed." But what were these values in the current study? Combination of different variables? Which variables?

The study found that the method that combined three techniques was the best. However, this is typically the case for any machine learning situation - ensemble methods outperform single-method approaches. So, the authors need to clarify if there is anything unexpected in this finding, and what it means for future work - could different ensemble techniques be tested against each other?
Additional comments: Discussion should be enhanced with an evaluation of practical implementability of the tested methods. While the most complex method yielded the best performance, is it feasible to implement? If so, under which conditions (and under which other methods could be better).

Discussion should be enhanced by discussing the weaknesses of the current work. What are they? Honestly explicate how the research could be improved.

Discussion should be enhanced by adding directions for future research. What are the next steps? What other methods could be tested? How "good" are the results in terms of practical implementation and how far are they from yielding a substantial positive impact to patients' recovery?


Instruction: You are one of the reviewers, your task is to write a review for the article. You will be given the title of the article, the number of the round in which the article is located, and your order among the reviewers.

Input:
Title: IMPROVING PATIENT REHABILITATION PERFORMANCE IN EXERCISE GAMES USING COLLABORATIVE FILTERING APPROACH
Review round: 3
Reviewer: 1

Output:
Basic reporting: English is far much better compared to the the first version of the paper (last time there were some english mistakes that made difficult to follow the entire  - easier to read now.
The abstract is clear: it does specify the background, the need, the purpose of the research, methods and results.
The introduction section is comprehensive of background research on rehabilitation tools such as exergames.
The article structure is fine.
Definitions of concepts is included .
Experimental design: Research question well defined, relevant & meaningful:
The abstract is clear: it does specify the background, the need, the purpose of the research, methods and results.
The introduction also states clearly what are the research contribution by the authors compared to available literature.
The MIRA platform is described, including methods used, datasets, and a discussion of all the approaches (ReComS, ReComS+, ReComS++).

Methods described with sufficient detail: Recommender Systems benchmarks has also been investigated and applied to get an estimation of the recommendation accuracy.
Validity of the findings: According to what has been states in the introduction, the paper seems to be novel compared to what's available in the literature regarding exergames.

The conclusion looks good, I would just suggest to make it longer mentioning again the strength of the proposed approach and the lack of similar works in literature.

The provided Raw data can be opened. Might be good to add a README file in the Raw data folder to facilitate the understanding of what each script does.
Additional comments: I'm going to accept the paper.
My only notes are that the conclusion could have been stronger, and that a README file in the raw data package could have been included, explaining what every script does and how to run the entire recommender system (I can see csv files, consent forms, scripts, but no README).


Instruction: You are one of the reviewers, your task is to write a review for the article. You will be given the title of the article, the number of the round in which the article is located, and your order among the reviewers.

Input:
Title: A COMPREHENSIVE REVIEW OF DEEP LEARNING-BASED SINGLE IMAGE SUPER-RESOLUTION
Review round: 1
Reviewer: 1

Output:
Basic reporting: STRONG ASPECTS
   -Concise and readable abstract.
   -Adequately mentions the contribution.

WEAK ASPECTS
   -Ambiguity in readability and connectivity because of lots of sub-sections
   -Figure 1 should mention the color-coding.
Experimental design: STRONG ASPECTS
   -Categorization of SR methods seems to add value in the research domain
   -Detailed tables and figures for defining the inclusion and exclusion criteria and datasets

WEAK ASPECTS
   -Research could have considered  more paired LR-HR image datasets described in the table
    3
   -Survey Methodology section 3, mentions the study covers papers from 2008-2021
     whereas the next section named Conventional Methods for Super-Resolution i.e.
    section 4 refers to studies and methods from 1971 and onwards.
Validity of the findings: STRONG ASPECTS
   - The research states all the learning processes used in ML paradigm which makes it
      highly recommended for the different novel ML scenarios
   - Up-to-date literature and methodologies presented which would be relevant over years

WEAK ASPECTS
   - Figure 11 should be discussed in detail.
Additional comments: A thorough review of all the existing methodologies representing their mathematical basis. Goodluck!


Instruction: You are one of the reviewers, your task is to write a review for the article. You will be given the title of the article, the number of the round in which the article is located, and your order among the reviewers.

Input:
Title: A COMPREHENSIVE REVIEW OF DEEP LEARNING-BASED SINGLE IMAGE SUPER-RESOLUTION
Review round: 1
Reviewer: 2

Output:
Basic reporting: Although the writing in this paper is good, where it is difficult to find grammatical errors or typos, the mathematical symbols used need to be improved.  Three main comments on mathematical symbols in this paper:
1) Some of the symbols are strange, mostly because the original font style is not embedded into the pdf file.
2) The symbols used in the equation, should be exactly the same with the one use in the text. For example, in equation (5), symbol L is used, but in text (i.e., on line 137), different symbol is been used.
3) Mathematical symbols in the text should be written in Italic.

The title of the paper limits the scope of the review to only for single image super-resolution methods.  However, this paper failed to introduce the background of single image super-resolution, such as what is its definition, and how it differs from super-resolution from multiple images or video input.  Thus, the author should give some background in this aspect in Section 1.

The paper is well organized, with related and nice figures provided.  Table 5 has been provided to compare the structure of the methods.  Yet, in my opinion, it would be much better if the authors could also provide the tables to summarize each section of this review.  For example, at the end of Section 2.2., a summary section is provided to compare the quality measures for super-resolution.  Then, in this section, better to provide a table to compare those quality measures, such as what are the advantages and disadvantages of PSNR, MSE, SSIM, etc.

The review not only covers a broad approaches for deep-learning based super-resolution methods, but it also reviews a broad possible applications for super-resolution.  In my opinion, this will attract the attention and interest of the readers.

In line 66 to 75, the authors mention that there is no literature review on super-resolution using deep-learning approaches.  However, this is not true.  There are already many available literature review in this field, for examples:
1) W. Yang, X. Zhang, Y. Tian, W. Wang, J. Xue and Q. Liao, "Deep Learning for Single Image Super-Resolution: A Brief Review," in IEEE Transactions on Multimedia, vol. 21, no. 12, pp. 3106-3121, Dec. 2019, doi: 10.1109/TMM.2019.2919431.
2) L. Zhou and S. Feng, "A Review of Deep Learning for Single Image Super-Resolution," 2019 International Conference on Intelligent Informatics and Biomedical Sciences (ICIIBMS), 2019, pp. 139-142, doi: 10.1109/ICIIBMS46890.2019.8991477.
3) Ooi, Y.K.; Ibrahim, H. Deep Learning Algorithms for Single Image Super-Resolution: A Systematic Review. Electronics 2021, 10, 867. https://doi.org/10.3390/electronics10070867
4)  Viet Khanh Ha, Jin-Chang Ren, Xin-Ying Xu, Sophia Zhao, Gang Xie, Valentin Masero and Amir Hussain. Deep Learning Based Single Image Super-resolution: A Survey. International Journal of Automation and Computing, vol. 16, no. 4, pp. 413-426, 2019 doi:  10.1007/s11633-019-1183-x
5) Zhang, H.; Wang, P.; Zhang, C.; Jiang, Z. A Comparable Study of CNN-Based Single Image Super-Resolution for Space-Based Imaging Sensors. Sensors 2019, 19, 3234. https://doi.org/10.3390/s19143234
Therefore, it is better if the authors could highlight how this review paper differs from the other available reviews.
Experimental design: This review paper is within the scope of PeerJ Computer Science.  The review is organized.

Section 3 presents the survey methodology.  The terms used for the search are listed in line 343 to 344, and Table 4 presents the inclusion and exclusion criteria.  However, based on these descriptions, there is no criteria set for "single-image" for super-resolution.  Thus, the results from the search may not limited for single-image approaches.
Validity of the findings: As stated in Part 1, it would be better if the authors could provide a summary at the end of each section, and provide a table to compare the reviewed approaches in terms of their advantages and disadvantages.
Additional comments: In addition to the comments in Parts 1-3, the following are the minor concerns regarding to the paper:

1) In line 36, the author mention that "The image-based computer graphics models lack of resolution independence".  However, this statement is not true.  Computer graphic is not necessarily  in raster format (i.e., bitmap format), but may also in generated as a vector graphic.  The vector graphic is resolution independent.

2) In line 48, the author mention "shows the importance of this topic in recent years".  However, for the surveillance (in line 47), the references are from year 2003 and 2010, which are not recent.  Better to provide newer references.

3) In line 91, the sentence is referring to Figure 1.  However, no elaboration on Figure 1 is provided.  Please give some elaboration on Figure 1 in this paragraph.

4) In line 399, "explored in Section 3".  It is a wrong section.  Please provide the correct section.

5) Why the structures for "Attention-based learning" (section 5.2.6) and "Wavelet transform based learning" (section 5.2.7) are not included in Figure 6?

6) In line 808, "used after the removal of batch normalization used the additional memory to design a large model" is confusing.  Better to rephrase.


Instruction: You are one of the reviewers, your task is to write a review for the article. You will be given the title of the article, the number of the round in which the article is located, and your order among the reviewers.

Input:
Title: A COMPREHENSIVE REVIEW OF DEEP LEARNING-BASED SINGLE IMAGE SUPER-RESOLUTION
Review round: 1
Reviewer: 3

Output:
Basic reporting: No comment.
Experimental design: No comment.
Validity of the findings: No comment.
Additional comments: In this study, Bashir et al. conducted a literature review on deep learning-based single image super-resolution. In my opinion, the review is comprehensive and fits the scope of the journal. However, it is a well-known study and some previous studies have reviewed it before i.e., https://doi.org/10.1007/s11633-019-1183-x, or https://doi.org/10.1007/s11554-019-00925-3, ... Therefore, the idea to have one more survey is not necessary maybe, and then the contribution of the study is limited.

The authors should explain more on the keyword query and collected data.

Why did the authors need to query ScienceDirect and IEEE Xplore since Google Scholar already covers these two?

There still have grammatical errors and typos. For example, even in Fig. 3: "Scient Direct".

Deep learning is well-known and thus, in its description, the authors should mention more deep learning-based studies in different fields such as PMID: 33816830 and PMID: 31750297.

What is "initial screening"? The authors should describe this part clearly.


Instruction: You are one of the reviewers, your task is to write a review for the article. You will be given the title of the article, the number of the round in which the article is located, and your order among the reviewers.

Input:
Title: A COMPREHENSIVE REVIEW OF DEEP LEARNING-BASED SINGLE IMAGE SUPER-RESOLUTION
Review round: 1
Reviewer: 4

Output:
Basic reporting: The review paper was comprehensive and specific to the chosen area of review
Experimental design: Not applicable. This is a review paper and its study design is limited in scope to reviewing articles in the given field.  The given review article meets the aims and scope of the journal. This review includes sufficient details and information needed for a
Validity of the findings: The review paper summarizes the recent researches and their significance in the Image processing field. Its a needed review compilation and the first of its kind in the niche area
Additional comments: No additional comments


Instruction: You are one of the reviewers, your task is to write a review for the article. You will be given the title of the article, the number of the round in which the article is located, and your order among the reviewers.

Input:
Title: A COMPREHENSIVE REVIEW OF DEEP LEARNING-BASED SINGLE IMAGE SUPER-RESOLUTION
Review round: 2
Reviewer: 1

Output:
Basic reporting: STRONG ASPECTS
- OK

WEAK ASPECTS
-Authors have not highlighted the significance of their study as compared to previous similar studies in the literature
- References need to be updated.
Experimental design: STRONG ASPECTS
-Categorization of SR methods seems to add value in the research domain
-Detailed tables and figures for defining the inclusion and exclusion criteria and datasets
Validity of the findings: STRONG ASPECTS
- The research states all the learning processes used in the ML paradigm
- Up-to-date literature and methodologies presented which would be relevant over years

WEAK ASPECTS
- Similar studies are available in the literature. The reader can not determine what additional value to the body of existing knowledge is increased by this research.
Additional comments: No additional comments


Instruction: You are one of the reviewers, your task is to write a review for the article. You will be given the title of the article, the number of the round in which the article is located, and your order among the reviewers.

Input:
Title: A COMPREHENSIVE REVIEW OF DEEP LEARNING-BASED SINGLE IMAGE SUPER-RESOLUTION
Review round: 2
Reviewer: 2

Output:
Basic reporting: No comment
Experimental design: No comment
Validity of the findings: No comment
Additional comments: I am satisfied with the improvements done by the authors.


Instruction: You are one of the reviewers, your task is to write a review for the article. You will be given the title of the article, the number of the round in which the article is located, and your order among the reviewers.

Input:
Title: A COMPREHENSIVE REVIEW OF DEEP LEARNING-BASED SINGLE IMAGE SUPER-RESOLUTION
Review round: 2
Reviewer: 3

Output:
Basic reporting: No comment.
Experimental design: No comment.
Validity of the findings: No comment.
Additional comments: Thanks for addressing my previous comments. There are some comments that still need room for improvement:
1. How did the authors remove the duplicate samples among ScienceDirect, IEEE Xplore, and Google Scholar since they must have overlap?
2. I suggested the authors cite more works related to deep learning-based medical works i.e., PMID: 33816830 and PMID: 31750297, but they skipped it.
3. Quality of figures should be improved.


Instruction: You are one of the reviewers, your task is to write a review for the article. You will be given the title of the article, the number of the round in which the article is located, and your order among the reviewers.

Input:
Title: A COMPREHENSIVE REVIEW OF DEEP LEARNING-BASED SINGLE IMAGE SUPER-RESOLUTION
Review round: 2
Reviewer: 4

Output:
Basic reporting: The authors have diligently worked on the comments the reviewers had on the reporting structure in their new revision
Experimental design: This review work is in compliance with the aims and scope of our esteemed journal
Validity of the findings: This review paper encompasses the gaps which were discussed with the authors and they have come back with the mentioned corrections
Additional comments: Appreciate your nimbleness in doing the needed changes and getting back this fast


Instruction: You are one of the reviewers, your task is to write a review for the article. You will be given the title of the article, the number of the round in which the article is located, and your order among the reviewers.

Input:
Title: DISCOVERING GENERATIVE MODELS FROM EVENT LOGS: DATA-DRIVEN SIMULATION VS DEEP LEARNING
Review round: 1
Reviewer: 1

Output:
Basic reporting: The article presents an empirical comparison of two approaches for process discovery/predictive monitoring. The first approach is based on simulation mining: a detailed simulation process model (e.g., a Petri net) is discovered from the data. Then, the model is enriched and used for the analysis. The second approach is based on the deep learning - a generative neural network is trained from the data. Both models are essentially trace generators (including rich features that can be generated into the future). The empirical evaluation is very nice and the topic is worth investigating. The paper is very relevant for practitioners and scientists who are working in the area of process mining. One of the novelties of the approach is the realizations that the two representations (simulation and deep learning) have equivalent usages.

What is missing in my opinion is a comparison of their expressive power. Given that the same event log is used to create both - do these two models actually capture the same amount of information? Are they equal in their modeling strength?

I am missing a model-to-model comparison to complement the nice empirical evaluation.

In terms of the standards: the paper is well-written, the literature is covered in a sufficient manner, the paper is structured professionally, is self-contained, and the results are presented in a satisfying manner.
Experimental design: The experimental results are well-aligned with the journal's Aims and Scope. The research questions are well defined, and the paper does fill in a gap in the literature. It is the first paper that considers this equivalence between two generative families of models. I would also like to see a more rigorous comparison of the two models: how are the building blocks compare to each other? are the models equivalent? is one more expressive than the other?

Otherwise, I have no complaints regarding the empirical evaluation.
Validity of the findings: The strengthen the validity of the results, the authors must compare between the two models- this must at least be a discussion, if not a formal comparison between the two languages that the models can generate, and the random variables that underlie the two models.

Conclusions and limitations are well-stated and the empirical evaluation is described in a valid and satisfying manner.
Additional comments: For me to vote accept you must provide a discussion or a formal section (depending on the depth and the comparability of the two approaches) that would satisfy the gap in expressiveness. I have the intuition that the comparison will be interesting and useful for the scientific community. Furthermore, that would really strengthen the novelty and the rigor of the paper.


Instruction: You are one of the reviewers, your task is to write a review for the article. You will be given the title of the article, the number of the round in which the article is located, and your order among the reviewers.

Input:
Title: DISCOVERING GENERATIVE MODELS FROM EVENT LOGS: DATA-DRIVEN SIMULATION VS DEEP LEARNING
Review round: 1
Reviewer: 2

Output:
Basic reporting: The language used is clear and mostly unambiguous. However, DDS/SIMOD and DL/LSTM/GRU are used in an interchangeable way. DL and DDS  have a higher level of abstraction than SIMOD/LSTM/GRU, whereas in a number of situations - which I will later report in detail in my minor concerns - the authors use them interchangeably.
The work is structured in a reasonable way.
The figures are of high quality and the captions are accurate. I would suggest some minor adjustments to some captions and tables which I will report in my minor concerns.
The raw data are supplied and available.
Experimental design: The scope of the work is within the scope of the journal.
The research questions are well stated and meaningful.
The methods are described with high detail but some meta-parameters for the tuning of hyperparameters could benefit a better description/motivation.
Validity of the findings: The work impact for the field is important and the novelty is well assessed.
The statistical significance of the results could not be assessed due to a motivated shortage in the amount of datasets available in the literature. The results are sound, but the control should be improved due to a not well motivated usage of meta-parameters for the the tuning of hyperparameters.
The conclusions are sound and coherent with the rest of the work.
Additional comments: In the work “Discovering generative models from event logs: data-driven simulation vs deep learning” the authors aim to empirically compare Data-Driven Simulation (DDS) and Deep Learning (DL) approaches for the discovery of generative model from event logs.
The DDS approach of choice is SIMOD and the DL approach of choice is DeepGenerator. In turn Deep Generator uses two different architectures, one is exploiting LSTMs and one GRUs.
The three approaches are compared on 5 different datasets, one of which is synthetic, and four are real-life datasets.
Both DDS and DL approaches are tuned automatically exploiting hyperparameter optimisation techniques.
The discovered generative model is capable of generating a new dataset that resembles the original one.
The quality of the retrieved new dataset and therefore of the generative model is evaluated through 4 different metrics.
The results suggest that DDS approaches are suitable for capturing the sequence of activities/categorical attributes of a process. Whereas DL approaches are suitable for predicting the timing of the activities. These results raise the prospect of combining the two approaches into hybrid techniques.

I think that the paper needs some refinements but it is interesting and timely for the field.
The work is well written and pretty clear, nonetheless it could benefit of a number of adjustments.

The major concerns I have regarding your work are the following ones:
1. The methodologies you evaluate in your work are limited to specific implementations of one DDS and two DL approaches. In the threats of validity you claim that the other state-of-the-art DDS methodologies require manual intervention to be tuned, and that the other available DL methodologies are left as future work.
While I am fine with leaving some of the DL methods as future work and I believe that SIMOD is an extremely innovative approach, I would ask you that:
   a. either you better motivate why you only use (these) three methodologies (and this should be clear from the beginning), while in your title and introduction you claim to be comparing the DDS and DL families of approaches for the discovery of generative models;
   b. or you assess at least another DDS method in order to provide more general results.
2. The meta parameters used to optimise the hyper parameters of your approach, i.e., the number of configurations and the runs carried out for each configuration, are not well stated/motivated. For the DDS method you state to explore 50 random configurations with 5 runs each.
It seems to be a big amount of models to explore, how did you come up with these two parameters (i.e. 50 and 5)?
On the other hand, for the DL method you state you use random search to determine the hyperparameters.
Are there any meta parameters for the random search? If so, can you report and motivate them (e.g. the maximum amount of runs, the maximum time elapsed or the convergence of the retrieved models)?
3. Roles are a very peculiar and interesting sort of data to be used in process mining. While you motivated why you do not use them in Section 3, you do not mention them in Section 2. How would the investigated techniques behave with the generation of event logs not only with timestamps but also with attributes as roles?
I would ask:
   a. either you clarify why you do not take into account the generation of event logs enriched with roles (e.g., SIMOD does not support the generation of roles)
   b. or experiment and compare the techniques of the two families of approaches for the generation of event logs with timestamps and roles.

Other than that, there is a number of minor concerns - mostly regarding the coherence of the presentation of your work - which I would appreciate if you could address:
- Line 96: would it be possible to introduce why you use SIMOD, like discussed in the threats to validity at Line 326
- Figure 1: the caption reports SIMOD instead of DDS, whereas in Figure 2 the label reports DL instead of LSTM and GRU, I would prefer to see the two label aligned
- Figure 4: in the label “SIMOD parameter extraction” could you align to the other label by writing it as “DDS parameter extraction”?
- Table 2 could you rewrite DDS as SIMOD, as you did for DL, reporting them as GRU and LSTM?
- Line 57: would it be possible to report a measure of difference between the datasets? e.g., using ELS between the datasets
- Line 187: would it be possible to provide more details on the reason to use DeepGenerator?
- Line 195: What do you mean by: “DeepGenerator selects among these possible events randomly, taking into account the associated probabilities”? Is it a random or does it use probabilities?
- Line 62: The title “Data-driven simulation” is not in line with the title “Generative Deep Learning Models of Business Processes”
I suggest either to rewrite the latter as “Deep Learning models” as they are presented at line 42 or the former in a more specific manner
- Line 94: “i.e..” -> “i.e.,”
- Lines 124 to 128 are a bit convoluted. I would prefer to see it rephrased, in particular, the concept of hyperparameter optimisation is not new to the audience of this venue, therefore you could rephrase it by simply stating that: “in the post processing, SIMOD uses an hyperparameter optimisation technique to discover the SIMOD configuration maximising the ELS between the produced log and the ground truth”
- Line 191: Can you give a reference for the hallucination methodology?
- Figure 3: I would prefer to see a little incipit to the caption enumeration, instead of starting with the enumeration, I would prefer to see something like: “Explored network architectures: a) [..]”
- Table 1: why is the format of the last two columns different? They are both reporting duration, either use the hour/minute specification or the floating hour
- Table 1: Why are you using both average and mean in the columns “avg. activities per trace” and “mean duration”. Aren’t Average and Mean the same thing in statistics?
- Lines 222 and 223: Are these metrics a contribution of the work? if not, could you give a reference for them?
- Section 4.2 could you make bold the metrics in the respective paragraphs where you explain them? Like you did in section 2 and 3 with the various processing phases
- Line 260: could you give reference to the Hungarian algorithm?
- Line 288: what do you mean by predictable behaviour? Can you support this claim by some sort of metrics?
- Line 344 “Francescomarino” -> “Di Francescomarino”
- Line 351 “Rosa” -> “La Rosa”
- Line 364 “Francescomarino” -> “Di Francescomarino”
- Lines 370 and 374 incomplete reference missing publisher or at least venue
- Line 393 “Rosa” -> “La Rosa”


Instruction: You are one of the reviewers, your task is to write a review for the article. You will be given the title of the article, the number of the round in which the article is located, and your order among the reviewers.

Input:
Title: DISCOVERING GENERATIVE MODELS FROM EVENT LOGS: DATA-DRIVEN SIMULATION VS DEEP LEARNING
Review round: 1
Reviewer: 3

Output:
Basic reporting: The paper is well organized; in fact, despite its structure deviates from the standard PeerJ template, I believe that this allows it to gain in readability.
The language used in it is professional, clear and unambiguous.

The figures and tables are readable and clear enough (see my detailed comments for some advice on how to improve Figure 1).

The paper is self contained, and it provides sufficient background information for understanding the proposal.

Relevant related work in the literature is covered, which allows for positioning the proposal adequately.

The auxiliary material made available for reproducibility purposes (raw data, code and results) can be accessed easily and opened correctly, complies with the journal's Data Sharing policy, and it looks well documented and explained.
Experimental design: The paper does not offer novel contributions from a technical/methodological viewpoint (e.g., in terms of novel algorithms, methodologies, quality metrics).

However, I do believe that it fits well the Aims and Scope of the journal, for it provides an nice empirical analysis of two alterative approaches to process trace generation/simulation (based on the construction of model-based process simulation models and of DL-based generative models, respectively).
Such an important comparative study is indeed lacking in the field.
The paper identifies this gap in the literature and clearly states the goal of filling it through a well defined experimentation. In fact, some of the experimental findings presented in the paper look pretty novel and interesting.

The data, methods and evaluation procedures adopted in the study are described in a complete enough way, but the addition of certain technical details could help improve the paper (see Section "General comments" for detailed comments on this respect).

Unfortunately, however, the scope of the study is somewhat limited, in terms of both the techniques analyzed and the datasets employed. See section "General comments" for a detailed discussion of this limitation and of possible ways to reduce it.
Validity of the findings: The experiments are fully replicable.

The testing procedure is correct, but not complete enough in my opinion, due to the lack of "recall-oriented" evaluation criteria (see section "General comments" for details on this point).

The discussion of the test results is articulated and clear, but it needs to be improved. In particular, I am afraid that certain conclusions are not fully supported by the results obtained (see section "General comments" for details).
Additional comments: *** PAPER SUMMARY
The paper presents a comparative empirical study of two alternative existing approaches to the (data-driven) generation of business process traces:
(A) building up a Data-Driven Simulation (DDS) model, and
(B) training a Deep Learning (DL) model for next-event prediction.
Specifically, for constructing DDS models the Authors adopt the pipeline procedure and the SIMOD toolset published in (Camargo et al., 2020), while they induce DL models by training different RNN-based architectures proposed in (Camargo et al., 2019) --more precisely, some variants of these architectures devised to generate both the start time and end time of the next activity, but not the role of their associated executors as in (Camargo et al., 2019).
The results of experiments performed on five public logs are presented, while using a number of ad hoc measures to evaluate the similarity between the traces generated and real ones.
The relative strengths of the two approaches are discussed, and the opportunity of developing hybrid approaches, capable of combine these strengths, is prospected.

*** EVALUATION
I appreciate the idea of conducting a comparative empirical analysis of model-based process simulation techniques and DL-based generative models, relatively to their capability to generate good-quality process traces.
Such an important kind of empirical study is indeed lacking in the field.
The paper is well organized and is easy to read, and its figures and tables are readable and clear enough.
The paper contains sufficient background information and references to relevant prior literature.
Auxiliary material (raw data, code and results) is of good quality and it can be opened correctly.

However, I am afraid that the quality of the work is undermined by a number of flaws, discussed below, which would need to be addressed appropriately.

1) Limited range of DL-based generative models analysed.
The induction of DL-based generative models for predicting/simulating process behaviors is a topical problem, for which there were recently proposed solutions, like those in (Lin et al., 2019) and (Taymouri et al., 2020).
In fact, you have mentioned these works, but excluded them all from the experimentation because of their "inability to predict timestamps and durations".

However, you might well decide to partially evaluate these methods by only using the CFLS metrics, provided that a public implementation is available for both methods --actually this seems to hold for the method in (Taymouri et al., 2020).

More importantly, it seems to me that the GAN-based method presented in (Taymouri et al., 2020) can predict/generate event timestamps, which can be used to compute activity durations (despite they are not returned directly by the method itself).
Indeed, you could train the model of (Taymouri et al., 2020) against traces containing a pair of instantaneous events for each of their activity instances (namely, a start event and a completion event for each activity instance), and then use the trained generator to produce traces having the same structure.
I presume that the computation of evaluation metrics concerning activity durations and cycle times can be easily adapted to traces of this form.

In my opinion, extending the experimentation to this method would add value to your empirical study, beside making it more complete.

2) Biased evaluation criteria.
The evaluation metrics computed in the tests only focus on precision-like aspects, so that one cannot assess whether the traces generated by a method really cover the variety of process behaviors that occur in the test log. In other words, there may well be modes of the test data distribution that can be missed!
I hence suggest that additional, recall-like, metrics be defined, quantifying how well the test traces are represented by (the sample traces yielded) each of the discovered simulation/generative models.

3) Limited range of datasets.
Three of the logs are very small and, thus, hardly contain sufficient data to train a deep model appropriately.
More importantly, I am afraid that all the processes considered in the experimentation are rather simple and pretty regular/structured.

Thus, it may be risky to draw, from your current experimental results, general conclusions on the relative strength of the two classes of process simulation approaches under analysis.
In particular, I am not fully convinced that DSS models are good (and better than DL-based ones) at capturing control-flow behaviors accurately.
I think, indeed, that such a claim would need to be substantiated by a wider range of tests, encompassing logs of more flexible and/or more complex processes (e.g., featuring long-distance activity dependencies).
Indeed, it may well be that DL models will capture the variety of control-flow behaviors of such processes better than DSS models hinging on local activity dependencies and local time distributions --as partly the CFLS scores that these methods obtained on the last log seem to suggest.

I do understand that it is difficult to find other public logs containing both start and end times for each event, but adding logs of less structured processes would really strengthen the value, coverage and significance of your empirical study.
To this respect, please consider the possibility to relax the requirement of dealing with traces containing only non instantaneous events (i.e., traces where each step encodes the execution of one activity lasting over the time, and associated with both start and end timestamps).
In principle, one could artificially keep all the activity durations fixed to 0, when trying to simulate a log that only stores the start or completion timestamp for each activity instance. It seems to me that the MAE and EMD scores would still make sense in such a simulation setting, and would give useful information on how accurately the model generates the event timestamps (and, indirectly, the cycle time).

4) Some conclusions look too strong (and not fully supported by test evidence).

4.a) In the concluding section you say that "DDS models are suitable for capturing the sequence of activities (or possibly other categorical attributes) of a process."
However, this claim does not seem to be not fully aligned to the test results.

Indeed, the CFLS score of GRU and LSTM models looks better than that of DDS on the real-life log BPI2017W, which is the only log containing more than 10K traces (and 100K events).
May it be the case that DL models capture activity sequence as accurately as (or even better than) DSS models when applied to complex enough processes (and provided sufficiently large amounts of training data)?

On the other hand, on the log of simple processes featuring few activities, DL models could be penalized by using embeddings for the activities (and low-dimensional categorical event attributes). There is, indeed, some evidence in the literature (Everman et al., 2019) that one-hot encoding schemes may help obtain more accurate models than embeddings in such a case --in principle, low-dimensional embeddings may lead to loss of useful information.
I suggest that you consider the possibility to test DL models obtained with one-hot encodings for the activities (and, possibly, for the users/roles), when the number of them is small.

4.b) Still in the concluding section, you say that "DL models outperform DDS models when predicting the timing of activities, specifically the waiting times between activities."
However, the assumption of eager-resources, related to the generation of waiting times, is rather rough.
In my opinion, some way of overcoming this limitation of your DSS models should be prospected, which does not necessarily rely on exploiting DL models.

5) Some lack of technical details and of critical discussion.
- Which specific families of PDFs are used to model inter-arrival times and activity durations?
- The discussion on hybrid approaches combining DSS and DL methods, in order to exploit the strengths of both synergistically, is rather shallow from a technical viewpoint.  Some more technical reflections on this respects could add value to your work.

*** MINOR DETAILED NOTES.

In my opinion, Figure 1 is a little misleading.
The "post-processing" stage (lines 124-128) of the SIMOD pipeline looks more like a part of an optimization loop that also encompasses the "simulation" stage.
In other words, after initializing the DDS model in the "preprocessing" stage, one can  iteratively refine it by generating a number of traces and optimizing the current parameters of the DDS, based on the quality of the traces generated.
As a matter of fact, Bayesian optimization does entail itself a number of simulation runs (actually, 5 runs for each of the 50 parameter configurations explored).
Please, try to describe more clearly this part of the SIMOD pipeline.

Lines 136-138:
It looks like you are presenting GANs and RNNs as two orthogonal solutions.
In fact, a GAN can include RNN components --this is indeed the case of the DL architecture proposed in (Taymouri et al., 2020)

Line 175:
The expression "duration of an event" may sound odd to readers who are used to consider process events as instantaneous events, related to lifecycle transitions of the activities --while the latter can last over time when executed.
Please, clarify this point and define explicitly the meaning of term "event" in your setting.

Figure 3:
Please, clarify the meaning of label "role prefixes".
It seems to me that roles are not used at all in your experimental framework --indeed, the term "role" never appears in Section 4. Anyway, no information is provided about how executors roles were defined/extracted for each log.
Please, clarify this point.
All of the three DL architectures shown in the figure should produce three kinds of output for each generated event: its type (activity), its start time and its end time.
Please make this explicit in the figure, in order to prevent the reader from interpreting that the three output dimensions correspond to the input ones (i.e., activity, role and times).

Line 229:
What is the cost of substitution operations?
Usually, in the computation of trace-to-model alignments, substitutions are disallowed, since substitutions may lead to too optimistic alignment measurements.

Line 230:
You say that the transposition of mutually concurrent activities is allowed for free (i.e. exchanging the occurrences of mutually concurrent activities costs 0).
How do you decide whether two activities are in parallel? Do you exploit some kind of oracle/expert or log-driven heuristics?

In principle, in the case of a DDS model, this information can be extracted from the control-flow structure of the DSS model itself. Do you use the same information for DL-based generative models (which do not make activity dependencies explicit)?
Or do you resort to classical log-driven ordering relationships, induced heuristics from the input log?

To be frank, I believe that using log-driven relationships is definitely reasonable when building/optimizing a DSS model.
However, I am not convinced that it makes sense when it comes to evaluating the quality of simulation results.
In such a case, indeed, log-driven relationships should be validated preliminary by an expert (possibly yourselves) or, at least, by using some well-defined criterion for automatically assessing the significance of these relationships (and filtering out those that are not significant enough) or for making the cost of exchanging two activities X and Y depend on how likely X and Y seem to be in parallel.
It could be beneficial for the reader to have this point clarified (and discussed appropriately) in the revised version of the manuscript.

Line 258:
Please add details on the penalty mechanism.

Line 265:
"holdup" -> "hold-out"


Instruction: You are one of the reviewers, your task is to write a review for the article. You will be given the title of the article, the number of the round in which the article is located, and your order among the reviewers.

Input:
Title: DISCOVERING GENERATIVE MODELS FROM EVENT LOGS: DATA-DRIVEN SIMULATION VS DEEP LEARNING
Review round: 2
Reviewer: 1

Output:
Basic reporting: Basic reporting
The article presents an empirical comparison of two approaches for process discovery/predictive monitoring. The first approach is based on simulation mining: a detailed simulation process model (e.g., a Petri net) is discovered from the data. Then, the model is enriched and used for the analysis. The second approach is based on the deep learning - a generative neural network is trained from the data. Both models are essentially trace generators (including rich features that can be generated into the future). The empirical evaluation is very nice and the topic is worth investigating. The paper is very relevant for practitioners and scientists who are working in the area of process mining. One of the novelties of the approach is the realizations that the two representations (simulation and deep learning) have equivalent usages.

In terms of the standards: the paper is well-written, the literature is covered in a sufficient manner, the paper is structured professionally, is self-contained, and the results are presented in a satisfying manner.
Experimental design: The experimental results are well-aligned with the journal's Aims and Scope. The research questions are well defined, and the paper does fill in a gap in the literature. It is the first paper that considers this equivalence between two generative families of models. I am happy with the update to the paper.
Validity of the findings: I am satisfied with the validity of the findings, after reading the comments by the authors.

Conclusions and limitations are well-stated and the empirical evaluation is described in a valid and satisfying manner.
Additional comments: Excellent rework of the paper.


Instruction: You are one of the reviewers, your task is to write a review for the article. You will be given the title of the article, the number of the round in which the article is located, and your order among the reviewers.

Input:
Title: DISCOVERING GENERATIVE MODELS FROM EVENT LOGS: DATA-DRIVEN SIMULATION VS DEEP LEARNING
Review round: 2
Reviewer: 2

Output:
Basic reporting: I have a couple of minor comments which have to be considered only as a refinement of the already well done work.

Line 196: approach or approaches?

Figure 1: "interarrival dist." and "activities dist." could be written extensively (if you have enough space)
Figure 1: "Resource avail calendar" -> "Resource avail. calendar"

Figure 3 top left both in diagram (a) and (b): "role category" -> "roles category"

Figure 4.b bottom right: "sufixes"-> "suffixes"
Experimental design: no comment
Validity of the findings: no comment
Additional comments: The authors properly addressed all my concerns regarding their work.


Instruction: You are one of the reviewers, your task is to write a review for the article. You will be given the title of the article, the number of the round in which the article is located, and your order among the reviewers.

Input:
Title: DISCOVERING GENERATIVE MODELS FROM EVENT LOGS: DATA-DRIVEN SIMULATION VS DEEP LEARNING
Review round: 2
Reviewer: 3

Output:
Basic reporting: No comment. See section "General comments for the Author".
Experimental design: No comment. See section "General comments for the Author".
Validity of the findings: No comment. See section "General comments for the Author".
Additional comments: The quality of the work is good, in terms of readability, novelty, relevance, technical depth, experimental design and discussion of the experimental results --as I already noticed in my first-round review.

I believe that the Authors have overcome the flaws that affected the former version of the manuscript.
In particular, I am fully satisfied with how they have addressed my comments.

Please notice that there is something wrong with text formatting in page 12:
It looks like the title and beginning of Section 7 have been written within footnote 6.


Instruction: You are one of the reviewers, your task is to write a review for the article. You will be given the title of the article, the number of the round in which the article is located, and your order among the reviewers.

Input:
Title: MACHINE LEARNING ANALYSIS OF TCGA CANCER DATA
Review round: 1
Reviewer: 1

Output:
Basic reporting: The authors present a review to reflect the state of the art in machine learning particularly on The Cancer Genome Atlas Program data.
First, some of the work of the TCGA consortium itself is described to give the reader a good foundation. However, the main goal of this work is to identify and discuss those works that have used the TCGA data to train different ML approaches. The authors classify them according to three main criteria: the type of tumor, the type of algorithm, and the predicted biological problem. One of the conclusions drawn in this work shows a high density of studies based on two main algorithms: Random Forest and Support Vector Machines and naturally an increase is described from the use of deep artificial neural networks. An increase of integrative models of multi-omic data analysis is also presented. Biological problems were classified into five types: Prognosis prediction, tumor subtypes, microsatellite instabilities, immunological aspects, and specific signaling pathways.
A clear trend was found in the prediction of these conditions according to tumor type. This is why a greater number of papers have focused on the BRCA cohort, while specific papers for survival, for example, have focused on the GBM cohort, due to its largeNumber of events.

Any work dedicated to fighting cancer is important. This work is a very good contribution and very useful for the international research community and it is very well written, well motivated and good to read. For all these reasons, this reviewer endorses this work, recommends acceptance, and makes some recommendations below to help further improve the work:

1) For the beginner in this field, a list of all abbreviations used would be very helpful, e.g. MSI = Microsatellite Instable is not mentioned anywhere and newcomers should be able to find their way quickly.

2) Figure 2, the descriptions are very hard to read - maybe the image can be optimized

3) Figure 2 b - is practically illegible

4) Figure 4 also very hard to read

5) In the summary section or before, a bit of an outlook on future important research topics should be given, e.g. comprehensibility, interpretability to further explore causal relationships which is eminently important in cancer research. Here, however, it is totally important to point out that in the medical field always several different components contribute to a result - but this is often negated in current machine learning, consequently, it would be good to say so, and to point to a brand new current work that deals exactly with this matter [x].
[x] Holzinger, A., Malle, B., Saranti, A. & Pfeifer, B. 2021. Towards Multi-Modal Causability with Graph Neural Networks enabling Information Fusion for explainable AI. Information Fusion, 71, (7), 28-37, doi:10.1016/j.inffus.2021.01.008

6) Readers within the subsection "A general perspective of unsupervised learning with TCGA data" may also interested be interested in this brand new work [y]:
[y]	Jean-Quartier, C., et al. A. 2021. Mutation-based clustering and classification analysis reveals distinctive age groups and age-related biomarkers for glioma. BMC medical informatics and decision making, 21, (77), 1-14, doi:10.1186/s12911-021-01420-1
Experimental design: nothing to add - very good
Validity of the findings: nothing to add - very good
Additional comments: please see comments above.


Instruction: You are one of the reviewers, your task is to write a review for the article. You will be given the title of the article, the number of the round in which the article is located, and your order among the reviewers.

Input:
Title: MACHINE LEARNING ANALYSIS OF TCGA CANCER DATA
Review round: 1
Reviewer: 2

Output:
Basic reporting: No comment
Experimental design: No comment
Validity of the findings: No comment
Additional comments: Summary of Paper:
In this paper, the authors provide a comprehensive review on literature using machine learning techniques for the analysis of different types of cancer using TCGA data. Starting with explaining the methodologies used in this survey, the authors present the main results obtained by the TCGA consortium followed by reviewing the capabilities of machine learning algorithms to solve biological problems.  Overall, the paper is well written. Below are my minor concerns on the paper.

Minor Concerns:
•	On line 102, 'Machine learning as a source o new knowledge' -> 'Machine learning as a source of new knowledge.'
•	On line 107, ‘To his aim …’ -> ‘To this aim’.
•	It is advisable to write the article inclusion/exclusion criteria on lines 127-136 more clearly. For example, I am assuming that the last point ‘Article/conference manuscripts using machine learning marginally or without solid biological conclusions’ should be the exclusion criteria.
•	‘neuron networks’ on lines 241 and line 244 should be replaced by ‘neural networks’.
•	Consider rephrasing the sentence ‘In this Figure 4 is interesting to observe how the different problems addressed are distributed differently in the types of tumour.’ on lines 391-392.
•	‘Immnulogical phenotype prediction’ -> ‘Immunological phenotype prediction’ on line 441
•	‘immunologicc phenotype’ -> ‘immunologic phenotype’ on line 455.


Instruction: You are one of the reviewers, your task is to write a review for the article. You will be given the title of the article, the number of the round in which the article is located, and your order among the reviewers.

Input:
Title: MACHINE LEARNING ANALYSIS OF TCGA CANCER DATA
Review round: 2
Reviewer: 1

Output:
Basic reporting: no comment
Experimental design: no comment
Validity of the findings: no comment
Additional comments: The authors have addressed all my previous comments.


Instruction: You are one of the reviewers, your task is to write a review for the article. You will be given the title of the article, the number of the round in which the article is located, and your order among the reviewers.

Input:
Title: GRIMOIRELAB: A TOOLSET FOR SOFTWARE DEVELOPMENT ANALYTICS
Review round: 1
Reviewer: 1

Output:
Basic reporting: Here are my concerns:

- Introduction. First Paragraph. "They showed how data retrieval..."; "They explored the
48 limits to scalability"; etc. ->  Who is the subject that the authors want to refer to when using "They"? "The papers cited? Previous Work? Make clear this aspect in the paragraph. Maybe the entire paragraph can be rephrased.

- Introduction. It would be nice to have the aim of the paper described more clear. From my perspective, the goal is to present the infrastructure and tools, to provide architectural details about it, and also some "usage scenario - proof of concept" to validate the architecture. Making the aim clear,  the authors can better "set" the manuscript expectations for the reader since during the manuscript there are a lot of technical terms and examples. The goal can be better defined in the abstract as well.

- GHArchive Grigorik (NA) - ???, SARA (NA) - ???? - It seems that some references are missing. Please, revise all the references in the manuscript.

- Figures need to be improved. The first eight figures are large with large icons. Figures 9 to 14 have some colors that cannot be visualized by colorblind persons. The different types of icons used are not explained in the text. I also recommend improving the image quality (some parts are blurred).

- 1.3 Contributions. -> The fact of having an open-source community around the Grimoire Labs, would be stated as a contribution?

- Section 2.1 and 2.4 -> Must have a general explanation about the data retrieval component as we have for sections 2.2 and 2.3.

- Lines 133 to 143 must be described together with the Perceval explanation (line 125).

- Section 4 - Table 4. "Retrieval (github) 10:02:12 hours 160 items/min" -> Itens means, pull request/issues and their messages? Commit Metadata? - Please clarify in the text.

- EXAMPLES OF METRICS AND VISUALIZATIONS Section -> The authors provided examples and explanations about metrics and visualizations available. My concern here is: Is it possible to create new metrics and visualizations? How can the users extend it? It would be worth having some instructions and explanations.

- Ethical issues. Recently, GHTorrent lead with some ethical issues (The #issue32 incident) - https://ghtorrent.org/faq.html. I think that this topic deserves some special discussion in this paper. Is there any concern/issue/implication for researchers/practitioners when using GrimoireLab?

- The paper description has too many bullets (e.g Related Work, Components description, Discussion, conclusions, etc). I suggest reviewing the manuscript to avoid this practice that breaks the text fluency.

Minor
- Line 343 - GrimoireLabis - GrimoireLab is
- Line 378 -  Bugzilla timing9): -> Exclude )
- The term "experiment" is used to describe the case studies/scenarios
Experimental design: No comments. The manuscript describes the Metrics Grimoire infrastructure. The infrastructure is validated with a ``Proof of concept'' presented by describing some usage scenarios.
Validity of the findings: No comments. The validation was described by presenting some usage scenarios, by providing the replication package and access to the tools and source code.
Additional comments: The paper presents GrimoireLabs, a toolset composed of five main components, supporting about 30 different kinds of data sources related to software development. The toolset has been tested in many environments, for performing different kinds of studies, and providing different kinds of services. The components and infrastructure are presented. The authors used some usage scenarios to validate the presented toolset.

In general, I liked the paper idea. As a Researcher, I also would like to thank the authors for their effort to built this set of tools and the infrastructure. A scenario replication is available, which is really good for practitioners and researchers. However, the paper needs to address presentation issues (image quality, too many bullets, etc). It is also worth to include discussions about some potential Ethical issues and Metrics/Visualization extension.

Check my comments to improve the paper manuscript.


Instruction: You are one of the reviewers, your task is to write a review for the article. You will be given the title of the article, the number of the round in which the article is located, and your order among the reviewers.

Input:
Title: GRIMOIRELAB: A TOOLSET FOR SOFTWARE DEVELOPMENT ANALYTICS
Review round: 1
Reviewer: 2

Output:
Basic reporting: Comments about basic reporting are done in the "General comments for the author" field.
Experimental design: Comments about experimental design are done in the "General comments for the author" field.
Validity of the findings: Comments about the validity of the findings are done in the "General comments for the author" field.
Additional comments: # General comments

This paper presents GrimoireLab, a set of components to data retrieval and analysis of software repositories. The paper describes the structure and functionality of the main components of GrimoireLab (data retrieval, data storage, identities management, analytics, and orchestration). The paper presents real-world use cases of the proposed components, as well as examples of metrics that can be calculated and visualizations that can be generated by GrimoireLab. The paper also provides a discussion of challenges that need to be addressed to collect and analyze software engineering data in both the industrial and academic settings. The strongest point and greatest contribution of the paper is a set of components that were developed during many years of research effort and have matured as open source projects. As for the weakness associated to this paper, it is possible to highlight the lack of validation of the claims performed as discussion points. Below I provide specific comments for the overall paper presentation of for each section of the paper.

# Overall presentation

In general, the paper is adequately written but a few clarifications are needed, which I specifically point in the detailed comments for each section. Regarding presentation, there are two major issues.

- The first issue is the excessive usage of pronouns with an unclear reference (i.e., antecedent noun), which yields too many ambiguous sentences. For example, in the sentence “They showed how data retrieval …” [P2,L46], the antecedent noun is not clear (does “they” refer to repositories or tools?). Occasionally, the antecedent noun can be inferred from the context, however this is not always the case. Other examples include: “They explored the limits of scalability …” [P2,L47], “They demonstrate …” [P2,L48], “Some of them …” [P2,L56], “Most of them …”[P2,L63], “They use …”[P2,L64], “… of how it …” [P3,L105], “… that is can provide.” [P3,L106], “… contained in it.” [P8,L239], “… from them …” [P8,L243], “… these cases …” [P8,L245], “… visualize it.“ [P9,L277], “… all of these scenarios …” [P11,L304], “.. some of them,” [P11,L308], “… its analysis starts,” [P12,L318], “… all its items …” [P12,L318], “… depending on it.” [P18,L405], “… deals with this …” [P18,L411]. Many other similar cases appear throughout the paper.

- The second issue is the occurrence of vague descriptions of rather important information. For example, in the sentence “When the data to be collect is really large …”, it is hard to judge what “really large” is. For such technical descriptions, the text should be as precise as possible. Other examples include: “… some idea …” [P11,L312], “… for a little while …” [P12,L319], “… every few minutes.” [P14,L336], “… some data …” [P14,L338], “… some custom Kibana visualizations …” [P14,L351], “… key data …” [P14,L357], “… some exceptions …” [P16,L365], “… some fields …” [caption of Figure 19], “… in some cases …” [P20,L504], [P20,L507], “… some other goodies …” [P21,L561], “… some if its properties.” [P21,L577].

- Typos: “… we will us …” instead of “… we will use …” [L111], “… HatStallThen,” instead of “… HatStall. Then,” [P9,L277], “Data collection for “git” Includes …” instead of “Data collection for git includes …” [P12,L324], “GrimoireLabis …” instead of “GrimoireLab is …” [P14,L343], “… more indexes is produced …” instead of “… more indexes are produced …” [P16,L364], “… the should fixed …” instead of “they should be fixed …” [P19,L26], “… kinds consumption…” [P19,L480] instead of “… kinds of consumption …”, “This make their …” [P20,L508]

- [P20,L494-502] The usage of the word “you” in that context is discouraged and inconsistent with the rest of the paper.

- Unclear sentences: “… consult analysis and visualizations of it” [P2,L56], “… specifically to retrieve source code or data” [P2,L59], “Interlaced with this process, SortingHat process identities found.” [P12,L320], “For many data sources, this is software not difficult to write …” [P19,L26], “… plugged to the data.” [P19,L447]

- Citation format: all citations are in the format “Author et al. (Year)”, while in many cases the correct format seems to be “(Author et al., Year)”

# Introduction

Main point: The motivation for another system to collect and analyze data from software repositories is not well described. Although the Introduction reviews relevant literature, there is no clear description of the motivation for a study on another related system. For example, while the Introduction says “In many cases they are not easy to reply and operate, and in others they are difficult to use for large-scale, continuous data retrieval. Not all of them provide support for retrieval, storage and analysis of the data …”. It is not clear which of these limitations GrimoireLab addresses. Also, there is no description of how and why the approach taken by GrimoireLab to tackle those limitations is better than the approach taken by other systems that also tackle these limitations. Most importantly, the Introduction describes no empirical evidence that GrimoireLab can overcome previous’ systems limitations. I would suggest having a table that compares the features and functionalities of the related systems compared with the features and functionalities of GrimoireLab.

[P1,L42] The definitions of “… tools and complete systems …” it is not clear what the difference is. Is GrimoireLab being proposed as a tool or complete system?

[P2,L49] The sentence “… different approaches to avoid harming the project hosting systems …” needs clarification. What are those approaches and what type of harm are they trying to avoid?

[P2,L75] Although the types of repositories whose data collection is supported by GrimoireLab are defined in later sections, it would be interesting to define (or perhaps give examples of) different types of repositories in the Introduction.

[P2,L80-82] Is the claim of efficiency validated in the paper? It would be nice to describe how efficient is GrimoireLab compared with other solutions that are mentioned in the Introduction.

[P2,L84] It would be nice to motivate the identity management module upfront in the Introduction. When “module for identity management (…) in combination with custom code to merge or tag identities” is mentioned for the first time, it is hard to understand what this module does. Also, if this feature is unique for GrimoireLab, this should be highlighted.

[Section 1.3] The list of contributions reads more like a list of features of GrimoireLab. Beyond GrimoireLab itself (which is a great contribution), I would expect to see other contributions in terms of 1) novel techniques to investigate the efficiency and efficacy of GrimoireLab in coping with the described challenges, or 2) novel findings or results of applying GrimoireLab in the field, 3) or data that allows the comparison between GrimoireLab and related systems. Moreover, Section 1.3 seems like a right place to thoroughly describe the differences and similarities between GrimoireLab and the related systems.

[Section 1.4] Some definitions appear after their first usage (e.g., data source, kind of data source). I would suggest having a structured glossary in this section, with a formalization of all definitions used in the paper. For example, the definition of “kind of data source” is imprecise, as the data sources given as examples have a notable different API. Also, readers that are less familiar with mining of software repositories would benefit from having a definition of related terms.

# Section 2

Main points: 1) There is no discussion of the rationale behind the existence of each component. The paper describes the functionality of each component. To some degree, the relationship between these components is also described. However, except for the identity management component, it is unclear what challenges each component solves, both in terms of structure (e.g., why is this specific architectural design chosen?) and functionality (e.g., how a component overcome the current limitations of data collection and analysis?). It is also unclear how GrimoireLab addresses the flaws (in terms of design, efficiency, or efficacy) of related systems. 2) Given the paper's descriptive characteristics, I would recommend having a formal notation (e.g., UML) to describe the components and their relationships (Figures 1-7 and Figures 10-14). For example, the meaning of the relationship shown between components (Figures 1-7) is not defined (does the arrow represent data flow? functional dependency? or something else?). Also, although some colour and shape code is used in Figures 9-14, the meaning of such codes is unclear. 3) There is no discussion regarding the separation of concerns among components.

[P3,L104] Could one interpret the sentence "... describes the structure of GrimoireLab and its module ..." as "... describes GrimoireLab's architecture ..."? If so, which architectural level the paper describes? Also, the text interchangeably mixes the words "structure" and "design". Please, clarify if they have the same intended meaning.

[P3,L128-131] "Graal runs third party tools on git repositories.", "... runs a collection of tools on checkouts ...", "... produced by those tools ...". The mentioned tools need to be clearly defined, at least what are their outputs.

[P5,L150-159] Please, clarify that are the "common features" that are reused to collect data from different data sources and which components implement those features.

[P5,L164] "... controlling the details of the job ...". Which are the controlled details and why?

[Perceval and Graal] Perceval and Graal components seem to have a strong coupling. Please, clarify how they are typically integrated.

[Section 2.3] The motivation for the identity management component is well described. However, I would recommend addressing the following points: 1) There are many academic papers describing different heuristics for identity disambiguation (e.g., resolving users that adopt different e-mail addresses to commit to the codebase). Different heuristics are associated with different performance (e.g., accuracy). How does GrimoireLab's approach to identity disambiguation compare with related work in this area? What is the performance of GrimoireLab's approach? 2) "... sends them back to be added to the enriched data." [P6,L202] is not clear in Figure 7. Also, by Figure 7 it is unclear how "enriched data" relates to other sub-components of SortingHat. In general, the textual description of the figures needs to be improved.

[Section 2.4] Although the section states Visualization and Analytics, what is actually described relates to visualization and data exporting. Also, please clarify which "documents" are referenced in [P7,L213] and what "actionable inspection" [P7,L216] means in this context.

[Enriched indexes] The concept of "enriched index" needs to be clarified and better formalized.

[Figures] There is excessive usage of Figures that hinders readability. While I appreciate the effort to describe the relationship between GrimoireLab's components incrementally, many of the figures are redundant and can merge into a single one that conveys the same message. Also, the textual description of many figures (e.g., Figures 4 and 8 ) is lacking.

# Section 3

Main point: this section lacks a detailed description of the techniques used by each component. Also, it seems that a large portion of the section's contents overlaps with the previous section's contents (in particular, most of the section describes the relation between components and how they individually work). Please, consider improving the separation between the sections.

[Section 3.1] What are the advantages of using Perceval compared to having a script that uses CURL with pagination, for example? Also, does Perceval collects only issues and pull-requests from GitHub? If that is the case, the paper needs to discuss this issue as a limitation of GrimoireLab. More generally, although the system can collect data from more than 30 data sources, not all data from each source can be collected.

[P8,L242] Section 3.1 describes a data retrieval scenario and claims that "... which will use Perceval (as library) in the background, to clone the repository and get the list of commits, and then run third party tools that analyze each relevant file in each commit to get complexity metrics from them in a single JSON ". It seems that Perceval is doing both data retrieval (e.g., cloning repositories and getting commits) and data analysis (e.g., calculating complexity metrics). I wonder how much this characteristic of GrimoireLab (specifically, Perceval) is based on best practices. For example, why does the same component perform retrieval and analysis? As these are different concerns, it seems that different components should address them. Another question raised by the quoted sentence is how "relevant files" are set in Perceval.

[P9,L276] In the sentence, "SortingHat will work based on heuristics ...", what are those heuristics and how well do they perform?

[P9,L282-284] The technique adopted by GrimoireLab for incremental data retrieval needs to be way better described. For example, it is unclear from the sentence "... which can be used to select which items from the database need to be retrieved." [P10,L283] how such items are selected. Also, the sentences "... how often will data sources be visited for incremental retrieval." [P10,L287] and "... refresh periods (how often data will be retrieved incrementally from repositories)" [P10,L290] gives the impression that the incremental data retrieval is not real-time but instead performed in batches. If that is the case, how this feature compares with real-time techniques such as processing an event stream?

[P10,L294] The paper says that "... Graal analyzes source code, by running third party tools with the help of Perceval.". From Figure 2, it is possible to verify that Graal is within the "data retrieval" component. This observation evidences that the paper lacks a discussion on the separation of concerns of the components. Also, there is no illustration of the relationship between Graal, third-party tools, and Perceval in any of the figures.

# Section 4

Main points: I am surprised by how much Section 4 overlaps with prior sections. It claims to describe “three real use cases for GrimoireLab”, while the previous section claims that “GrimoireLab can be used in many different scenarios. In this section we describe some of them …”. Please, consider improving the separation of the contents described in Sections 2, 3 and 4. It seems that Section 4 has the potential to describe insightful empirical studies regarding GrimoireLab. However, there is no definition of research questions, no description of rigorous methods to perform such empirical studies, and no discussion of the learned lessons. The lack of a clear and well-thought study design leads to severe issues to the validity of the claims.

[Section 4.1] The main takeaway from the analysis presented in this section does not stand out. For example, how does the running time change by changing the associated factors (what are the factors in the first place)? Also, what is the baseline for comparison of the presented results? How good or bad are the presented results compared with other similar systems? What are the strengths and limitations?

[P11,L312] How is GrimoireLab’s performance evaluated?

[P11,L314] The sentence “The main figures of that experiment are included in Table 1”. I do not understand what “figures” means, neither which “experiment” is described. In addition, what is Mordred in the sentence "The experiment was run by Mordred"?

[P12,L319] "... sleep for a little while ...". This sentence is imprecise. Also, how much of the "little while" is reflected in the results shown in Table 2.

[P12,L320] "The data shown in this paper ...". Please, clarify which data.

[P12,P324] "Data collection for git includes cloning of git repositories for GitHub, and production of the raw index by analyzing those clones". How each of these steps is reflected in the results shown in Table 2?

[Table 3] There is no reference of Table 3 in the text

[Section 4.2] What is the main takeaway of Section 4.2?

[P14,L329] “Table 4 shows data about an example of large deployment of GrimoireLab”. What does “large deployment” mean?

[P14,L335-336] “The deployment has been running continuously for more than four years, retrieving data incrementally from repositories every few minutes”. How many minutes?

# Section 5

[P16,L365-367] “… metrics are not (except for some exceptions) a part of the indexes stored in Elasticsearch: they are computed either by the visualizations, or by the tools producing the reports.”. What is the rationale behind this design decision? Why is it better recalculating the metrics instead of storing them?

# Section 6

Main point: this section needs to improve the discussion of the novel learned lessons after years of research and development of GrimoireLab. For example, prior studies discuss the same pointed issues: “minimizing interactions with data sources”, “data source APIs are not uniform”, and “data provided are not uniform”. How good or bad are the solutions proposed by GrimoireLab? More importantly, which findings and evidence support the claims made in this section? Without clearly providing evidence to support the discussion, most of the claims are understood as speculation. Another important point is that, in general, there is a good amount of discussion about what the system does, but barely any discussion on how and why.

Some excerpts in this section deserve clarifications. Below I point to specific excerpts:

[P18,L405] “… do some different tricks depending on it”. Please, clarify what those “tricks” are and provide the motivation for using them. Preferably, use more specific and technical wording.

[P18,L414] “… the code may be more generic …”. It is unclear which “code” is being referenced.

[P19,L450] “… to address the first approach …”. Please, clarify which “first approach”, as the previous sentence was talking performance issues of data collection.

[P19,L450] “… to avoid those that could cause trouble.” Please, clarify what type of trouble. Also, please, prefer a more technical and specific wording.

[P19,L470-471] “GrimoireLab recovers as nicely as possible by retrying, and by continuing after failure”. Please, elaborate on how this feature is implemented. Is there any increase in the waiting time after retrying? How each type of failure is handled? Is the failure handling approach dependent on the data source?

[P19,L475-476] How can consumers detect errors in the retrieved data? What are those errors? What are the most common ones?

[P19,L477-478] What is a “bug in time interpretation”?

[P20,L494-502]  Reproducibility: why providing a reproducible build of GrimoireLab instead of providing a dump of previously collected data? The argument that “that would be enough for anyone to get the same data again, provided data sources didn’t change” does not hold, as rarely a data source will not change.

# Section 7

[P20,L503-506] I do not understand what this paragraph conveys. For example, what does “production of merged collections” mean? Please, clarify the sentence “The availability of origin fields simplify the merge, allowing for complex combinations, such as using raw indexes for different data sources to compose a meta study”.

# Section 8

[P20,L530-533] How the two proposed categories of related work arose? Is this classification by the own authors? What process was used to achieve this classification?

[P20,L541-542] “… not only metadata about the projects, but as much data as possible about software development …”. Please, clarify what are the differences between “metadata about the projects” and “data about software development”.

[P21,L563-567] I appreciate the comparison between GrimoireLab and GHTorrent (same for the paragraph of [P21,L591-599]). While I agree that using the projects API will require a reduced number of queries to the endpoint, the events API (as used by GHTorrent) provides a richer source of data about the activities performed in a repository. Please, clarify which type of data can be obtained by using the projects API in comparison with the events API and vice-versa.

[P21,L576-577] “Boa is a system …” [L576] and “Boa is a programming language …” [L577]. What is Boa anyways?

[Lack of subsections] To improve readability, I would recommend separating Section 8 into sub-sections (per related work’s theme).

[P22,L628] Please, define the acronym “SQO-OSS”.

# Section 9

I appreciate seeing a mature and well-supported set of tools being provided to the academic and industrial communities of mining software repositories. Many of the provided tools are tested in-field and used to support different types of research, which clearly shows the value of those tools. As a side note, it would be a great addition to this paper if any lessons learned from the usage of GrimoireLab by the core services of a company could be discussed, even if the discussion stems from public information.

# Section 10

[P23,L685] “… an industry-strength platform …”. What evidence is given for this claim?

[P32,L687-699] This excerpt claims unique characteristics of GrimoireLab that differentiates these tools from others. However, in the list of characteristics, we can read “support of different data sources”, “flexibility and configurability”, “identity merging”, “tested in real-world”, etc. Unfortunately, none of these characteristics are “unique” to GrimoireLab and can be observed in other systems or academic papers. Perhaps, the unique feature of GrimoireLab is how all these characteristics are put together, which reinforces the need to present a strong comparison between GrimoireLab and other existing related systems.


Instruction: You are one of the reviewers, your task is to write a review for the article. You will be given the title of the article, the number of the round in which the article is located, and your order among the reviewers.

Input:
Title: GRIMOIRELAB: A TOOLSET FOR SOFTWARE DEVELOPMENT ANALYTICS
Review round: 1
Reviewer: 3

Output:
Basic reporting: Overall the manuscript is well structured. The writing is clear with only few syntax, grammar, or spelling mistakes. I recommend carefully re-reading the references to fix some capitalization errors.

The numerous figures provided are helpful. I suggest using a standardized notation (such as UML) or explaining the meaning of the various used shapes (e.g. rectangle, vs ellipse, vs, rounded rectangle. Also, I recommend using a lighter color for the purpse shapes, which are difficult to read.

Extend Table 2 to include the time it took for each phase to run.

Suffixing the duration with hours/min in Table 3 is confusing. Provide all durations simply in HH:MM:SS units.

Table 3: Clarify what "items" refers to.

I recommend rephrasing the abstract's  "Conclusions" section to state what GrimoireLab has achieved, rather than summarize the paper.
Experimental design: According to the framework provided by Stol and Fitzgerald (2018), this is solution-seeking rather than a knowledge-seeking study.  Solution seeking studies aim to solve practical problems for which solutions to practical problems can be engineered. In such studies, researchers design, create, or develop solutions for a software engineering challenge; in this case by building the GrimoireLab tools. As such, I do not expect a research question filling an identified knowledge gap. Instead, I expect to see description of a tool advancing the state of the art.

The described system (GrimoireLab) advances the state of the art by supporting more data sources and analyses as well as enhanced interoperability.  The study can benefit from a rigorous comparison between GrimoireLab and other existing systems in terms of performance, supported data sources, and analyses.

Two case studies, one on IoT repositories and one on Wikimedia Foundation repositories illustrate the power of GrimoireLab.  These can be profitably extended to showcase what research questions can be answered through the provided quantitative data.
Validity of the findings: The provided conclusions mostly follow the conducted study.  They can be extended through more rigorous benchmarking and more ambitious example studies.

The authors provide links to a companion package on Zenodo, which contains data, logs, and configuration files for the IoT case.  I recommend that this should be expanded to include the Wikimedia foundation case, and also additional data regarding the recommended additions. Uploading a source code snapshot on Zenodo can ensure the source's long-term survival.

The provided online documentation is quite detailed and well-written.

I was able to install and run Percical using the provided instructions.  On the other hand, the installation of Grimoirelab appeared to get stuck in a loop with lines such as the following.

Requirement already satisfied: setuptools in ./venv/lib/python3.7/site-packages (from importlib-metadata->flake8>=3.7.7->graal==0.2.3->grimoirelab) (50.3.2)
Requirement already satisfied: wheel in ./venv/lib/python3.7/site-packages (from importlib-metadata->flake8>=3.7.7->graal==0.2.3->grimoirelab) (0.36.1)
Requirement already satisfied: setuptools in ./venv/lib/python3.7/site-packages (from importlib-metadata->flake8>=3.7.7->graal==0.2.3->grimoirelab) (50.3.2)
Requirement already satisfied: wheel in ./venv/lib/python3.7/site-packages (from importlib-metadata->flake8>=3.7.7->graal==0.2.3->grimoirelab) (0.36.1)

The denormalized output of the grimoirelab tools seems to result in considerable waste.  For example, each commit contains, again and again, the full details of its author and committer: about 49 elements. What is the rationale of this decision and what is its effect in terms of waster storage and processing cost?
Additional comments: L. 36: Your phrasing focuses on the process. Consider stating that tools provide data about the software development process and the developed artifacts.

L. 91: The described attributes refer to coverage breadth, rather than exensibility.

L. 129: Generally, running tools on a commit's checkout is needlessly expensive. Accessing directly Git's file-system is more efficient, because it avoids the cost of copying all files to the OS's file-system.

L. 244: I find it a pity that Graal uses Perceval internally, rather than allowing the user to compose various tools through a common protocol and data format.

L. 254: Consider providing an actual example, rather than an abstract description. How could one e.g. measure the evolution of comment spelling mistakes over time?

L. 490: I recommend backing up this claim with benchmark results.

L. 855: One citation to the smartshark ecosystem should be enough.

I recommend clarifying in the manuscript following questions

- How does GrimoireLab  handle GitHub pull requests?
- What is the schema of the data provided by Perceval?
- Do SortingHat and HatStall offer an help to address GDPR requirements?  How do you propose that the tools should be employed for handling them?
- L. 234 Perceval appears to retrieve the repository's metadata and also a clone of the actual repository.  How does it handle clashes of repository names between repository names from different sources?
- How can GrimoireLab be extended?  What does it take to create a new data source or analysis metric?  How can such additions be contributed back to the community?
- What does an index hold?  What operations does it facilitate?
- What are the practical limits of GrimoireLab in the number of repositories or data volume it can process?  On what volumes has it been tested?  What support is offered for running GrimoireLab on multiple hosts?
- How is GrimoireLab tested (c.f L. 508)?  Are there some metrics (e.g. code coverage) that can help track the quality of its testing?  Where do these stand today?
- How are the provided GrimoireLab Docker images supported and updated? Given GrimoireLab 's evolution how can GrimoireLab users ensure the long-term replicability of their studies?


Instruction: You are one of the reviewers, your task is to write a review for the article. You will be given the title of the article, the number of the round in which the article is located, and your order among the reviewers.

Input:
Title: GRIMOIRELAB: A TOOLSET FOR SOFTWARE DEVELOPMENT ANALYTICS
Review round: 2
Reviewer: 1

Output:
Basic reporting: I suggest proofreading the article. Some typos need to be fixed before publication. It is probably because of the "diff" document, but please, revise accordingly.

Just some examples:
Line 390 -> GrimoireLaballows - GrimoireLab allows
Line 301 -> Percevaland - Perceval and
Line 426 and 428 -> Gerrit efficiency) - remove a bracket
In this section we discuss - ","
Table 6 need to be formatted as the other tables
Experimental design: Section 2 was improved to better describe the proposed software architecture.
Validity of the findings: The authors properly improved the examples and usage scenario. Also, the replication package was updated.
Additional comments: I am grateful for the authors 'efforts to reorganize the article and address it according to the reviewers' suggestions. I have no more suggestions for improving the article.


Instruction: You are one of the reviewers, your task is to write a review for the article. You will be given the title of the article, the number of the round in which the article is located, and your order among the reviewers.

Input:
Title: GRIMOIRELAB: A TOOLSET FOR SOFTWARE DEVELOPMENT ANALYTICS
Review round: 2
Reviewer: 2

Output:
Basic reporting: I thank the authors for their effort in addressing all comments of the major revision. The two main problems related to the presentation were thoroughly addressed, the description and motivation of the Grimoirelab’s components were improved, and a new case study to evaluate Grimoirelab’s performance was presented. Some minor specific points are still unclear. Below I describe each of these specific points:

1. The authors improved the motivation for Grimoirelab, in specific by adding Table 6 and a sentence to the introduction. The sentence in [L114-119] still lacks a clarification of how Grimoirelab “improve the situation in solving the practical problem of retrieving data from software development repositories, preparing it for further analysis, and providing basic analysis and visualization tools help in exploratory studies”. Is this due to the holistic approach taken by the tool? In the description of the features shown in Table 6, please clarify the meaning of the “selected projects” scope. Does the “collection of projects” need to be in git repository? What is the difference with respect to “GitHub subset”?
2. If possible, would be preferable to adopt a consistent notation for the arrows throughout the figures that explain Grimoirelabs’ architecture. For example, the dashed arrow is used to represent invocations in Figure 6, workflow in Figure 7, and “entities flow” in Figure 10.
3. I believe that a brief description of the “conservative approach” to identity management, as well as the examples given in the response letter, would be of great aid to readers.
4. Typos:
     a. [L230] missing closing parenthesis.
     b. [L390] “GrimoireLaballows”, “Percevaland”
     c. Caption of Figure 10: “sold arrows”
     d. [L731] “the the”
5. Sentences that could have grammar revision: [L156-157] “… and may need some other components to work, which in that cases are installed as dependencies of it.”
Experimental design: Please, see the comment in "basic report". The new version of the manuscript improves the experimental design by adding a new case study to evaluate the performance of the tool.
Validity of the findings: Please, see the comment in "basic report".
Additional comments: No additional comments


Instruction: You are one of the reviewers, your task is to write a review for the article. You will be given the title of the article, the number of the round in which the article is located, and your order among the reviewers.

Input:
Title: GRIMOIRELAB: A TOOLSET FOR SOFTWARE DEVELOPMENT ANALYTICS
Review round: 2
Reviewer: 3

Output:
Basic reporting: See previous review.
Experimental design: See previous review.
Validity of the findings: See previous review.
Additional comments: The authors have diligently addressed almost all the review comments. As a result, the paper now clarifies many important issues.  I do not think that another review round is required. However, the authors might want to take the following comments into account when revising their manuscript.

The figures are now more readable.  The Perceval class diagram is useful,
but I don't think that UML class diagrams are the most appropriate for
illustrating the other tools.  I would still have preferred to see
them follow a UML notation (e.g. that of collaboration diagrams) rather than
the invented ad hoc ones, but I think that this choice can be left to the
authors.

The system's performance is now nicely illustrated in Table 5.  The numbers
for enriched issues and merges may indicate a pathology regarding I/O
operations; you might want to investigate why clock time is two orders of
magnitude higher than processor time only for these specific two use cases. Could caching or better indexes help?

Table 6 now provides a very useful summary of related systems.  Consider
adding the corresponding references below the headings or in a separate row at the end.

Line 898,899: Consider rephrasing as "GrimoireLab provides a simple way
of getting the data needed for *a* study in reasonable time"


Instruction: You are one of the reviewers, your task is to write a review for the article. You will be given the title of the article, the number of the round in which the article is located, and your order among the reviewers.

Input:
Title: EMPIRICAL EVALUATION OF METHODS FOR DE NOVO GENOME ASSEMBLY
Review round: 1
Reviewer: 1

Output:
Basic reporting: no comment
Experimental design: I do not see a real research question that the authors have investigated in this paper. They do not state how their work fills an identified knowledge gap - reassembly of well-known reference genomes is not a gap.
Validity of the findings: no comment
Additional comments: While generally well written and present, I fail to see what the authors were trying to study in this manuscript. Reframing the topic in the sense of validation of certain hypotheses about particular species (and including more than 1 genome) would certainly help.


Instruction: You are one of the reviewers, your task is to write a review for the article. You will be given the title of the article, the number of the round in which the article is located, and your order among the reviewers.

Input:
Title: EMPIRICAL EVALUATION OF METHODS FOR DE NOVO GENOME ASSEMBLY
Review round: 1
Reviewer: 2

Output:
Basic reporting: The article is comprehensive and describes in detail most of the widely used assembly methods. The literature review is very thorough. The results are well described and convey the drawbacks and advantages of different assembly tools. One of the figures needs more work, but everything else looks good.
Experimental design: The experiments are well designed and the metrics on which assemblies are evaluated are correct and widely used in the literature. I have a couple of comments here. First, the data used for experiments are not very well defined - for eg, the coverage of data, the technology, error rate, etc. Second, one of the assemblers called HiFiAsm is not used for the evaluation. This assembler is currently the best in class so I would like to see it compared with the existing tools in the manuscript. Also, please provide the refseq/genebank accession for the reference genomes used for evaluation. Same for raw reads - please provide SRA accessions if the data was used from there. Although assemblers are compared across assembly metrics, the runtime/memory of different assemblers are not provided. For eg, Canu typically requires more memory and runtime compared to other assemblers. Please provide this information. Please also provide the metrics for one of the larger genomes such as the human genome.
Validity of the findings: 1. Please make numbers right-aligned2. in the tables for better interpretation.
2. Figure 4 is confusing. Please make it as bar plot and color bars by the assembly algorithm.
3. Please provide the assemblies generated by different tools in a public bucket/link.
Additional comments: No additional comments