Instruction: You are one of the reviewers, your task is to write a review for the article. You will be given the title of the article, the number of the round in which the article is located, and your order among the reviewers.

Input:
Title: AN EXPERT STUDY ON HIERARCHY COMPARISON METHODS APPLIED TO BIOLOGICAL TAXONOMIES CURATION
Review round: 2
Reviewer: 1

Output:
Basic reporting: Since my first review, the authors have answered most of my concerns in the basic reporting section. However, while the figure has been split into four separate larger figures with improved legibility, they still have compression artefacts in them - this is probably just a battle to be had with the screen capture and pdf generating software but it does make the detail within them murky, especially in the matrix figure.
Experimental design: In their revision, the authors have tackled the comments in my original review as well as can be expected within the text. The reasoning for their method of iterating the tests by question rather than by visualisation is now clear. The acknowledged weaknesses in some of the visualisations' designs are now discussed at greater depth in the 'threat to validity' section, though 1) it would be good to state why the binding of datasets to methods seemed an acceptable risk, (or re-word it to say "trade-off" rather than risk, which then ties into the remainder of the sentence more clearly), and 2) state that learning effects are a risk to all small-scale studies, not just this one in particular.

The million dollar question is whether clearly stating the weaknesses is enough in everyone's view to compensate for them being there in the first place. My own opinion is that at the very worst, there may be a case to remove the agglomeration findings compared to the other visualisation types, as this was the visualisation type with the difficulty with colour choices.
Validity of the findings: I didn't have much to comment on that needed fixed or expanded on in this section. They've explained why the matrix method wasn't mentioned in some of the results discussion.
Additional comments: I'd like to thank the authors for taking the trouble to go through mine and the other reviewer's comments in such detail. I'd still be interested in more details on the artificial taxonomy generator, but maybe that's for another paper (people have made papers out of synthetic data generators).

One other point that the author's revision has brought up is that there have been quite a few tools for taxonomic comparison developed in the past, but they have mostly disappeared due to software/hardware obsolescence (some used unsecured java webstart, some only run on old operating systems, some just disappeared etc). How do they plan on stopping the same happening to their tool?

In summary, I would recommend a further minor revision.


Instruction: You are one of the reviewers, your task is to write a review for the article. You will be given the title of the article, the number of the round in which the article is located, and your order among the reviewers.

Input:
Title: AN EXPERT STUDY ON HIERARCHY COMPARISON METHODS APPLIED TO BIOLOGICAL TAXONOMIES CURATION
Review round: 2
Reviewer: 2

Output:
Basic reporting: I'm satisfied with the actions taken by the authors to address the reviewers concerns. I'm still very worried though about the tables/figures at the end format, it shouldn't be used for this type of paper and the journal should do something about it.
Experimental design: I'm satisfied with the actions taken by the authors to address the reviewers concerns.
Validity of the findings: I'm still very worried though about the tables/figures at the end format, it shouldn't be used for this type of paper and the journal should do something about it.
Additional comments: No additional comments


Instruction: You are one of the reviewers, your task is to write a review for the article. You will be given the title of the article, the number of the round in which the article is located, and your order among the reviewers.

Input:
Title: GENOME ANNOTATION ACROSS SPECIES USING DEEP CONVOLUTIONAL NEURAL NETWORKS
Review round: 1
Reviewer: 1

Output:
Basic reporting: The quality of the written english could be improved throughout the manuscript. Some areas of the paper are technical and need improvements in the quality of the writing to make them understandable. There are also duplications of some text in the methods section ("implementation" and "hyperparameter tuning" sections).
Experimental design: The authors do not describe how the TSS annotations used for training are collected. Are these TSS’s identified using cDNA clones, Illumina RNAseq, RACE sequencing/ CAGE sequencing of 5’ tags, GROseq/NETseq of nascent RNA, 5’GROseq of nascent 5’ tags, or PacBio/Oxford Nanopore long read sequencing, etc, etc. Many commonly used techniques, e.g. Illumina RNAseq, give very poor estimations of transcriptional start and termination sites due to tailing off of coverage towards the 5’ and 3’ of genes. The technique used to produce the annotations will therefore give some measure of the quality of the positively labelled training data. I expect that for a well annotated species such as human the TSS positions will generally be good, but this is not always the case.

The authors implement a data augmentation strategy, namely the sliding of the sequence window relative to the positive and negative examples. They report that the model trained with data augmentation performs better on the test dataset but worse on the held-out chromosome 21 test. This sounds like an example of data contamination/leakage – if the authors performed data augmentation before splitting the data into training and test data, there will be examples in the test set which are almost identical to those in the training set. The authors in fact allude to this themselves on line 294 - "The [data augmentation] approach is biased by the fact that similar sequences, albeit with different offsets... are found in both training and test sets". This makes any evaluation of the model on the validation or test set worse than useless as the model can achieve perfect scores by overfitting to the training data. Data augmentation should always be performed after data has already been split into training, test and validation subsets, to prevent contamination. I think that this need to be corrected as it is a major experimental flaw.

The authors apply models trained on human TSSs to other vertebrates (mouse, chicken, zebrafish). I would argue that training a model on one species before applying it to another is not a sensible proposal, since there may be unexpected differences in the sequence composition of TSSs in other species which lead to spurious results. To take an example from computer vision, you would not train a convolutional neural network to recognise dogs using only pictures of labradors, and then expect it to correctly identify pictures of poodles, since this is outside the bounds of the training data.
Validity of the findings: The authors suggest that using imbalanced training datasets is a good way to prevent model over-prediction when applied to the genome, where the positive class is rare. Will simply choosing a more stringent prediction threshold produce the same result?

The data used to train the model are not provided but are publicly available. Code used for training models is provided via github.
Additional comments: No additional comments


Instruction: You are one of the reviewers, your task is to write a review for the article. You will be given the title of the article, the number of the round in which the article is located, and your order among the reviewers.

Input:
Title: GENOME ANNOTATION ACROSS SPECIES USING DEEP CONVOLUTIONAL NEURAL NETWORKS
Review round: 1
Reviewer: 2

Output:
Basic reporting: Regarding the Abstract, I would recommend to further elaborate the following two sentences for clarity purposes: "These approaches rely on the generation of sequences batches with known annotations for learning purpose" and "This cross-species application of convolutional neural networks trained with genomic sequence data provides a new technique to annotate any genome from previously existing annotations in related species". I would also discourage the use of "optimal" to describe the proposed strategy cause CNN offer an optimization that is not guaranteed to be a perfect/optimal formula.

Regarding the Introduction, while sufficient context is provided, the first half of the section would benefit from additional references to support the numerous messages provided.

Line 41: CNN are here introduced so the authors may want to rephrase the sentence saying something like "In particular" instead of "indeed".

Line 43: "illustrate [...] reliably" may be considered a stretch - CNN attempt to approximate an unknown function/mapping. Overall, the sentence would probably benefit if further elaborated to clarify the message the authors aim to deliver.

Line 61: "Firstly, we do not take into account prediction scores obtained from test sets initially retrieved from the training sets as a quality measure" is quite confusing. Do the authors mean "prediction scores obtained [...] *like(/in a similar fashion as)* the training set"? Test sets examples cannot be obtained *from* the training set.

Line 67: "We then propose a new application of the method" - which method? No method seems to be presented this far.

Line 74 (and 100): "human regions containing TSS" instead of "human TSS containing regions" may improve clarity.

Regarding the Methods,  this section would benefit from additional references, e.g. no reference is provided for RefGene and all the data sources, sub-sampling and data augmentation, and the genome-wide standard score.

Line 82: the authors introduce the "positive *input sequences*" while they talk of "negative *class*" in line 87. This is somehow confusing.

Figure 1: the one-hot encoding does not match the encoding outlined in line 97. The dropout labels are represented before the max-pooling layers, is that correct? Why are dropout layers represented as labels instead of layers/masks? Isn't the size of the widest max-pooling layer = 64 (instead of 128)? Also, the figure shows only 1 input sequence although multiple annotations are represented in the output. More importantly, the convolutional layers appear to be arranged in parallel while Table 1 and 2 describe a more standard "pipeline-like" arrangement.

Caption of Figure 1: the last 2 sentences may benefit from further elaboration. The first one does not specify that the model has to be applied multiple times to give probabilities "over the full chromosome length" while the second one might contain too many details, e.g. the average z-transformed probability hasn't been introduced/mentioned yet.

Lines 102 and 106: it is confusing to talk of "layer kernels" or "convolution operation" instead of "convolutional layers" - which would also keep consistency with the paragraph.

Equation 1: j is not initialized.

Lines 117, 183 and 203, and caption of Figure 1: there is a refuse "over-fitting".

Lines 117-121: the authors may want to move these lines to the end of this subsection, after the concluding passage describing the model adopted, i.e. the fully-connected layer, etc.

Line120: dropout is typically applied after the non-linearity and pooling layer and not after "each layer" (in a CNN). Please clarify also whether dropout is applied after the fully-connected layer as well.

Line 136: the authors may want to specify that training on a GPU is typically faster than on a CPU.

Line 145: a full stop appears to be missing; the network architecture is detailed in Table *2*.

Line 146: no data appears to be on GitHub, only the program and respective utilities.

From line 150 onwards: there appears to be an issue with labeling the different (sub)sections as in "(see Section )".

Lines 148-162: this subsection may be better positioned after subsection "Conventional performance measures".

Lines 164-175: it is a copy of lines 135-146.

Lines 177-178: the authors may want to specify the unit of measurement, i.e. bp.

Lines 184-185: the authors appear to jump on "imbalanced data issue" from "training a CNN model on the balanced dataset" [line 177]. This creates several inconsistencies in the following subsections as well.

Table 1: "The first column" does not seem to "represents [only] different kernel sizes for 3 convolutional layers". Kernel sizes for 3, 4 and 5 convolutional layers are listed.

Line 222: a parenthesis is missing.

Line 227: the authors may want to support the "wide" with some references or, possibly, merge the line with the following one.

Line 239: AUROC and AUPRC have not been introduced in "Conventional performance measures" so it would be sensible to introduce here the acronyms.

~~Results
Line 246: it appears to be non-sensical since the predictive model is generating the predictions, i.e. the result of its attempt to differentiate positive and negative TSS.

Caption of Figure 2: please introduce the acronym "GC". It may also be beneficial to specify that the "30 trained CNN" are trained on different datasets and are not an ensemble.

Line 261: it may be beneficial to specify the value of Q for the 100* dataset. A similar explanation may go in line 91 as well.

Lines 272-273: this information may be more appropriate just in Caption of Figure 2.

Line 284: the authors may wish to clarify whether they talk about test set examples and training set instead of "test sets" and "learning examples", respectively.

Line 285: there is a refuse "Figure Supplementary Figure".

Line 300: the AUROC is typically defined as the "Area Under the ROC" and not as the "Area Under *an* ROC *Curve*".

Supplementary Figure 4a: the authors may want to specify that different *training* datasets are represented along the x-axis (same for Figure 1). The authors may wish to clarify whether they applied the same weighting scheme used at training time to calculate the entropy.

Line 302: "PR is" *not* "considered as a trustworthy measure [...]", is it?

Line 311: Binary Cross Entropy is the loss function chosen by the authors for this work, back-propagation works with other loss functions as well. Please clarify this.

Line 314: a perfect model is arguably a model with log loss = 0. What about generalization and overfitting? What *if* the dataset contains noisy information?

Lines 320-323: the authors may want to say that what they describe is intrinsically linked to the loss function adopted at training time, i.e. for this manuscript, the entropy discussed a few lines before.

Lines 324-328: the authors may wish to further elaborate to clarify the messages they aim to deliver. It may be sensible to also identify and label the test set used.

Line 330: this sentence is confusing as the authors are talking of "training and validation sets". Moreover, the only MCC scores reported on the test set (S_Figure4) disagree with this line.

Line 335: please clarify whether Zg corresponds to "the standard score". Same in Caption of Figure 3.

Lines 336-338 and 342-344: these lines appear to describe Figure 3 instead of "Comparing 1* and 100* models over a full chromosome". Thus, it may be sensible to move these lines to Caption of Figure 3.

Line 355: the authors may want to specify that the t-RNA cluster is highlighted in green *in Figure 3a,b*.

Line 375: the authors may want to clarify that "the signal to noise ratio is improved in the 100* models" with respect to the 1* models.

Line 384: the authors may want to refer to Figure 3 along to Figure 2.

Lines 395-396: it appears like an overstatement as some signal appears to be present. Please clarify.

Line 401: Down, T. 's PhD thesis appears to be unavailable. The authors may wish to cite an easier-to-access manuscript co-authored by Down T., e.g. DOI: 10.1126/science.1112014, or other works.

~~Discussion
Line 432: the cited manuscript [Wainberg et al.] claims that "there are now over 1 million genome *datasets*". Please clarify.

Line 434: the authors may wish to further elaborate this sentence and talk of "majority class" (or "most represented class") for clarity and consistency with the following lines.

Line 448: citation(s) required for "many genes are homologous".
Experimental design: ~~Introduction
Line 45: the sentence appears to be vague, some additional detail or possibly reference may be missing. Do the authors aim to highlights the limits of current approaches, e.g. training and test sets do not cover the entire genome (which is not clearly stated at present)?
Moreover, the following lines do not seem to specifically address how the "training sets are obtained", e.g. line 47 mentions "Prediction" while line 49 talks of "Similar training datasets".

Lines 58 and 59: the authors may wish to further elaborate on the outcome and the differences of Kelley et al.'s and Jaganathan et al.'s works with respect to the present manuscript. How does this study fill in the current literature?

Line 91: the authors do not specify why the maximum considered value of Q is 100, e.g. why not 399?

~~Methods
Line 143: the authors may want to specify that they use a sigmoid function and, possibly(?), a softmax function in input and output of the dense layer, respectively. Do the authors use a sigmoid function (as in Table 2) instead of a classic softmax function in the output layer? Using a sigmoid function would result in a regression instead of a classification.

Line 158: it is unclear to me (1) how the authors calculate an element-wise average of a vector and (2) how such operation results in a new vector.

Lines 175-176: the authors may want to specify how many folds they used and what share of the data available. However, this information does not seem to match with the splitting outlined in line 178. It is unclear whether the final models are trained in cross-validation or on the entire training set. Moreover, the authors may want to clarify whether the architecture detailed in Table 2 is the only one used in section Results.

Line 192: when a = 0.5, equal importance is given to precision and recall; if a = 1, each TSS example will be weighted x times where x = (noTSS/TSS examples). Therefore, when a = 1, equal importance is given to "both classes". Please clarify.
Did the authors use this weighting scheme for any result reported in this manuscript?

Lines 206-206: do the authors ensure that the window sliding "around the TSS positions" contains TSS? Is the dataset still balanced after performing the described data augmentation strategy? The authors may want to specify that data augmentation has not been used to build all the datasets assessed in the manuscript and that the only method using DA is labeled as the "DA model".

Lines 216-217: FP, FN and TN are wrongly introduced.

~~Results
Line 270: the authors may want to clarify whether the positive TTS in the 30 1* datasets are the same.

Line 293: did the authors ensure that augmented examples stayed in the same split with the respective source examples when splitting the data in training, test and validation?

Lines 411-414: the authors may want to follow the same strategy previously followed to support the presumable superiority of more precisely annotated genomes. In particular, one chromosome or data not used at training time would give stronger results if used at test time.
Validity of the findings: ~~Results
Line 240: the results of Umarov and Solovyev are on a different application. Please clarify.

Line 265: it sounds like an overstatement. Please elaborate.

Line 358: it may sensible to support this sentence citing some related work.

Line 406: the authors may want to give additional context about the choice of using a different r value, i.e. r = 400bp instead of r = 2kbp, for what essentially is one of the main messages of the manuscript.

~~Discussion
Line 450: the authors may want to add that the described methodology can *sometimes be applied* to more distance(/ different) species.

Line 452: it is unclear whether a more thoroughly annotated fish genome would do (as the authors state in the previous subsection).

Lines 454-456: these last few lines are essentially speculating on the possibility of applying CNN "to address the conservation of the [...] sequences" which is not validated in this manuscript.
From PeerJ's guidelines: The conclusions should be appropriately stated, should be connected to the original question investigated, and should be limited to those supported by the results. In particular, claims of a causative relationship should be supported by a well-controlled experimental intervention. Correlation is not causation.
Additional comments: The authors present a study on the potential of applying CNN on different species to predict genome annotations. The manuscript is quite comprehensive and some effort on improving the presentation would benefit the overall study performed.


Instruction: You are one of the reviewers, your task is to write a review for the article. You will be given the title of the article, the number of the round in which the article is located, and your order among the reviewers.

Input:
Title: GENOME ANNOTATION ACROSS SPECIES USING DEEP CONVOLUTIONAL NEURAL NETWORKS
Review round: 2
Reviewer: 1

Output:
Basic reporting: The paper is much improved from its previous submission. In places the English could still be improved slightly. However, both the biological and computational concepts are explained with much more clarity.

There have been recent relevant publications which should now be cited, namely:  https://doi.org/10.1038/s41598-020-57811-3

I think that perhaps the term "promoter" better (and more canonically) describes the sequences that are being predicted than "Gene start site".

Line 183: The model was tested on chromosome 21 which was withdrawn from the training set. This should be stated in the methods.

Figure 3: What are the green and yellow shaded areas on figure 3A and B heatmaps?
Experimental design: Line 210: The authors state that the performance of the model varies over different regions of chromosome 21. Human chromosomes are not homogeneous in the sequence content and have isochores with large differences in GC content. It would be interesting to see whether plot of local Z-score across a whole chromosome follows the isochores/GC-content of chromosome 21.

The authors have missed an opportunity to interpret the sequences identified by their model. Attention and saliency scores can be used to identify sequences that are important for predicting promoters. These could be compared with known sequences in e.g. the JASPAR transcription factor motif database. I do not think this needs to be done for publication, however it would improve the interest of the paper.
Validity of the findings: The authors have done a better job explaining the pitfalls of training on one organism and applying to others.
Additional comments: No additional comments


Instruction: You are one of the reviewers, your task is to write a review for the article. You will be given the title of the article, the number of the round in which the article is located, and your order among the reviewers.

Input:
Title: GENOME ANNOTATION ACROSS SPECIES USING DEEP CONVOLUTIONAL NEURAL NETWORKS
Review round: 2
Reviewer: 2

Output:
Basic reporting: Although the manuscript has been improved considerably, the help of a native speaker may considerably improve the overall clarity and make the present manuscript even more professional.

Line 15: "learning [..] sets" is an uncommon way of referring to training sets.
Line 32: "in other species"? The message is unclear, please clarify.
Line 37: please extend on why "these annotations [...] cannot be determined experimentally for every organism and every cell".
Line 41: the authors may wish to specify that ML has been particularly successful "in Genomic applications" as well.
Lines 57-59: the authors may wish to cite [24].
Lines 65-66: did the authors attempt to reimplement [18] or analyze how much memory is "lot of memory"? It may be interesting to give a bit more context on the benefits and disadvantages of [18], i.e. to adopt windows covering the whole genome.
Lines 103-112: it is somehow confusing to interchangeably refer to human or mouse genome - sometimes without stating it but only the number of sequences.
Line 135: the authors may wish to cite the relevant manuscript descriving the back-propagation algorithm.
Line 149: the authors may wish to cite Keras as outlined at https://keras.io/getting-started/faq/#how-should-i-cite-keras.
Line 178: please clarify what "the input data" are. Does it refer to the training set or also the validation set? This is clear only after reading the Supplementary material.
Line 184: the authors may wish to clarify whether chromosome 21 has been withdrawn from the training *phase*, i.e. the validation set as well as the training set.
Lines 252: the authors may wish to support the former statement referring to the literature.
Line 257: the authors may wish to support the following statement "The two chromosomes carry homologous genes" citing [32].
Experimental design: Line 112: the authors may wish to introduce the sets used for the zebrafish and chicken genome.
Line 120: s is not explicitly introduced; does s have the same value for any kernel? The authors may wish to introduce the stride number adopted.
Lines 131-133: the authors may wish to specify that what they adopted (and describe) is a softmax layer.
Line 171: there is a discrepancy between "r kbp" and how r is introduced, i.e. r= 2000 bp or 400 bp (line 164). Also, the authors may wish to outline the equation to calculate λ.
Table 1: please clarify why the output shape of the first Conv2D is 32x284x1 instead of 296, i.e. b - s +1 = 299 - 4 + 1 = 296.
Line 181: the authors may wish to report the results they obtained on the mouse genome as well.
Lines 233-245: the authors may wish to specify the λ obtained in the four randomly selected GSS.
Line 284: the authors may wish to introduce a baseline training a model on the 4 sets of GSS curated, i.e. human, mouse, chicken and zebrafish, or building an ensemble using the 4 CNN, or selecting the best n sets/CNN, e.g. an ensemble using only the human and mouse genome.
Validity of the findings: Line 253: the cited work appears to be preliminary since the mammalian test is based on "313 sequences, of which 50 were kept aside for test purposes" [28]. Please clarify this in the manuscript, and possibly expand on the implications.

Supplementary material: 100* appears to be the best model in figure 4 see, although the authors say "the CNN model applied on the balanced data (1*) yields the best performance on the test set regarding Receiver Operating Characteristic curve (ROC)". Please clarify.
Additional comments: I would like to thank the authors for carefully addressing my comments and those of the other peer-reviewer. I believe that the manuscript has been considerably improved since the first submission and, thus, I congratulate the authors for the tremendous amount of work.


Instruction: You are one of the reviewers, your task is to write a review for the article. You will be given the title of the article, the number of the round in which the article is located, and your order among the reviewers.

Input:
Title: ENHANCEMENT OF CONFORMATIONAL B-CELL EPITOPE PREDICTION USING CLUSMOTE
Review round: 1
Reviewer: 1

Output:
Basic reporting: The English writing is poor.
Experimental design: no comment
Validity of the findings: no comment
Additional comments: I think this is a potentially interesting contribution on a challenging problem of solving the class imbalance problem in  B-cell epitopes prediction. However, there are some major issues with the dataset construction, experimental design and English writing.
1.
The quality of the dataset need to be assessed. How is the sequence identity between any two proteins within the dataset?  Any redundant sequences should be removed from the dataset to avoid bias.
As the author mentioned, the model validation was conducted by the 10-fold cross-validation method in the overall dataset. N-fold cross validation is not sufficient enough and another independent dataset should be used to test and compare with different sampling methods.
2.
I am confused by the purpose of this work.
If the author wants to prove ClustSMOTE to be an improved sampling approach,  more datasets are needed to be used to compare with other sampling methods.
If the author would like to provide an method to solve the B-cell epitopes prediction problem, they need to compare their performance with other B-cell epitopes predictors.
3.
The text in this manuscript is often inaccurate or incomplete and needs to be revised.
In many places, the text is very sloppy/careless and often hard to make sense of, e.g., produced many features; Vaccine design experts use certain characteristics of residues to be identified; the risk of failure is still quite high; However, their performance level is still not high….
I suggest a careful rewrite for the whole paper.

4.
The method should be made publicly available in the form of a web-server or downloadable code. Either ways, the mode of availability should be indicated in the manuscript.

Minor:

1. Why the performance using DT is always better than SVM?
2. The full names of SMOTE, ASA and RSA should be explained.


Instruction: You are one of the reviewers, your task is to write a review for the article. You will be given the title of the article, the number of the round in which the article is located, and your order among the reviewers.

Input:
Title: ENHANCEMENT OF CONFORMATIONAL B-CELL EPITOPE PREDICTION USING CLUSMOTE
Review round: 1
Reviewer: 2

Output:
Basic reporting: The article would benefit of a revision regarding writing style and phrasing. There are some ambiguities that make understanding what the authors intend to communicate challenging.

- Line 41, when you write "independent of expertise and precision", did you mean "dependent"?
- Line 45: "from others part of antigen" -> "from other parts of antigen"
- Lines 69-70. "the prediction of conformation epitope has not been explored..." and "feature extraction methods..." How are both facts related? If the same data is used for both, why the lack of a dataset affects only to the prediction and not to feature extraction/balance? I would suggest rephrasing lines 69-72.
- Line 83: I suggest to briefly introduce the basics of each category.
- Line 87: clustering-based approaches belong to the data-level category, correct? I suggest to specify this in the text.
- Line 89: "approaches used which are regardless" -> "approaches used: regardless"
- Line 125: "were" -> "where"
- Line 127-128: "item data" -> "samples"
- Line 129: "surrounding" -> "resulting"
- Line 129: If one class contains 944 samples and the other contains 12464, why is the ratio imbalance 1:10? Wouldn't it be 1:12/1:13?
- Lines 137-138, I suggest to rephrase so that the explanation is easier to follow
- Line 143: "used feature" -> "applied the feature"
- Line 158: shouldn't the names of the features be in bold too? Also, why is Contact Number (CN) not in Table 2?
- Line 184: "However, it also has an advantage", please clarify which advantage?
- Lines 184-185: "advantage", perhaps you mean "disadvantage"? And which is the advantage of undersampling?
- Line 255, "DT": this is not the first time in the document that you refer to the decision tree, the acronym should be defined earlier.
- Line 282: I suggest to present the name that you give to your method (CluSMOTE) earlier, and to use through all the paper. As you are comparing your proposal with regular SMOTE and clustering, it is at some points difficult to follow which method are you referring to.
- Line 336: "Data is available at", there seems to be some text missing here.

I suggest to add some more references, specially:

- On lines 55-61 you provide several references to characteristic-based approaches, which you seem to be focused on. Then, on lines 62-65 you barely give any examples on graph-based (only two cites, and both from the same author). Is this because there are not many works falling in this classification. If this is the case, you may considering stating so. If not, consider adding more references.
- On lines 65-68, please provide some references on learning-based approaches.
- Line 80, please provide references for each method.
- Line 85, please provide a reference for random undersampling.
- Lines 153-154, please provide the reference where it is concluded that there is no difference.
- Lines 165-166, I suggest to put the URL as footnote, or to cite it using url citation style instead of including it on the text. It is also relevant to include the date when you accessed that URL.
- Line 189: please provide reference for HDBSCAN.
- Line 279: please provide reference for k-medoid.
Experimental design: The explanation of the methods should be extended to provide enough information to make it repeatable.

- Lines 193-194, further explanation is needed: how are you defining "optimal" here? what metrics are you using to guide this step?
- Line 197: consider using the equation editor (if using Microsoft Office) or the adequate latex library for writing your formulation. Also, explanation on several parameters is missing (you only define r).
- The parameters used for the classifiers (SVM and DT) need to be detailed for repeatability. Moreover, several approaches have been proposed for both methods through the years, so I would suggest including references for exactly which configuration of SVM and DT are you using for your experiments.
- Section starting in line 205. AUC and TP, TN, FP, FN definitions are widely known, and I don't think is necessary to include the equations 1-3
- In equation 6, there are some parameters that are not defined.
- Line 237: does this mean that you are computing the average of the validation set for the 5 times that the process is carried out? please clarify a bit these sentences.
- Lines 239-241, please state the software versions.
- Line 241, please detail the modifications, for repeatability purposes.
- Line 248, you state that "r=1 states that the number of the negative class dataset is the same as the positive class or without SMOTE". I understand that r=1 means that only the clustering, undersampling step, is applied, correct? I suggest to rephrase it to make it more clear.

In general, I would suggest that, when referring to results that are in a Table, you should explicitly reference the table.

- Lines 260-261, "did not affect the performance of the algorithm", however in Table 4 the values are slightly different.
- Lines 265-268, further clarification is needed. What are the optimal conditions in this context?

- Table 2: I suggest to highlight the best value for each column in bold.
- Table 3 is missing the units (I assume that they are milliseconds, as in the figures, but you should indicate it). I would suggest to remove this table entirely, because it does not add much if compared to the figures (and it is easier to observe the trends in the figures).
- Table 4: why only the columns for AUC, Gmean, Adjusted Gmean, and Fmeasure are highlighted? I would highlight the best values in all the parameters/columns. Also, it would be better to indicate the r value in each case.

In the experiments, I would suggest to include also the values without applying balancing methods (that is, the results from trying to classify the original dataset as-is), so that we can observe the improvement achieved by the proposed method over the original.
Validity of the findings: - Line 257, you mention that "the increase in r value decreases TPR and increases TNR", but this only applies to DT, not SVM. I suggest to clarify this in the text, and also to explain why the classifiers behave differently.
- Lines 258-259: "this is in contrast with the use of the SMOTE method". However, your results varying r (Table 2) do not compare regular SMOTE and your approach. The only results that actually compare both approaches are in Table 4, and you use only 1 r value. You would need to include more results or references to support that regular SMOTE behaves differently than your approach.
- Line 260: "from the AUC parameter, the optimal r value = 2", this also holds true only for DT, while in SVM higher r values seem to yield better results.

- Figure 2: I find these results a bit odd. I would need to know the specifications of the algorithms (model, parameters, if the software and hardware were the same in both cases), but I would recommend the authors to double-check the training of the SVM and, if the values are indeed correct, to provide an explanation of why is this happening. In any case, you need to add the specifications of your computer for this data (Figures 2 and 3) to be meaningful, as now it is unclear how to interpret your times. Moreover, perhaps it is better to change the scale of Figure 2 to seconds instead of milliseconds?
- Figure 3: do you have any explanation for the behavior of the SVM? it seems to get faster when the ratio increases, but from 3 to 4 it became slower again.
Additional comments: The main points which I think would benefit this article the most are:
- Review of the phrasing and style, to improve clarity. For example, for consistency, define the terms the first time they appear (specially for method's names) and then use them through the whole document
- Extend the methods: more information is needed, the article is missing information for it to be fully repeatable


Instruction: You are one of the reviewers, your task is to write a review for the article. You will be given the title of the article, the number of the round in which the article is located, and your order among the reviewers.

Input:
Title: ENHANCEMENT OF CONFORMATIONAL B-CELL EPITOPE PREDICTION USING CLUSMOTE
Review round: 1
Reviewer: 3

Output:
Basic reporting: Cluster-based undersampling and SMOTE for class imbalance on conformational B-cell epitopes prediction method

Binti Solihah Corresp, Azhari Azhari, Aina Musdholifah

Handling class imbalance is a chalanging problem in conformational epitope prediction. This study proposes a cluster-based sampling method to overcome the chalange. To that end, Machine Learning methods (SVM, RFC) have been applied for the classification purposes with the implementation of SMOTE and the performance is compared (by means of a 10-fold cross-validation) to the SMOTE, Random Undersampling, and cluster-based undersampling. The results shows improvement over earlier methods (AUC and Gmean parameter is 0.87% while Adjusted Gmean achieved 0.88% score and F-measure achieved 0.53% score). My overall assessment is that its an okay paper which requires major revision to meet the publication standard of the journal and the corresponding field.

The first lecuna of the paper I find is that it is full of abbriviations! The authors should explicitely spell out several of such abbriviations when they are first mentioned! The other shortcoming is that they don’t take care to explain a few basics in the intro for example what is «class imbalance»? What is «random undersampling»? Where is the cross-reference for such terms? Do the authors consider their readership of their paper to be restricted to a few real exparts in the field alone?

1. What SMOTE  stands for ? Should be declared in the abstract. What is the underlysing principle / philosophy of this methodology – should be described in the intro.

2. CEP: utilizing the characteristics of solvent accessibility (Kulkarni-Kale et al., 2005) (Introduction, Line 56) – should be rephrased as ‘solvent accessibility based properties’.

3. Discotope: combining characteristics of solvent accessibility and log-odd ratio – should likewise be ‘ solvent accessibility based properties and log-odd ratio’! ‘log-odd ratio’ of what?

4. what is AAIndex? Amino Acid Index?

5. ‘However, it is constrained by many problems such as imbalance classes that occur in the formation of datasets for the learning process.’  - ‘imbalance classes’ should be ‘imbalanced classes’!

6. imbalance class handling should be imbalanced class handling or handling of class imbalance!

7. The defination of ‘negative (non-epitope) and positive class’ in the given context is given much later before the terms are used for the first time!  (Line 93)

9. Overall, the introduction requires substantial rewriting to make it more lucid and readable. It is too full of gergonswithout proper defination / references !

10. Features Extraction: ASA: The state-of-the-art to compute ASA is the Lee and Richards method implemented in NACCESS. The authors should explain the choice of their PSAIA software (Mihel et al, 2008).

11. RSA is also (better) known as burial of solvent accessibility (Basu et al., BMC Bioinformatics, 2011, 12:195). The authors should definitely mention that they are the same.

12. Calpha should be Cα  - wherever applicable.

13. Protrusion Index: The percentage of residual atoms in an ellipsoid centered on atoms exposed
in the outermost position (Ponomarenko et al, 2008). - the sentence is incomplete. How residual atoms are defined in this context?

14. B-factor states atomic mobility – due to thermal vibrations!

15. The equations should be better formatted using proper equation editor.

16. «In order to test the reliability of the resulting model, the model validation was conducted by the 10-fold cross-validation method in the overall dataset. Each cross-validation process was carried
out five times, for every process the corresponding performance is recorded and expressed in the mean and standard deviation.»  - why five times? To check stabilityof the results? Then 5 is a very low number. The authors should explicitely mnention the random components that might differ between each run effecting the change in  the overall performance. Usually an n-fold cross validation is good enough and can be complemented by another independent validation (see: https://www.ncbi.nlm.nih.gov/pubmed/27307625). To my view, the authors should carry out an independent validation and compare the performance with the competing methods – if an indendent benchmark is available.

17. After all this effort, what is the net percentage increase in performance (say, based on AUC) of the current method with respect to the privious state-of-the-art?  Should be explicitely mentioned.
Experimental design: Some of them should be revised as suggested in the 'basic reporting'
Validity of the findings: suggested in the 'basic reporting'
Additional comments: The authors should carefully consider addressing all points raised in the review.


Instruction: You are one of the reviewers, your task is to write a review for the article. You will be given the title of the article, the number of the round in which the article is located, and your order among the reviewers.

Input:
Title: ENHANCEMENT OF CONFORMATIONAL B-CELL EPITOPE PREDICTION USING CLUSMOTE
Review round: 2
Reviewer: 1

Output:
Basic reporting: The goal of the manuscript is to tackle one of the main drawbacks in the current epitope prediction methods: class imbalance. To that end, they propose a method that combines undersampling of the majority class using clustering, and oversampling of the minority class applying SMOTE. The authors compare their outcome with other methods for epitope prediction. They also compare the results obtained applying two machine learning classifiers to the data resulting a) applying only the undersampling part of the method b) applying only the oversampling part of the method and c) applying the complete method, with different configurations.

The quality of the manuscript has been much improved since the previous review, and there are only some minor suggestions that I would like to share with the authors.

- Is HDBScan an acronym? if so, explain at first occurrence.
- Same for SVM and DT, use the complete name once before starting to use the abbreviation.
- Line 60, you refer to the last group as "machine learning", and then in Line 78 as "learning-based". I would suggest to choose one of them, and use it always in the other references to the same group.
- Line 89, class imbalance could be defined in a more general way as when the amount of samples from one of the classes is larger than in the other (if the ratio positive/negative classes were 10:1 instead of 1:10, that would still be an imbalanced dataset, and lead to classification issues).
- Lines 103-104, you can remove the sentence "Many methods have been proposed to overcome the class imbalance problem", as it is a repetition from Line 93.
- Line 114, "Providing different weights is categorized as cost-sensitive in the algorithm level". Is this still within the algorithm-level group or to introduce the cost-sensitive group? If it is the second, I would suggest to state so, as you did with the previous groups ("In the algorithm-level approach...").
- Line 120, "top", change to some more specific, such as "most common" or "best performing", depending on what you meant.
- Line 212, "An exposed residue is likely to bind to other molecules for accessibility reasons", this sentence does not add much information on its own, and does not seem to be related with the explanation that follows.
- Line 218, "a number of" -> "several"
- Line 252, if the file is available as supplementary material, please state so.
- Line 319, could you provide a reference for the decision stump learner?
- Line 326, could you add a reference for the C4.5 algorithm?
- Line 342, I would suggest to translate "jika" to either English ("if") or mathematical notation
- Line 352, is the word "complex" equivalent to "fold" in this context?
- Line 431, "better than other methods in the general protein antigen" -> "better than other methods in classifying the general protein antigen"
Experimental design: - From the abstract, it is still a bit unclear what classification task are you performing, I would suggest to state it more clearly.
- Line 206, please clarify why do you choose that specific value.
- Line 258, "sliding window 9" -> "sliding window of size 9". Was this value taken from a previous work, or determined empirically for this experiment. Please clarify in the text.
- Lines 323-326, I would suggest to not refer to the specific functions within the software that you are using, and substitute that for the more general algorithm concepts (e.g. do not specify Integer.MAX_VALUE, just state that the maximum depth of the tree is 10). This makes your method repeatable for readers that use different software.
- Lines 401-420: specify what r value is used for these experiments.
Validity of the findings: - Line 372, "This tendency shows that the more negative class data that are sampled, the lesser the loss of information". Do you mean that with larger r you undersample less, so you remove less information from the data/the model has more information to work with? I would suggest to rephrase this sentence and clarify it.
- Line 395, are you referring to both combined? otherwise, TPR decreases, and TNR increases in a similar proportion. The AUC and the Gmean are the best for r = 2, and the FPR also improves at r = 2 if compared with r = 1.
- Lines 398, "performance of SVM is affected by kernel selection problems" -> if this is an hypothesis from your results, rephrase to "performance of SVM is likely to be affected by kernel selection problems" or "in view of the results, the performance of SVM is probably affected by kernel selection problems". I would also suggest to include a remark in your conclusions stating that other models or configurations may be explored, and that your classification results may become even better, as the classifier fine-tuning was not the main goal of this work.
- Line 427, "considerable increase in TPR", please clarify what are you comparing to: previous works, using regular SMOTE...
- Table 2: there is a typo on "Precision". In the column PPV, why is the 8th row highlighted? Also, in the column FPR, you want the false positive rate (FPR) to be as small as possible, correct? the highest value for each model is now highlighted in that column.
Additional comments: No additional comments


Instruction: You are one of the reviewers, your task is to write a review for the article. You will be given the title of the article, the number of the round in which the article is located, and your order among the reviewers.

Input:
Title: ENHANCEMENT OF CONFORMATIONAL B-CELL EPITOPE PREDICTION USING CLUSMOTE
Review round: 2
Reviewer: 2

Output:
Basic reporting: Although the authors have revised the manuscript there still are issues that need to be addressed.

1. "The disadvantage of NACCESS is that it can only count one molecular structure at a time; thus, users must create additional scripts to count a number of molecular structures at a time (Mihel et al., 2008)." (Line 216)

"The PSAIA is not only limited to counting one molecular structure but can also be used to calculate other features, such as RSA, PI, and DI."  (Line 219)

- So the authors mean to say both NACCESS and PSAIA have the same limitation that they can only count one molecular structure at a time. Then, why are they using PSAIA rather than the more standad NACCESS method  - is not clear at all, especially given the fact that they themselves elaborate on the different ASA values returned by NACCESS.

2. The use of the Gly-X-Gly peptide fragment is relevant to compute burial of solvent accessibility (bur) - which is required if you want to demarcate individual residues into burial bins (buried / exposed). I am not sure as to whether they require residue-wise burial for a similar purpose or that the purpose can be served by ASA for the residue surface patch alone without normalizing it in a residue-wise manner! So, I am not sure whether the GXG tripeptide theory works here! It's not very clear from the statement:  "whether a residue is likely to bind to other molecules for accessibility reasons"! In that case, what are the criteria they have adopted for declaring a residue completely buried or partially buried or completely exposed - must be elaborated!

I would deeply encourage them to use the already set standards in the field as in

Basu et al., BMC Bioinformatics, 2011, 12:195

0.0 <= bur <= 0.05 : Completely buried (bin1);
0.05 <= bur <= 0.15 : Partially buried with higher burial (bin2);
0.15 <= bur <= 0.30 : Partially buried with lower burial (bin3);
0.30 <= bur : Completely exposed (bin4);
Experimental design: N/A
Validity of the findings: The suggested minor revision is a must. Then only it can be validated.
Additional comments: No additional comments


Instruction: You are one of the reviewers, your task is to write a review for the article. You will be given the title of the article, the number of the round in which the article is located, and your order among the reviewers.

Input:
Title: ENHANCEMENT OF CONFORMATIONAL B-CELL EPITOPE PREDICTION USING CLUSMOTE
Review round: 3
Reviewer: 1

Output:
Basic reporting: I am extremely displeased and taken to a surprise that the authors don't really care to cite the source article (Basu et al., BMC Bioinformatics, 2011, 12:195) proposing the standardization of RSA bins even after acknowledging the same in the rebuttal.

The original point raised in the review:
---------------------------------------------------------------------------------------------------------------
I would deeply encourage them to use the already set standards in the field as in

Basu et al., BMC Bioinformatics, 2011, 12:195

0.0 <= bur <= 0.05 : Completely buried (bin1);
0.05 <= bur <= 0.15 : Partially buried with higher burial (bin2);
0.15 <= bur <= 0.30 : Partially buried with lower burial (bin3);
0.30 <= bur : Completely exposed (bin4);

----------------------------------------------------------------------------------------------------------------------
Author's reply:

In the study of conformational epitope prediction, the RSA value is used for two purposes: (1) creating a dataset to form a classification model. (2) as a feature in the formation of classification models. Related to the formation of the dataset, Rubinstein et al. (2009) use the definition of RSA> 0.05 to get the residue exposed (using the standard as used in Basu et al. (2011)). In Kringelum et al., (2012), the RSA limit was 0.01,0.05 and 0.5. Ren 2014 uses RSA> 0.25 because 75% of epitopes have RSA> 25.9%

-------------------------------------------------------------------------------------------------------------------

The burial bins were first formalized by Basu et al., BMC Bioinformatics, 2011, 12:195 and therefore it is imperative that any resetting of the bins afterword (in other words, if the methodology is used in a variant form) should cite the source article. This is very very common in academic publishing and a violation of this, to my modest opinion qualifies as unethical.

I would sincerely request the editor to strictly look into this matter.

The rest of their responses are okay with me, though there are several grammatical flaws that should be corrected.
Experimental design: Same as 'Basic reporting'
Validity of the findings: Trustworthy and logical
Additional comments: Please refer to the basic reporting.


Instruction: You are one of the reviewers, your task is to write a review for the article. You will be given the title of the article, the number of the round in which the article is located, and your order among the reviewers.

Input:
Title: ADAPTATIONS OF DATA MINING METHODOLOGIES: A SYSTEMATIC LITERATURE REVIEW
Review round: 1
Reviewer: 1

Output:
Basic reporting: The paper reviews the application of data mining methodologies/processes, either “as-is” or adapted for some specific use. The authors have followed the SLR (systematic literature review) method suggested by Kitchenham et al., which actually is the most widely accepted method for compiling, identifying, extracting and analysing studies on a given topic.

The review topic is not especially novel itself, as there is a large number of SLRs and surveys on the area, focused on different but related aspects. In contrast, the focus that the authors have tried to give to this paper is relevant and interesting for both the academia and industry. Unfortunately, the paper has some weak points that should be considered.

Introduction:
- Footnote 1. Unfortunately, I cannot agree with the authors when claiming that they use the terms “data mining” and “data analytics” as synonyms. It is broadly accepted that both terms are rather different, and, in the case of the paper, it might cause confusion and affect readability given that DM is itself a phase e.g. within the KDD process. Please rewrite where necessary to make this different explicit. This is, in fact, mentioned by the authors in line 122 (Background), where they need to disambiguate its use.

- Murnion and Helfert (2011) does not seem to be most appropriate reference for introducing the EDM area. I suggest the following cite instead:
Romero, C., Ventura, S. “Data Mining in Education”. Wiley Interdiscip. Rev. Data Min. Knowl. Discov. 3(1):12-27, 2013; DOI:10.1002/widm.1075

- Footnote 2. The reference (Kitchenham, 2004) is relevant enough to be inserted as a cite within the text, not as a footnote. A more recent and generalizable reference could be the following:
Kitchenham, B.A., Budgen, D., Brereton, P. “Evidence-based software engineering and systematic reviews”. CRC Press, 2016.

- Footnote 4. The use of this type of non-scientific, validated, peer-reviewed cites should be extremely limited. The required formality, precision and experimental replicability is not contrasted or guaranteed in this article, even when KDNuggets is a valuable blog for the community. I do not think that the authors could use a post, though interesting, as a way to motivate a research gap that justify their paper. This is a weak point for revision.

- Line 95. Please define from the beginning the term “grey literature”, and make explicit the scope limitation of these studies for this review.

Background:
- Footnote 5 (and others). Please reference web sites correctly including at least [if available: authors, date], title, URL and last access.

- Figure 2 is important for the review and interesting for the reader. Surprisingly, no references after 2008 are considered (more than 10 years ago), as the figure is directly extracted from (Mariscal et al., 2010). Given the topic of this paper, and the need to consider recent methodologies and processes, this figure should be properly completed for this study, for instance, to consider new approaches more focused on recent approaches, platforms and technologies like Big Data (such as the paper itself begins).

-  Related to the latest point above, a major concern with this paper, and later reflected in the methodology, is that the list of considered frameworks seems to be outdated from the beginning (otherwise, it is not properly justified the reason why no other process is considered). In fact, later, the authors mostly focus the review on CRISP-DM. I agree this is the most popular process, but -considering what the SLR seems to promise- it does not have to be the only one considered, not even assume that it is the majority. All this information could effect the search. According to Table 1, ASUM is apparently the only model since 2007. If so, any reason? Besides, this makes necessary to justify the need to this SLR itself, given the lack of model developments in the field (with some exception).

Research design:
Even though I will go deeper into this section later in the "Study design", let me please comment on some general aspect on this point.

- Sometimes the paper becomes somewhat verbose, in the sense that it explains the same aspects more than once. A clear example is the research questions, which appear at least 4 times in the paper (!) At this point, it does not help readability. Please make a critical reading to avoid it. For example, I would remove RQs from the abstract, and explain the objectives in the introduction (it does not mean to explicitly write them there), as they are introduced and detailed in the section “Research questions”. This is just a suggestion. The same happens when e.g. talking about the adaptations to methodologies and models (e.g. text between 495-499 has no specific contribution).

- Line 339. Please clarify what a “version” is. To what extend has a paper to be altered, extended, updated or rewritten to be considered as a new paper, instead of a version?

- Line 347. Screening steps (Figure 4, end of page 11) should be explained before being referenced in Table 2 (page 10).

- Figure 4. It seems odd that the summary is evaluated in step 6, and not in step 4, as expected. Can a title be significant enough to exclude a paper? It seems reasonable to consider both the title and the abstract in step 4 (the abstract is provided by most databases), while step 6 focuses on other sections such as the introduction or conclusions. This is a common practice, in fact. This point affects the methodology, so could be included in section “Study design” of this review too.

- Figure 4. “599” is the number of candidate papers, and “207” the number of primary studies, right?

- Lines 379 to 393 (including Figure 5). This should not be part of the Research design. Instead, I would suggest adding a new section called “Quantitative analysis” or “Quantitative results” in order to fully explain more numerical information of interest about the findings, e.g. number of authors, number of institutions, rate industry/academia, type of uses, etc.

- “Grey literature” has to be uniquely defined. For example, in line 391, “articles in moderated and non-peer reviewed journals” are referred. How can we know that this literature meets the minimum quality criteria to be considered as part of the SLR? This type of (“grey”) literature requires attention, since it should be fully verifiable, follow the scientific method with a minimum of quality, etc. How can we ensure it all in the case of non-peer reviewed publications? For instance, have predatory journals been considered? Again, this point affects the SLR method, and (as currently justified) I cannot agree with including this type of papers. Please clarify this point.

- Line 398. Threats to the validity should be presented as a separate section. If so, the reference to this section in this line should be removed.

- Figure 6. Why were 1997-2007 and 2008-2018 chosen as periods? Is there any disruptive/significant event that could imply a more relevant cut?

Findings and discussion:
- A major issue of the paper is the lack of evidence. The authors mention some relevant facts that are not easily verifiable. For instance, in line 414, they mention that their “review led us to identify three distinct adaptation scenarios”. How? Why? Where are these scenarios from? Are there any other minor scenarios that could be discarded? Were they obtained from the literature or predetermined beforehand? A previous detailed quantitative analysis endorsing the following decisions could help the reader to understand why the authors made one decision or another. Without evidence, any point would become speculative.

- Again, in line 522, it is said that “we noted that ‘extension’ to…”. Please provide some evidence on findings.

- Figure 7. As a reader, I would appreciate to know more about how the adaptation scenarios have been conducted by methodological model (CRISP-DM, KDD…).

- With respect to RQ2, I miss a further analysis on the use, types of frameworks and industrial opportunities. The latter is, in fact, something I would suggest reinforcing and discussing in further detail. How industry has adopted/adapted different models? Benefits? Lessons learnt?

- Along the paper (more particularly in this section), it seems that CRISP-DM is used as baseline. What about other models? This is never justified (and it is marginally related to the comment above about dates and models). This point is especially notorious when responding to RQ3 (e.g. lines 494, 708, 770).

- Line 489-520. I am aware that it is a SLR, not a survey. But, in any case, it remains unclear how these modifications are really applied. To what extend could they be extended or extrapolated to other uses?

- In my particular opinion, the paper improves its readability and analysis from line 538 (subsection “Extension").

- I would suggest considering a greater difference between industrial and academic studies, industrial cases and simulations/theoretical cases. This information would be valuable to infer how these proposals have actually affected real projects and developments.

- I cannot see the contribution of subsection “Summary”, mostly referred to lines 754 to 768.

General issues and minor comments:
- In line 45, what do you mean by “in the same year”? What year, 2016?

- Please check that items in lists are consistently ended with a period (e.g. lines 129, 131, 135, 138, 143, 146, 148, 195, 198, 200, 206, etc.)

- Figures are low quality. Please re-render figures with acceptable quality. Some are difficult to read. Some also replicate the same misspelling (Figs. 9-11).
Experimental design: A strong weakness of the paper is related to the lack of replicability. This is the base of any SLR, and should be carefully considered. Unfortunately, I would not recommend this paper for acceptance if this point is not properly addressed.

- Methodology is not complete:
1. Snowballing is not conducted, or explained.
2. “Data extraction” should be a subsection itself. The authors should explain the data extraction form (conformant to the form provided as additional material), the exhaustive method to extract these data, responsibilities among authors, how conflicts were faced, etc.
3. The authors mention the SLR protocol (e.g. line 785), but the document is not available as additional material, or for download. Please provide the protocol document, which is required for replicability.

Other general and/or minor comments related to the methodology:
- Line 283. Why are terms like “process” or “process model” omitted from the search? They seem to be related with the improvement and modification of frameworks.

- There is a limited use of databases. Even when the authors try to explain why they have not used other databases like ACM, ScienceDirect or IEEExplore, the reason is not really convincing without previous pilot searches that show such a correspondence. Further evidence is required or just adding these databases.

- Why did the authors used Google Scholar, but they did not try Arxiv for grey literature?
Validity of the findings: - Structure. I would suggest adding a new section for discussion on open issues, found gaps and trends in the field, as extracted from the primary studies.

- Structure. “Threats to the validity” should be a section itself, not a subsection. (Sorry if it is like this now, but it is confusing the format without numbering sections).
Additional comments: - I suggest creating two distinct sections for the bibliography: “REFERENCES” and “PRIMARY STUDIES”. This is a common practice in SLR, and much more intuitive and readable.

- Please check references. For example, Forbes(2017) is incomplete: author’s name, title, date, etc.


Instruction: You are one of the reviewers, your task is to write a review for the article. You will be given the title of the article, the number of the round in which the article is located, and your order among the reviewers.

Input:
Title: ADAPTATIONS OF DATA MINING METHODOLOGIES: A SYSTEMATIC LITERATURE REVIEW
Review round: 2
Reviewer: 1

Output:
Basic reporting: This submission is a revised version, in which the authors seem to have dealt with most of the concerns of the reviewers.  I am seeing this paper for the first time.

This paper has provided a review of different data mining methodologies. Overall, the paper is well written and has covered well the current state-of-the-art on data mining methodologies. Although there are already many review papers on the topic, I believe that the way that the authors are discussing data mining methodologies is novel and very interesting.
Experimental design: The design of the study is correct from my point of view.
Validity of the findings: The main conclusions and findings of this work are certainly valid, and well supported by the literature.
Additional comments: -	I believe that it is worth including a note on machine learning techniques and their differences with data mining.
-	Page 3 – line 122.  I disagree with the term big data. Data analytics does not need to analyse big datasets, it could be small datasets as well. Please consider rewording this sentence.
-	Page 16 – line 510. I am not sure I agree with the KDD processes being changed because of Big Data. The workflow remains the same, but there are differences (other challenges) to be addressed in terms of preprocessing, gathering information, storing information, and mining the data. The reader would benefit from some related references in this regard.
-	Be careful with the use of double quotes, they seem to be incorrect.
-	Modelling vs modelling (please stick to American English).


Instruction: You are one of the reviewers, your task is to write a review for the article. You will be given the title of the article, the number of the round in which the article is located, and your order among the reviewers.

Input:
Title: INFLUENCE OF TWEETS AND DIVERSIFICATION ON SERENDIPITOUS RESEARCH PAPER RECOMMENDER SYSTEMS
Review round: 1
Reviewer: 1

Output:
Basic reporting: The article investigates the effect of tweets and diversification on serendipity of recommender systems in the domain of research articles. The authors motivated their study with literature review, conducted a user study with 22 subjects, which is valued in the field of recommender systems and analyzed the results with rigorous statistical methods. The article is well structured and easy to ready, but it has a number of drawbacks.
1. It would be better to bring the argument regarding researchers’ private interests page 9 (lines 322 - 327) in the introduction (introduction only discusses tweets on research interests).
Experimental design: 1. The main drawback of the study is the way the experiment was designed. The authors asked users to indicate interesting articles and artificially calculated serendipity (eq. 5) with the recommendation list generated by the primitive recommender system (p. 7, line 275). Meanwhile, the authors could directly ask users if they find a particular article serendipitous.
Please indicate explicitly in introduction and abstract that serendipity has been calculated artificially (based on primitive recommender system) and not by asking users.
Validity of the findings: 1. The way serendipity was calculated in equation 5 (p. 7) does not correspond to the definition of serendipity given in the abstract, as this equation does not take into account novelty of the article (users were not asked about novelty of articles either). Please either change the definition of serendipity used throughout the article or change the way it is calculated.
2. RQ2 is missing an important reference, which discusses the same research question in the movie domain:
Kotkov, D., Veijalainen, J., & Wang, S. (2018). How does serendipity affect diversity in recommender systems? A serendipity-oriented greedy algorithm. Computing, 1-19.
The discussion section should include comparison of the obtained results with the results obtained by other studies.
Additional comments: Overall, the study is well reported and contains contributions important for the field. After the mentioned drawbacks have been corrected, the article can become a good candidate for publication.


Instruction: You are one of the reviewers, your task is to write a review for the article. You will be given the title of the article, the number of the round in which the article is located, and your order among the reviewers.

Input:
Title: INFLUENCE OF TWEETS AND DIVERSIFICATION ON SERENDIPITOUS RESEARCH PAPER RECOMMENDER SYSTEMS
Review round: 1
Reviewer: 2

Output:
Basic reporting: In this paper, the authors explore the serendipity of the paper recommendation system. The serendipity can be calculated by a summation of the rating divide by the number of unexpected items of the recommendations. The authors apply a variety of content-based recommendation systems using publication content and twitter to recommend publications. Content used for recommendation can be selected from the author's paper or twitter. The recommendation is calculated using multiple calculation techniques such as TF-IDF, CF-IDF, and HCF-IDF. And the distance metric is selected from cosine distance (rank by similarity) and IA-select (to diversify the search result).

They aim to (1) find the impact of twitter on a content-based recommendation while measure the serendipity score outcome and (2) see if algorithms like IA-select which can diversify the search result can improve serendipitous recommendation.

Overall, the paper explains the task and result quite well. However, it lacks quite a bit of a literature review in the introduction section. For example, there are a papers that present content-based recommendations such as Science Concierge (https://journals.plos.org/plosone/article?id=10.1371/journal.pone.0158423) and User Profile Based Paper Recommendation (https://ijisae.org/IJISAE/article/view/752). Lastly, adding more details in the experimental section can improve the paper. For example,  I would like the author to add one a section on how do they present the recommendation result sequentially to the subject i.e. which strategy from 1 to 12 gets presented first to last.

My main criticism of the paper is the usage of twitter profiles in the experiment. It can be the case that the users tweet about the publication with relatively short text with and extra of publication's URL. I assume that if all tweet text and content from the tweet URL are included in the recommendation engine, it might give a better result for twitter. This is a little hard to re-experiment but it can be noted in the discussion section.
Experimental design: - The subjects are well recruited. However, I think that the subject size is a little bit too small (n = 22) in the experiment given that the task can be finished relatively quickly in less than 10 minutes.
- As mentioned above, I would like the author to add one section on how do they present the recommendation result sequentially to the subject i.e. which strategy from 1 to 12 gets presented first to last.
- The dataset could include concepts extracted from twitter and the author can also visualize using the histogram for the length of the text from Twitter. It is a bit hard to know the distribution of tweets length in all the subjects seeing just average length and its standard deviation.
Validity of the findings: The finding is very interesting and novel in a sense of outcome measurement of serendipity. I agree that the usage of twitter alone might not help improve the recommendation engine much.

The current number of subjects is quite small. If possible, by increasing the number of subjects, it will make the paper much stronger.

The authors also only explore the publications in the computer science field. I think it would be good if they can clarify in the discussion section that it has to explore more on different fields such as biomedical science (Pubmed/MEDLINE), social science, or economics.
Additional comments: No additional comments


Instruction: You are one of the reviewers, your task is to write a review for the article. You will be given the title of the article, the number of the round in which the article is located, and your order among the reviewers.

Input:
Title: INFLUENCE OF TWEETS AND DIVERSIFICATION ON SERENDIPITOUS RESEARCH PAPER RECOMMENDER SYSTEMS
Review round: 2
Reviewer: 1

Output:
Basic reporting: For instance, Kotkov et al.  (2016) defined serendipity as “a property that indicates how good a recommender system is at suggesting serendipitous items that are relevant and unexpected for a particular user.”  - the direct quotation does not correspond to the original text of the cited article.
The authors should provide arguments for choosing this particular definition of serendipity.
There is a study that provides the effects of different definitions of serendipity on users:
Kotkov D. et al. Investigating serendipity in recommender systems based on real user feedback //Proceedings of the 33rd Annual ACM Symposium on Applied Computing. – ACM, 2018. – С. 1341-1350.
Experimental design: Experimental design is well explained
Validity of the findings: The findings are valid
Additional comments: No additional comments


Instruction: You are one of the reviewers, your task is to write a review for the article. You will be given the title of the article, the number of the round in which the article is located, and your order among the reviewers.

Input:
Title: ADAPTIVE DIVERGENCE FOR RAPID ADVERSARIAL OPTIMIZATION
Review round: 1
Reviewer: 1

Output:
Basic reporting: The paper is clear and well written. It does a good job of contextualizing the work w.r.t. to prior work. The paper is well structured, with results presented appropriately. The paper can be read and appreciated on its own. Theorems are presented clearly.

A section that could be improved is 5.2. For people unfamiliar with the Pythia program - which may be the majority of Machine Learning readers - this is not very understandable.
Experimental design: The paper is well structured, clearly defines a problem and the suggested approach. A sufficient number of experiments is run, though more would make for a stringer case. The methods are defined, though more details or looking at the source code release will be required for replicability.

I was able to download the code, however, I am not familiar with Pythia so could not test the code appropriately.
Validity of the findings: The findings have reasonable evidence as to their validity.

Since the code is provided and the results are simulation based, the findings should be robust and statistically sound.

Figure 5 seems to have no uncertainty intervals, as opposed to the other results. Was this run only once? Further experiments may be useful, though this step seems computationally expensive.

I am not sure if the results support the claimed hypothesis. While the paper mostly talks about reducing the number of required data points to be collected, the results support a different claim that AD improves the optimization surface so that better optima are found for the same number of data points. These claims are related but not the same.
Additional comments: The paper does not do a great job of discussing or explaining various choices made in the experiment. Overall, the paper is very interesting but it may be too terse in places and expanding on why choices were made or expanding on hypothesis as to why certain results were observed would make the paper much more valuable.

For example,
1. in D_{\alpha} where does the log 2 or the linear scaling with \alpha come from?
2. where exactly is the Bayesian Optimization coming in? I can kind of figure it out, but it isn't very clear from the definitions of e.g. Algorithm 3 and Algorithm 4.
3. The jump to dropout, layer norm and other optimization tricks (e.g. link 296) is stark, and many details are missing here. While I recognize that lots of tricks are required to get Adversarial models to train well, more details are discussion of the choices made here would be useful to the reader.
4. In the figures, it was not clear to me how 'distance to the solution' is defined. Is this the MSE between parameter configurations or JSD or something else?
5. Line 268 : with the only difference ... -> what is the significance of this difference?
6. Line 228 : as computational ... -> it would still be useful to comment on the other resources consumed and provide plots where possible.
7. Line 235 : As a  significant ... -> This seems very important, and much of the validity may rest on this choice, but I do not understand it at all. Please expand here.
8. Line 296 : Note that .. -> This is an interesting conclusion and should be discussed in more detail.
9. Line 207 : dropout ... -> why this choice?
10. Line 216 : Note that a ... -> Is this true? Dropout ~ L2 regularization so I imagine it could still be a AD as defined in this paper.


Instruction: You are one of the reviewers, your task is to write a review for the article. You will be given the title of the article, the number of the round in which the article is located, and your order among the reviewers.

Input:
Title: ADAPTIVE DIVERGENCE FOR RAPID ADVERSARIAL OPTIMIZATION
Review round: 1
Reviewer: 2

Output:
Basic reporting: The authors provide a thorough exposition into Adversarial methods for generative modeling and black-box optimization.  In this work, Adversarial methods motivate methodologies where data gathering is expensive, necessitating a restriction to weaker discriminators that can be trained with fewer samples.

The authors identify the relationship between the Jensen-Shannon divergence and the objective function of the minimax objective function of Goodfellow’s original GAN paper.  More specifically, given a perfect discriminator function, D, the generator, G, is tasked with minimizing the Jensen-Shannon divergence (JSD).  (The reviewer would prefer this to be made more explicit in the paper.)
Experimental design: To demonstrate the monotonic scaling of pseudo-divergence with model capacity, the authors provide experimental results on a Gaussian-mixture model and the XOR dataset.  (The referee would prefer there to be a brief discourse in observed cases where monotonicity is violated.)
	The authors then use their approach to show improved sample-efficiency on hyper-parameter estimation in computationally expensive physics simulations. By replacing the JSD with the adaptive metric, the authors’ methods achieve much quicker convergence.
Validity of the findings: For a model with finite capacity, however, the cross-entropy loss can only approximate (Jensen-Shannon divergence – log(2)).  In fact, the authors propose a family of pseudo-divergences that are parameterized by model capacity.  Furthermore, the single-parameter divergence increases monotonically, limiting to JSD as this parameter approaches unity.
	The novelty in this approach comes from the methodology in choosing a minimally expressive discriminator for a given pair of ground truth and generator distributions.  The problem is posed such that a discriminator will always be found.  Additionally, the search over discriminators starts by optimization of the lowest capacity model.  As the estimation becomes more difficult, due to the increasing similarity of the two distributions, the network capacity of the newly trained discriminator increases.
Additional comments: Equation (1):
There should be a slightly expanded exposition about how optimal discriminator in the minimax game leads to the JSD.  The second equality is a large jump that requires the definition:
 D* = p_{data} / (p_{data} + p_{gen})
Section 5:
Are there any particular examples where the monotonicity does not generally hold?  It seems this would occur when all discriminators yield pseudo-divergences near zero.  It seems worth a few sentences, if there are concrete experiments yielding this behavior.

Figures 2 + 3:
What is the definition of “distance to the solution”?  Is it the L2 norm of the difference between the ground truth and estimated vector of parameters?  This should be made more clear.
Is the experimental time / FLOPS linearly proportional to “examples sampled” when comparing the different methodologies?


Instruction: You are one of the reviewers, your task is to write a review for the article. You will be given the title of the article, the number of the round in which the article is located, and your order among the reviewers.

Input:
Title: SANGO: A STORAGE INFRASTRUCTURE SIMULATOR WITH REINFORCEMENT LEARNING SUPPORT
Review round: 1
Reviewer: 1

Output:
Basic reporting: Numbered points are in order of precedence, bullets signify points to be
addressed in no order of precedence.

Minor Spelling and Grammar issues (indicative, non-exhaustive):

- 22: Replicated effects should be clarified
- 62: "method of hybrid simulation" -> "The hybrid simulation technique"
- 101: Replace (Fishman) with the reference
- 107: "attributed metric information" -> "extra metadata describing attributes"
- 133: Reference missing (can be found in ?)
Experimental design: 1. The data required for the illustrative example needs to be provided, along with details of how the data was obtained.
Validity of the findings: Numbered points are in order of precedence, bullets signify points to be
addressed in no order of precedence.

1. 145: The Arrhenius model is not justified in this context. The reference
   is not of a journal either, and also does not account for the variation in
   the chemicals and junctions used in computer circuits.
2. 155: Similar concerns exist, however modeling these effects with an
   exponential is more reasonable due to the very limited variation of these
   factors in production.

The authors are directed to the following:

- https://ieeexplore.ieee.org/abstract/document/62568
- https://ieeexplore.ieee.org/abstract/document/275221
- https://ieeexplore.ieee.org/abstract/document/4292056
- https://www.springer.com/gp/book/9780387257624

Furthermore, there are several concerns with modeling temperature in this
manner. The nature of the Arrhenius equation, as well as the operating
temperature loads of most IC devices do not correlate well. For the Arrhenius to
be justified, the correlation between the activation energies of the
semiconductor chips used in the datacenter model correspond to the operating
temperatures.
Additional comments: The software URL (https://gitlab.com/lambda-hse/sango) is not publicly
accessible. This makes the software more complicated to test (via the codeocean
capsule). This needs to be rectified, especially since the codeocean capsule
lists the software under the permissive GNU-GPL_v3. Furthermore, the gitlab URL being inaccessible has the unintended effect of invalidating the installation
instructions. Though not necessary for this publication (nor has it factored in
my recommendation), a good API documentation should be ideally present. The
`bib` file should have been sanitized before usage, possibly by filtering via
Zotero, Mendeley, Jabref or others. More details of the data used is required.
The specific usage of OpenAI models should also be clarified.

The code is well written, and the work presented is worthy of inclusion.
However, given the severe misgivings regarding the modeling of environmental
effects, in its current form this is not ready for publication.


Instruction: You are one of the reviewers, your task is to write a review for the article. You will be given the title of the article, the number of the round in which the article is located, and your order among the reviewers.

Input:
Title: SANGO: A STORAGE INFRASTRUCTURE SIMULATOR WITH REINFORCEMENT LEARNING SUPPORT
Review round: 1
Reviewer: 2

Output:
Basic reporting: The authors should demonstrate an adequate understanding of the relevant literature in the field and cite an appropriate range of literature sources.
The paper contains significant information adequate to justify publication.
Experimental design: no comment
Validity of the findings: no comment
Additional comments: The authors should check again the paper. For example, on page 8 the number of the table that shows the full flag list is not mentioned. The conclusion should be checked too.


Instruction: You are one of the reviewers, your task is to write a review for the article. You will be given the title of the article, the number of the round in which the article is located, and your order among the reviewers.

Input:
Title: SANGO: A STORAGE INFRASTRUCTURE SIMULATOR WITH REINFORCEMENT LEARNING SUPPORT
Review round: 2
Reviewer: 1

Output:
Basic reporting: No comment
Experimental design: No comment
Validity of the findings: No comment
Additional comments: I thank the authors for the improvements to their initial manuscript.


Instruction: You are one of the reviewers, your task is to write a review for the article. You will be given the title of the article, the number of the round in which the article is located, and your order among the reviewers.

Input:
Title: SANGO: A STORAGE INFRASTRUCTURE SIMULATOR WITH REINFORCEMENT LEARNING SUPPORT
Review round: 2
Reviewer: 2

Output:
Basic reporting: No comment
Experimental design: No comment
Validity of the findings: No comment
Additional comments: The paper contains significant information adequat to justify its publication


Instruction: You are one of the reviewers, your task is to write a review for the article. You will be given the title of the article, the number of the round in which the article is located, and your order among the reviewers.

Input:
Title: EXACT ACCELERATION OF COMPLEX REAL-TIME MODEL CHECKING BASED ON OVERLAPPING CYCLE
Review round: 1
Reviewer: 1

Output:
Basic reporting: This paper develops an extension of the parking-cycle acceleration
technique for timed automata to consider multiple overlapping cycles.
The extension is shown to be exact with respect to reachability
properties, and supporting experiments show that it performs superior
to the standard parking-cycle technique, and also to the
appended-cycle method, and permits application of timed-automata model
checking to systems which hitherto were too complex for standard
timed-automata model checkers.

The paper is reasonably well-written and carefully covers previous
work on accelerating cycles as well as possible applications to
real-world IoT systems.  However, the papers is imprecise in some
important places, hence I recommend a major revision.  I do believe
that all results in the paper are correct, but in order for me to be
sure, some technicalities need to be stated much more precisely, see
below.

Major trouble:

1.
Both your abstract and your introduction and conclusion give the
impression that you here introduce a completely new technique for
accelerating cycles.  I would dispute this: in my opinion, you extend
the parking-cycle technique to take into account multiple overlapping
cycles which share their reset location.  Now this is a good
contribution to TA model checking; I do not dispute this.  But it
would help to pur your work in context to know from the outset that
you extend the parking-cycle technique rather than introducing a
completely new technique.

Please change your abstract, introduction and conclusion to indicate
that you extend the parking-cycle technique.

2.
Def.3 is not at all clear.  Please define precisely what are the
valuations v_i, v_i' in your trace.  Define precisely the notion of
"total delay".  What does it mean to "adjust the delays" in T_r(E_c)?

This defines the window of an acceleratable cycle, which is basic for
all which is to come; rather important to get this straight...

3.
In Thm.2 you introduce a notion of "obtain[ing] a continuous clock
zone" which you do not define precisely.  I don't understand what it
means, and everything after seems to depend on this notion, so this
inhibits me from checking all proofs after Thm.2.

Please define carefully what it means to "obtain a continuous clock
zone".

Minor details (numbers are line numbers):

49f: I did not understand "however, these ... test completeness."

62: "...when a time*d*"

114ff: You say twice here that "Larsen proposed/modeled", citing
papers with multiple authors.  Now Kim is a nice guy and does a lot of
important work; but you should give his co-authors credit and rather
say "Boudjani et al proposed" and "Chadli et al modeled".

174: "... cycle *may be* exited."

187: "... can be present*ed* as ..."

188: "... form*ula* ..."

197: What do you mean "clock y grows more gradually"?

200: "... when the time*d* ..."; what do you mean "detects a symbolic
model"?

212f: You use the same "c" in {x\le c} and {x\ge c}; but they should
be different

253: "... pro*ved* ..."

259: You have not introduced the notion "global clock".  Please do so

270: "... permi*tted* to"

275: The reader would expect here that an analogue of Thm.1 holds for
the parking-cycle enhanced timed automaton.  This is Thm.3 below, but
maybe you should announce it here already

331ff: I'm a bit surprised that you don't refer to Yin, Zhuang, Wang
2011 for this proof.  You say that parking cycles have been introduced
in that paper; but why, then, is Thm.3 not already in that paper? What
worth is the parking-cycle technique without a proof that it is exact?

336f: What do you mean "satisfy any reachable state"?

356: "... timing /of/ and ..." (delete "of")

361ff: Some imprecision here: what does it mean to be "more
effective"? Also, you should rather say "there is a cycle which is
most effective" or make precise which

372: What do you mean "a_j+a_i\ge \forall(a_j, a_k)"?

419f: Please say which cycle is deadlocked and discarded

495: "automat*a* ..."

486: "... when model*ed* as ..."

504-535: It is not necessary to spend so much space on explaining the
IoT example; this is not the main focus of the paper.  Please shorten

537: "... be*l*ow." (only one "l")

586: It is not clear from reading your paper whether your analysis of
overlapping cycles has been automated or whether you do this
manually. Please specify
Experimental design: no comment
Validity of the findings: no comment
Additional comments: no comment


Instruction: You are one of the reviewers, your task is to write a review for the article. You will be given the title of the article, the number of the round in which the article is located, and your order among the reviewers.

Input:
Title: EXACT ACCELERATION OF COMPLEX REAL-TIME MODEL CHECKING BASED ON OVERLAPPING CYCLE
Review round: 1
Reviewer: 2

Output:
Basic reporting: ## Summary
The paper is concerned with a problem in model checking of timed
automata containing cycles in the discrete graph. The problem is that
the cycles results in a fragmented zone-graph - this paper tries to
syntactically the timed
automata such that reachability properties are preserved while
minimising fragmentation. In the litterature this kind of
fragmentation reduction is called acceleration and has perviously been
studied by numerous authors.

- Larsen and Hendriksens method called appended cycles basically
  identifies cycles that can be accelerated, unrolls the cycles a
  number of times abd appends the "unrolled" version of the cycles to
  the automaton.

- Yin, Zhuang and Wangs methods is called parking cycle. In this methd
  the cycle is not unrolled - instead it is identified when the zones
  visited by the cycles becomes continuous. At this point a cycle is
  appended to the cycle in which time can delay infinitely long (this
  is the parking cycle).

The above methods focuses on accelerating just one cycles and
overlooks the fact that sometimes multiple cycles can be accelerated
from the same location. This special case is considered in this
paper. In particular their results indicate that accelerating just one
cylce (with the parking cycle technique) is better than accelerating
all of them.

The paper is concluded with an experimental section documenting the
effects of acceleration involving both Uppaal and Kronos.

## Structure of the paper
The paper is fairly well-structured first discussing the state of the
art, and then the results of the paper itself. One big annoyance of
mine is that all of the figure and tables are at the end of the paper
(even after the appendix). I am not sure if this is caused by the
template of PeerJ or the authors selected this - anyway I do find
it annoying. Furthermore, given the fact that the figures are all at
the end of the paper, the authors ought to provide proper references
(i.e. Figure 1 instead of Timed Automaton M).

It's not always completely clear when a proof is finished. (no visual
aid to separate the proof from the running text)

## Language
The language of the paper is okay, but definitely not perfect. As
an example I needed subtstantial time for understanding what the
"window of a cycle" ( Definition 3) was,  simply because I did not
understand the explanation. To my understanding, it is the  minimal
and maximal time it may take to pass through a cycle.
Experimental design: The experimental evaluation consists of the evaluation of a small
toy-example for which results show that the acceleration techniques
does indeed reduce the time verification time. I can however not see
this very clear indication of higher memory consumption of the
appended cycles technique. In fact,  all of the techniques (appended
cycle, parking cycle and this papers technique) use around 30000K.

A larger IOT  model was investigated as well. For this model it is
again revealed that  acceleration is quite nice for reducing the time
and in particular that the authors technique is better than the
parking and appending cycle techniques. Again the authors talks about
the memory consumption but results show that with and without
acceleration the verification uses around 30000K. THis is strange and
 I would almost think, that something has gone wrong with the memory
 measurement.

A curiousity in regards to the evaluation is, that the paper says the
IOT case study was conducted using UPPAAL and Kronos.  I find this
curious for two reasons:
1. To my knowledge Kronos has been discontinued for many years, and
2. The models presented in the paper are all Uppaal models - likewise
are the models in the supplementary material all uppaal models.
Validity of the findings: ## Correctness
In regards to whether the results are correct, I cannot tell. The
proofs seem okay most of them, however they  also depend on
concepts that are not really defined in the paper (e.g. continuous
clock-zone). Furthermore, quite a number of the proofs depend on
Theorem 2 which might be true, but the proof of Theorem 2 proves its
correctnes by assuming it is correct (see detailed comments later).

##  Theorem 2
As noted earlier this proof appears to be proven by assuming the
theorem is true. According to the Theorem it wants to prove that if
you have an acceratable cycle with window [a,b] with a < b then
there exists a number n such that after n executions of the cycles,
the clock zone becomes continuous. The proof starts out by saying,
"Let n be the rounds of cycle executions when the reset location
obtains a continuous clock zone....", but we want to prove that n
exists (thus the authors assume the theorem to be true while proving
it).
Additional comments: ## Overall Evaluation
Presently I do not see this paper as publishable. It does detail
interesting results, but I am unable to validate the correctnes of
proofs  - mainly because they depend on Theorem 2 which I do not trust
the proof of. Furthermore, the experimental evaluation is lacking in
regards to explaining its methodology (which tools (Kronos/Uppaal) was actually used,
how was memory measured etc.)