Instruction: You are one of the reviewers, your task is to write a review for the article. You will be given the title of the article, the number of the round in which the article is located, and your order among the reviewers.

Input:
Title: FUSION OF TEXT AND GRAPH INFORMATION FOR MACHINE LEARNING PROBLEMS ON NETWORKS
Review round: 1
Reviewer: 2

Output:
Basic reporting: The English language should be improved to ensure that your international audience can clearly understand your paper. I would suggest avoiding informal language in the manuscript.
TYPOS:
Add parentheses and commas to all references in the paper. For example, in line 33: matrices Perozzi et al. (2014) Grover and Leskovec (2016) -> matrices (Perozzi et al. (2014), Grover and Leskovec (2016))
I would add some references to each mentioned problem in the first paragraph of the introduction;
Line 29: Its main idea to map -> Its main idea is to map;
Line 29: low dimensional space -> low-dimensional space;
Line 41: semantic of words, using their context -> semantic of words using their context;
Line 43: which makes it more powerful -> which makes this method more powerful;
Line 71: structure -> structures;
Line 79: from word -> from a word;
Line 84: an approach allow -> an approach allows;
Line 89: Doc2Vec additionally create -> Doc2Vec additionally creates;
Line 89: When target word -> When a target word;
Line 92: BERT Devlin et al. (2018) the main idea is -> The main idea of BERT (Devlin et al. (2018)) is. It is better to rephrase this sentence in full.
Line 94: instead if employs -> instead it employs;
Line 95: SBERT Reimers and Gurevych (2019) framework propose -> SBERT framework (Reimers and Gurevych (2019)) proposes;
Line 110: embeddings (and vice versa) -> embedding and vice versa;
Line 114: TADW Yang et al. (2015) algorithm -> TADW algorithm (Yang et al. (2015));
Line 145: It’s important -> it is important. “It’s” is informal English;
Line 196: and therefore show much -> and therefore, show much;
Line 202: a quite high model -> a quite high, model;
Line 203: Therefore it is -> Therefore, it is;
Line 204: However this is -> However, this is;
Line 205: and therefore can afford -> and therefore, can afford;
Line 246: in most -> In most;
Line 251: Also it interesting -> Also it is interesting;
Line 253: In this case padded -> In this case, padded;
Line 255: mean and max functions applied -> mean and max functions must be applied;
Line 259: firstly -> Firstly;
Line 273: the results is reported -> the results are reported;
Line 276: remove the upside down question mark;
Line 303: becomes small -> becomes smaller;
Line 313: One can that -> One can note that. It is better it replace it with some synonyme because you already used this phrase in the last paragraph;
Line 385: one mask edges -> one masks edges;
Line 421: Therefore the embeddings dimension size -> Therefore, the embedding dimension size;
Line 423: embeddings algorithms -> embedding algorithms;
Line 423: obtained embeddings are to be mapped -> obtained embeddings are mapped;
Line 430: results -> result;
Line 432: Deep Walk’s visualization -> Deep Walk method;
Line 435: TADW shows results that are very -> TADW show the result that is very;
Line 439: the best results -> the best result;

English must be significantly improved. My suggestions are following:
Line 18:  a user people -> users;
Line 68: in order to be consistent use either “Word2Vec” as in the line 41 or “word2vec” everywhere in the paper;
Line 128: you have not specified anywhere in the paper for what the abbreviation LPP stands. Again, in order to be consistent either use node classification problem or Node Classification problem as in the line 22;
Line 129 and 130: the sentence “GCN simply applies the adjacency matrix...” is imprecise. You can not apply the adjacency matrix for something. So I would suggest to change this part and add more details to the explanation of GCN architecture;
Line 165: move the reference “Pan et al. (2016)” to the end of this sentence;
Line 195: it is unclear what LDA algorithm is. You did not mention it early in the paper, so it would be good to add some explanation.
Line 199: the sentence “When trained...” is not quite clear to me. Which two variations did you mean there? And what does “dimension equals to d and 64” mean?
Authors use a phrase “it is sensible to...” in the paper. I am not sure that it is the correct phrase. Instead, use: it makes sense/be worthwhile;
Line 205: replace “still” with more formal words such as however or nevertheless and start a new sentence.
Line 208: I would split this sentence and start a new sentence with “vector size...”;
Line 224: I believe instead of “three network embeddings” should be “three network embedding methods/techniques”;
Line 226: for me it is not clear in which settings the methods tend to outperform others;
Line 260: replace “After that” with a synonyme, since you use it too often in the paper;
Line 264: split into train test -> split into train set and test set / split into train and test sets;
Line 270: I guess regression classifier is trained on the learned embedding of graphs in the train set but not on the train set itself;
Line 291: the by far best model -> By far the best model;
Line 303: What is the absolute number of samples? I did not understand what the word “absolute” meant in this context;
Line 315: I would rephrase this sentence or at least not use the word “perform” twice in one sentence;
Line 317: DBLP dataset Table 6 one can see, DBLP dataset one can see in Table 6;
Line 318: for me it is unclear what you mean by stability. You could add some explanation;
Table 6: why did you bold the result in Sent2Vec pretrained/50%?
Line 337: as far as I understand here begins the results for graph embedding methods, so I would add some paragraph title. It is difficult to follow the paper;
Line 342: that for Cora network data -> that the Cora network. Moreover, the last sentence is hard to follow, so I would rephrase it in full.
Line 356: Aain I would add a paragraph title to separate results for graph embedding methods, text embedding methods and fusion methods.
Line 385: the same, add a paragraph title.
Line 393: the same, add a paragraph title, for example, “Result for Link Prediction problem.”
Line 398: the sentence “It is quite predictable...” is not clear for me.
Line 399: I would not use the word “though” in the scientific paper;
Line 422: I would use the word “entire” instead of “whole”, since it is more formal;
Line 424: add a reference for T-SNE algorithm;
Line 426: “In the case of GCN activations of the first convolutional layer” is not fully clear. What are GCN activations? Did you mean activation functions?
Line 429: Again, I would not use “though” in the paper. You can write: “However, there are some exceptions...”
Line 439 and 440: it would be very easy to make a clusterization -> it would be very easy to cluster points.
Line 448: “what is more” is a more informal way of saying, so use formal synonyms such as moreover, furthermore.
The last sentence in the discussion is too long. I would split it into two or three sentences.
Experimental design: In general, experiments are designed in a good way but sometimes experiments lack some details.

In the Validation section, I would add more details about what your input data is. Moreover, in line 260 you split your data into train and test sets for Node Classification problem and then in your tables you consider different percentages of node labels. So I think you could add in Section 3.5.1 that you consider different ratios of node labels and the same for Link Prediction problem;
In line 259 it is stated that the input data is preprocessed. How did you preprocess data? Is this preprocessing step described in Section 3.2 for text data? Or did you preprocess differently? If yes, then it would be better to add details. But then I did not find how you preprocessed the graph-structured data. Also, I did not understand the part “nodes of the network are mapped into latent space using the input information”. What kind of input information exactly? Do you mean the input data which you mentioned at the beginning of this sentence?
Line 269: What are xu and xvin your case? Add more details;
Validity of the findings: The experiments compare the accuracy of different fusion methods which allows the audience to comprehend how the text information could impact the results of node classification and link prediction problems.
The data, which is used in the paper, is well-known benchmark graph-structured data available online.
All conclusions are linked to original research questions and results are well-described.
Additional comments: No additional comments


Instruction: You are one of the reviewers, your task is to write a review for the article. You will be given the title of the article, the number of the round in which the article is located, and your order among the reviewers.

Input:
Title: FUSION OF TEXT AND GRAPH INFORMATION FOR MACHINE LEARNING PROBLEMS ON NETWORKS
Review round: 2
Reviewer: 1

Output:
Basic reporting: I suggest that the authors do another proofreading to improve the text. Most importantly, articles (i.e., a, an, and the) are largely abused in this paper, please correct these grammar issues.

Here are some obvious typos in Sections 1 and 2:
Line 18: the(a) great variety of
Line 45: (A) More advanced approach
Line 73: a(remove) text information
Line 78: remove "."
Line 107: learn to embed -> learns embeddings
Line 110: (a) more efficient, (the) random walk idea
Line 115: (an) important property
Line 129:  to use them(remove)
Line 137: the algorithm learn(s)
Line 142: link prediction problem(s) or node classification(s)
Line 147: Graph Attention n(N)etworks (GAT), (the) self-attention mechanism
Experimental design: The experimental design is good, with detailed settings.
Validity of the findings: The results seem to be comprehensive, and the authors present detailed explanations.
Additional comments: No additional comments


Instruction: You are one of the reviewers, your task is to write a review for the article. You will be given the title of the article, the number of the round in which the article is located, and your order among the reviewers.

Input:
Title: FUSION OF TEXT AND GRAPH INFORMATION FOR MACHINE LEARNING PROBLEMS ON NETWORKS
Review round: 2
Reviewer: 2

Output:
Basic reporting: The paper was significantly improved. I like the structure of the paper now. It is easy to read  the paper and follow the main ideas.

I have some minor remarks but in general, I think, the paper can be accepted:

- I would use a word “graph: everywhere throughout the paper instead of mixing “graph” and “network” words.
- Everywhere in the paper, you use "the loss" such as in line 56 or 155. Instead, I would use "the loss function".
Line 65: and models choice -> and the choice of models;
Line 73: focus is in -> focus is on;
Line 154: GIC leverage - > GIC leverages;
Line 178: the better the quality -> the better the quality is.
Line 277: I would use either logistic regression or Logistic Regression as in the line 394.
Experimental design: Now all methods are described in a good clear way.
Validity of the findings: All results of the experiments are analyzed clearly and with sufficient details.
Additional comments: No additional comments


Instruction: You are one of the reviewers, your task is to write a review for the article. You will be given the title of the article, the number of the round in which the article is located, and your order among the reviewers.

Input:
Title: FUSION OF TEXT AND GRAPH INFORMATION FOR MACHINE LEARNING PROBLEMS ON NETWORKS
Review round: 3
Reviewer: 1

Output:
Basic reporting: Except for the problem of Figure 1, the updated version look good now. Please update the figure.
Experimental design: N/A
Validity of the findings: N/A
Additional comments: No additional comments


Instruction: You are one of the reviewers, your task is to write a review for the article. You will be given the title of the article, the number of the round in which the article is located, and your order among the reviewers.

Input:
Title: EVENT DETECTION IN FINANCE USING HIERARCHICAL CLUSTERING ALGORITHMS ON NEWS AND TWEETS
Review round: 1
Reviewer: 1

Output:
Basic reporting: In general, the manuscript reads very well. The English (wording, grammar) is spot-on and the authors have clearly made an effort to make everything as clear as possible. The article is well-structured according to the standards and seems to be well-balanced in terms of lengthiness of the separate sections. To my taste, you could consider separating Section 4 into implementation and experiments, and subsequenlty each experiment into set-up and results, just to clarify the article even further and improve readability. Figures are of good quality and provide additional clarity to the story. Minor remarks: the explanation of boxplots in Figure 12 is redundant and can be removed. Positioning of the figures and tables is also a bit awkward at times, but can be fixed in the typesetting / editing phase.

The literature review seems thorough, although I am missing some relevant work that is targeted to the actual implementation of event detection in financial algorithms, e.g.:
- A News Event-Driven Approach for the Historical Value at Risk Method: Hogenboom et al. in: Expert Systems With Applications, 42(10):4667-4675, 2015.
- An Automated Framework for Incorporating News into Stock Trading Strategies: Nuij et al. in: IEEE Transactions on Knowledge and Data Engineering, 26(4):823-835, 2014.

In fact, the authors do not make a clear distinction between "historical" and "real-time" detection of financial events. It would be nice if this is incorporated into the literature review.

The work is self contained. I would like the differences with the 2020 conference paper to be explicitly stated in this article.
Experimental design: The title and abstract seem to indicate a focus on event detection. However, the article mainly focuses on the clustering part of the problem. Alert generation (which in my view are the "actual" relevant events) is only a small, final step.

The research question remains implicit, and I would like to see it defined explicitly. The contributions, however, are very clear. The NLP-driven clustering-based approach, to my knowledge, is novel in the (financial) event detection scene. The definition of events as clusters of news items / tweets makes sense, and the fact that they are based on extracted words seems logical to me as well. The fact that the authors additionally take into account time is essential. Moreover, I like the fact that domain-specific lexicons are created ad-hoc, as most existing approaches in fact make use of pre-defined lexicons, with or without an update mechanism. Therefore, it would be nice to improve the lexicon generation description in Section 3, as it is not explained very well. For example:

"For each news article in this set, we extract the text, consisting of the title, the snippet and the full body
of the article, and then we perform some standard pre-processing techniques on it, such as stop-words
removal, stemming and tokenization. In addition, we remove from the corpus all the words that appear
too frequently and too infrequently, according to given tolerance thresholds."

These sentences are a bit puzzling. Which algorithms are used for pre-processing? Later on, in Section 4, I understand the authors are using standard Python packages, but they have to make sure they know which algorithms are implemented. As methodology and implementation are in separate sections (which I think is very nice), I would suggest the authors already include the applied algorithms in the methodology section, and simply state in Section 4 that the algorithms are implemented using the Python packages.

The alert generation is flawed, as these are only based on the number of mentions in tweets / news messages. I understand that this is an indication of the importance of the event, but large clusters can in fact be very general clusters. The number of mentions is directly related to the number of clusters. Secondly, why do the authors make use of today's number of mentions, but ignore next day's stock rate differences? They are an important feature of the clusters, so why not here? To me, it would make sense that hot events have a larger impact on stock rates than regular events.

Finally, it is worth mentioning that in general, the methodology is described rigorously, which enhances the reproducibility of the research sufficiently.
Validity of the findings: Experiments performed in the article are thorough and are based on a large timespan of data. I am happy to see that the authors not only performed a quantitative, but also a qualitative analysis. In my view, the latter is absolutely necessary. One thing that remains unclear, is how the real-world (ground-truth) events have been determined. Did they determine the events based on the output clusters? Or did they create them beforehand? Is there an inter-annotator agreement or the like?

One major thing that is lacking, is an evaluation of the performance against other state-of-the art approaches. I think this is crucial for the acceptance of this paper. Other than that, the evaluation of the methodology is very thorough and statistically sound, though.

The code is available in a GitHub repo, which is very nice. At first sight, the data seems to be available, but I have problems identifying the full content. Some folders mentioned in the readme, i.e., "lexicons" and "word2vec_data" are missing. I believe for the former folder, the generation script has to be executed, and for the latter folder, large external datasets have to be downloaded. It would be nice to create stub folders in the repo to avoid some confusion.

As the research question remains a bit vague, the conclusions are also quite general. I am also left with questions, i.e., does this easily port to other languages? Or is the approach heavily dependent on available word2vec models? How does the performance stack up against existing work? Does the proposed algorithm scale well? What kind of applications are envisioned?

Future work suggestions are valid. Especially temporal aspects are worth looking into. If by Semantic Web technologies the authors mean ontologies, I also believe they could help out. In essence, you are including predefined domain knowledge, which should balance out some of the artifacts of word2vec-like approaches. However, you have to keep in mind the performance hit you might suffer.
Additional comments: The proposed method seems to heavily rely on Python packages. Have the authors thought about scaling issues? For instance, the document-term matrices that are generated suffer from the curse of dimensionality. What is needed for gigabytes or petabytes of data? Have they considered cluster-computing, or even map-reduce? What kind of implications are there for performance?

The applicability of the research in practice is a bit questionable, as the analysis is a historic one. It is not easily changed either, because one of the main clustering features is stock variation the day *after* the tweet or news article has been published. This makes it hard to apply in a trading environment, as the focus there is on the here & now, and of course the future. As far as I am concerned, the merit of this work lies more in the (automatic) application in existing algorithms, but the only application I see now is historical analysis. It would be nice if the authors discuss this in more detail.


Instruction: You are one of the reviewers, your task is to write a review for the article. You will be given the title of the article, the number of the round in which the article is located, and your order among the reviewers.

Input:
Title: EVENT DETECTION IN FINANCE USING HIERARCHICAL CLUSTERING ALGORITHMS ON NEWS AND TWEETS
Review round: 1
Reviewer: 2

Output:
Basic reporting: The article proposes an event detection methodology which is time dependent and domain specific. The domain of application is  financial markets where the events are extracted from financial news. The methodology consist of 3 main steps:
1. to create a time and domain-specific lexicon
2. clustering and filtering to distinguish relevant events discussed in the news
3. to spot hot topics by leveraging the news topics (events) with social media data (basically count the volume of talk for an event in Twitter)

The article is very well written, I found no grammatical mistakes, highly professional English is used. I must also praise the extensive and well-commented background literature (sect 2 Related Work) provided by the authors for it gives a good survey of the state-of-the-art of Event detection algorithms, and  puts their work in modern perspective. Against this background literature this article is certainly a contribution to the field of Event detection.

Now, I have some suggestions to make the description of the proposed approach for event detection more rigorous and founded on statistical theory.
My main concern is on step 1 the lexicon generation for which it is not clear why the heuristic proposed for selecting terms will work in general (in a formal sense, not just in some practical scenarios as shown in the Experimental section). I do note that the lexicon generation method comes from previous work by the authors (see in their references: Carta et al (2020)) which I read through.
Neither there or in this paper the method is given a statistical support/foundation, only empirical assessment.

First a suggestion to make the description of step 1 (Sec. 3.2 .Lexicon generation) more clear and unambiguous.
line 341: "each row corresponds to a news article **and date**, and each column ..."
(The words "and date" should be added: Each news must have its time-stamp explicitly for the next step described in lines 343-346 where return is associated to the terms)
L 344-346, I advice to use other letter for the "day after the article was published" (e.g. c), since d is the date at which we want to discover hot events and for which we are looking at news published on days c \in [d-l;d-1]. In this way it becomes evident that we are looking at news and returns at different time-stamps, whereas the current formula (1) gives a first impression that we are only looking at news on day d-1 (before the events of interest)

Now I offer the authors an argument that may connect their construction to the theory of "sure screening" (Fan and Lv, 2008), hence sustaining it on formal statistics.
I see their lexicon generation  equivalent to computing the frequency f(j) with which a term j co-occurs with a positive value of return.
Assume in the period [d-l;d-1] the algorithm collects N articles, out of which n<N contain the term j (articles can be sentences) . Then
f(j) = (1/n)*\sum_{1 \le k \le N} X_k(j)*\Delta_c(k)
where X_k(j) is a dummy variable for whether term j appears in article k and Delta_c(k) is the return on the day c for article k.
Thus, in this form, f(j) is the slope of a cross-article regression of
\Delta_c = (\Delta_c(1), ..., \Delta_c(N)) on the dummy variable X(j) = (X_1(j), ..., X_N(j)).
i.e. f(j) are coefficients of  a marginal regression [Genovese et al, 2012].
By next sorting them by decreasing scores and selecting those whose value is over (resp. under) some threshold t+ (respect. t-) -which is similar to taking the first n and the last n-, the authors are doing "marginal screening", a form of variable selection which is proven to be more efficient than the Lasso and with good statistical accuracy. What I mean with this last sentence is that if S is the index set of true sentiment charged terms (positive and negative), and
\hat{S} =  { j : f(j) \ge t+, or f(j) \le t- } then under certain conditions
Prob(\hat{S} = S) --> 1 as N and the number of terms go to infinity . This is the sure screening property (Fan and Lv).

This guarantees consistency of the procedure for screening terms for the lexicon (that almost surely are the true sentiment charged terms correlated with price variation) and if you were to work out the proof of the asymptotic result you'll get along the way an expression for the threshold, the number of articles and the terms needed.
I don't see how to prove this in this context easily (and I guess is difficult), so I would not pose this as requirement for publishing this article. However, it would be nice that the authors check this framework of sure screening and include at least an explanation like the one I am offering to mathematically sustain their lexicon construction heuristic.

References:
Fan, Jianqing, and Jinchi Lv, 2008, Sure independence screening for ultrahigh dimensional feature space, Journal of the Royal Statistical Society: Series B (Statistical Methodology) 70, 849-911.

Genovese, Christopher R, Jiashun Jin, Larry Wasserman, and Zhigang Yao, 2012, A comparison of  the lasso and marginal regression, Journal of Machine Learning Research 13, 2107-2143.

Minor comments:
In the legend of Figure 3 : "The bi-dimensional visualization ... by means of tSNE, ..." What is t-SNE? (I am not familiar with this nomenclature) . I short line explaining it, or a reference where to look it up will be appreciated.
Experimental design: No cooment
Validity of the findings: No comment
Additional comments: In the Introduction (lines 85-86) you claim the approach (of event detection) "can be applied to various domains with minimum modifications". However I see that a key component is to have some form of quantifying the behavior of the domain one want to characterize the important events, in order to construct a sentimental-lexicon conditioned to that target characterising  the domain. In the case of financial markets, represented by the market index (e.g SP500), this quantification of its behavior is the variation of the price of the index (or return). For another domain (e.g. politics) what could be this numeric feature that characterizes the domain and that serves as driver for the lexicon generation? An example would be a plus for this nice and interesting event detection proposal.


Instruction: You are one of the reviewers, your task is to write a review for the article. You will be given the title of the article, the number of the round in which the article is located, and your order among the reviewers.

Input:
Title: EVENT DETECTION IN FINANCE USING HIERARCHICAL CLUSTERING ALGORITHMS ON NEWS AND TWEETS
Review round: 2
Reviewer: 1

Output:
Basic reporting: No comment
Experimental design: No comment
Validity of the findings: No comment
Additional comments: The revised version is a major improvement over the initial manuscript. The authors have clarified all of my questions and addressed them in the manuscript appropriately.

Two remarks:
1) The new references in sections 2.4 and 3.1 plus figure 3 seem to have been parsed incorrectly. There are question marks instead of author/year statements. This seems only to be the case in the tracked changes document, but a final check of all references might be good anyway.
2) I am not fond of the constructions "the reader notes" and "the reader notices". Perhaps the authors could consider different phrasing?

All in all, job well done!


Instruction: You are one of the reviewers, your task is to write a review for the article. You will be given the title of the article, the number of the round in which the article is located, and your order among the reviewers.

Input:
Title: EVENT DETECTION IN FINANCE USING HIERARCHICAL CLUSTERING ALGORITHMS ON NEWS AND TWEETS
Review round: 2
Reviewer: 2

Output:
Basic reporting: This is a re-review of the article that I have already revised in fully. My previous praise about the articles proposal, experimental design and results carries through to this revised version.
In this updated version of their orignal article, the authors have complied with all suggested corrections (my own remarks and other reviewer) improving as desired their original manuscript. However there are still some minor typos to be fixed. After fixing this I feel confident about publishing their work.

I will list my new comments below and for further reference I will submit the manuscript with my comments registered in it so that they can be easily found.

- The authors have included the statistic explanation of the lexicon generation process I suggested  to them. Good. However, I feel it should not go in section 3.1, where they put it, but rather at the end of section 3.2. In section 3.1 a general overview of the system is given, and expanding here on the mathematics specificity of the lexicon generation breaks the generality of the description. It looks out of context, and in fact the notation needed (Delta(k), ...) has not been introduced, as it is part of sec. 3.2.

- In the same paragraph (p 11) the equation for f(j) please replaced * for \cdot. To use * for product is unusual, and also you use \cdot later on for other products.

- insert  $t^+$ ($t^-$), after "specified threshold"

Now
p 5 (abstract) l 28: remove "we create". This should be "the algorithm dynamically builds a lexicon by looking ..."

l 289:  although --> accordingly,

l 290: "considering the future information of a day".. substitute by "considering information beyond the day "

l 736: "The reader notices " substitute by The reader should notice
Experimental design: No comment
Validity of the findings: no comment
Additional comments: Nice work!


Instruction: You are one of the reviewers, your task is to write a review for the article. You will be given the title of the article, the number of the round in which the article is located, and your order among the reviewers.

Input:
Title: EVENT DETECTION IN FINANCE USING HIERARCHICAL CLUSTERING ALGORITHMS ON NEWS AND TWEETS
Review round: 3
Reviewer: 1

Output:
Basic reporting: This is the third round of reviewing. The authors have satisfactorily made all my suggested minor corrections. Now the paper is of my entire satisfaction. Go ahead to publication.
Experimental design: Fine
Validity of the findings: Fine
Additional comments: Good job!


Instruction: You are one of the reviewers, your task is to write a review for the article. You will be given the title of the article, the number of the round in which the article is located, and your order among the reviewers.

Input:
Title: MOBILE-ASSISTED AND GAMIFICATION-BASED LANGUAGE LEARNING: A SYSTEMATIC LITERATURE REVIEW
Review round: 1
Reviewer: 1

Output:
Basic reporting: no comment
Experimental design: no comment
Validity of the findings: no comment
Additional comments: The article presents a systematic literature review of mobile assisted and gamification based
language learning. The article covers different important aspects of the study including teaching
and learning approaches, contents and curriculum, and evaluation measures. The discussion is
well structured and well presented.
The selection of articles follows a systematic and well-defined methodology. A reasonable
number of recent articles published in high quality venues have been selected for this study.
However, the abstract and introduction of the article need to be improved. In abstract the authors
should provide the major contributions preferably in quantitative terms. Similarly, the motivation
and justification along with major contributions needs to be clearly presented in the introduction
section.
At several places, the authors have provided a list of references together during the discussion. It
will be more appropriate to discuss relevant articles while discussing their relevance as well as
the differences in their contributions (instead of placing them under one discussion point). Some
more details should be shared to highlight the differences between simple mobile applications
and serious games.
Lastly, the language quality needs to be improved.
As a whole, the article presents useful and valuable contribution which can be helpful for the
researchers working in this area, and can be accepted for publication after addressing the given
comments.


Instruction: You are one of the reviewers, your task is to write a review for the article. You will be given the title of the article, the number of the round in which the article is located, and your order among the reviewers.

Input:
Title: MOBILE-ASSISTED AND GAMIFICATION-BASED LANGUAGE LEARNING: A SYSTEMATIC LITERATURE REVIEW
Review round: 1
Reviewer: 2

Output:
Basic reporting: In this article, the authors have provided a systematic literature review on an interesting topic of learning languages using mobile phones that involves mobile applications as well as mobile games. It presents a review of more than 60 good-quality articles.
The whole discussion has been structured in a very interesting flow that starts with theories, frameworks, and models. Then tools and approaches used for teaching and learning with and without gamification have been discussed. Similarly, contents used for teaching and learning have also been discussed. While different aspects of the evaluation of such approaches are also present. A useful taxonomy and interesting future directions have also been proposed. The article is generally easy to follow, covers an ample amount of background studies and state of the art, and is structured in a good way.
Experimental design: I believe the article falls within the domain of the journal. The methodology provided herein seems comprehensive and consistent throughout. The article is overall organized in a well-organized way with coherence among the paragraphs. The strength of the article is its vast coverage with the help of good quality literature.
Validity of the findings: The results of the paper are encouraging and the conclusion supports the findings of the papers in general.
Additional comments: There are still some minor issues with the article that need to be fixed before its acceptance.
1-It will be useful if the authors could write the article in an impersonal manner and avoid using the pronouns (we and us etc). The quality of some figures needs to be improved.
2-During the synthesis and discussion, there is a need to enrich discussion while explaining the obtained statistics with reasoning, for instance, in the Taxonomic Hierarchy sub-section there is a need to reason with the help of statistics.
3-Similarly, future directions need to be presented in a more structured manner. At different places, they seem to be pretty arbitrary.
4-At a couple of places, there are long streaks of references, which look unusual. Particularly, in the Tools used for Analysis sub-section, there is a long list of references that takes almost two lines. This needs to be improved.
5-Language quality in Abstract, Introduction, and Future Directions needs to be improved.


Instruction: You are one of the reviewers, your task is to write a review for the article. You will be given the title of the article, the number of the round in which the article is located, and your order among the reviewers.

Input:
Title: MOBILE-ASSISTED AND GAMIFICATION-BASED LANGUAGE LEARNING: A SYSTEMATIC LITERATURE REVIEW
Review round: 2
Reviewer: 1

Output:
Basic reporting: The paper presents the topic very well now and accumulates suggested changes now.
Experimental design: The recommended changes in technical content are addresses.
Validity of the findings: The presentation of results is better now.
Additional comments: The authors have successfully incorporated all the suggested changes.
Overall, the paper seems in better form as it accumulates the topic and content under it very well now.


Instruction: You are one of the reviewers, your task is to write a review for the article. You will be given the title of the article, the number of the round in which the article is located, and your order among the reviewers.

Input:
Title: MOBILE-ASSISTED AND GAMIFICATION-BASED LANGUAGE LEARNING: A SYSTEMATIC LITERATURE REVIEW
Review round: 2
Reviewer: 2

Output:
Basic reporting: I have checked the revised version. It seems the authors have addressed all of my comments of basic nature.
Experimental design: The minor design issues have been resolved.
Validity of the findings: I don't have further comments on the findings.
Additional comments: I have checked the revised version. It seems the authors have addressed all of my comments and I don't have any further comments. Hence, the paper could be accepted in the current form.


Instruction: You are one of the reviewers, your task is to write a review for the article. You will be given the title of the article, the number of the round in which the article is located, and your order among the reviewers.

Input:
Title: ANDROANALYZER: ANDROID MALICIOUS SOFTWARE DETECTION BASED ON DEEP LEARNING
Review round: 1
Reviewer: 1

Output:
Basic reporting: Although the idea is interesting, I recommend to improvement article with some comments as below:
- More description for experiments should be done.
- The method can be explained in a clearer way.Try to explain the theory more detailed in discussion.
Experimental design: - The reason of using certain parameters in the experiments should be discussed in more details.
Validity of the findings: no comment
Additional comments: - The references in this manuscript are somewhat out-of-date.In clude more recent researches in this field.
- The manuscript has not been carefully written.
- There are many grammar mistakes as well.
- The reason of using certain parameters in the experiments should be discussed in more details.


Instruction: You are one of the reviewers, your task is to write a review for the article. You will be given the title of the article, the number of the round in which the article is located, and your order among the reviewers.

Input:
Title: ANDROANALYZER: ANDROID MALICIOUS SOFTWARE DETECTION BASED ON DEEP LEARNING
Review round: 1
Reviewer: 2

Output:
Basic reporting: A clear view of the contribution is given in the paper.
Experimental design: The sample size differs from that used in related works. The authors, for instance, used 6661 Malware App, while the others used 33259, 928,... , and 19620. The distinction between these samples needs to be illustrated, so we can ensure that the authors have not compared the accuracy of basic malware apps with sophisticated ones used in similar works.
Validity of the findings: The accuracy values written for the work in Table 4 are in the wrong format. (e.g., 0.989 rather than %98.9).
Also, the header for this row is confused "Proposed Model DL(376502(300,300,300,300))"
Additional comments: None.


Instruction: You are one of the reviewers, your task is to write a review for the article. You will be given the title of the article, the number of the round in which the article is located, and your order among the reviewers.

Input:
Title: ANDROANALYZER: ANDROID MALICIOUS SOFTWARE DETECTION BASED ON DEEP LEARNING
Review round: 1
Reviewer: 3

Output:
Basic reporting: The paper describes a model using AndroAnalyzer that uses static analysis and a deep learning system is proposed. The model is tested in numerous applications, and additional tests were conducted with machine learning. The authors claim that 98.16% accuracy value was reached compared to 26 traditional machine learning techniques, where Precision and F-measure were 98.78, 27 99.24, and 98.90, respectively.
The paper is generally good. The motivation and research problem are well defined. The methodology is clear and the results are discussed and compared with some similar tools.
The paper needs more linguistic and grammatical improvement. It should proofread.
In the paper structure at the end of the Introduction Section, preplace the word “Chapter” with the word “Section”.
Heading and Sub-Heading are not numbered.
The Literature Review should be a standalone section named related.
Many papers are surveyed and classified, however, the studies reviewed are in between 2018-2020, why no mention of the studies contacted before 2018.
Change the sentence “In the [13] numbered study performed by Shahriar et al., a model was proposed to detect malicious software ….” to be “In [13], a model was proposed to detect malicious software …. ” and use this throughout the paper.
The description of the existing studies is not enough to shed light on the research gap. The existing work has to be compared and critically evaluated.
The methodology needs more clarification about its stages, and the figure should be linked with the stages described in detail in numbered subheadings and bearing the numbers of the stages in the illustrated figure. As well as clarify what is meant by some of the acronyms and not leave an indulgence for the reader to devise them
Experimental design: The experiments  are conducted, explained, discussed, and evaluated with other related approaches
Validity of the findings: Results are explained, discussed, and evaluated with other related approaches
Additional comments: No additional comments


Instruction: You are one of the reviewers, your task is to write a review for the article. You will be given the title of the article, the number of the round in which the article is located, and your order among the reviewers.

Input:
Title: ANDROANALYZER: ANDROID MALICIOUS SOFTWARE DETECTION BASED ON DEEP LEARNING
Review round: 2
Reviewer: 1

Output:
Basic reporting: None
Experimental design: None
Validity of the findings: None
Additional comments: The contribution is clear and significant, but still the Language needs to be improved. For example, the following statement in the result section “ Both obtaining the best deep learning structure ” is confused.


Instruction: You are one of the reviewers, your task is to write a review for the article. You will be given the title of the article, the number of the round in which the article is located, and your order among the reviewers.

Input:
Title: ANDROANALYZER: ANDROID MALICIOUS SOFTWARE DETECTION BASED ON DEEP LEARNING
Review round: 3
Reviewer: 1

Output:
Basic reporting: The paper describes a model using AndroAnalyzer that uses static analysis and a deep learning system is proposed. The model is tested in numerous applications, and additional tests were conducted with machine learning. The authors claim that 98.16% accuracy value was reached compared to 26 traditional machine learning techniques, where Precision and F-measure were 98.78, 27 99.24, and 98.90, respectively.
The paper is generally good. The motivation and research problem are well defined. The methodology is clear and the results are discussed and compared with some similar tools.
The paper needs more linguistic and grammatical improvement. It should proofread.
In the paper structure at the end of the Introduction Section, preplace the word “Chapter” with the word “Section”.
Heading and Sub-Heading are not numbered.
The Literature Review should be a standalone section named related.
Many papers are surveyed and classified, however, the studies reviewed are in between 2018-2020, why no mention of the studies contacted before 2018.
Change the sentence “In the [13] numbered study performed by Shahriar et al., a model was proposed to detect malicious software ….” to be “In [13], a model was proposed to detect malicious software …. ” and use this throughout the paper.
The description of the existing studies is not enough to shed light on the research gap. The existing work has to be compared and critically evaluated.
The methodology needs more clarification about its stages, and the figure should be linked with the stages described in detail in numbered subheadings and bearing the numbers of the stages in the illustrated figure. As well as clarify what is meant by some of the acronyms and not leave an indulgence for the reader to devise them
Experiments Results are explained, discussed, and evaluated with other related approaches
Experimental design: Experiments Results are explained, discussed, and evaluated with other related approaches
Validity of the findings: The existing work has to be compared and critically evaluated.
Results are explained, discussed, and evaluated with other related approaches
Additional comments: The Paper seems good but needs some modification


Instruction: You are one of the reviewers, your task is to write a review for the article. You will be given the title of the article, the number of the round in which the article is located, and your order among the reviewers.

Input:
Title: LOG-BASED SOFTWARE MONITORING: A SYSTEMATIC MAPPING STUDY
Review round: 1
Reviewer: 1

Output:
Basic reporting: 1. The title is misleading, “Contemporary software monitoring: a systematic mapping study.” In particular, while the title is about software monitory, the survey mainly discusses the usage of software logs, including logging, log parsing, log storage, and log analysis. I suggest the authors either include papers about software monitoring that are not based on software logs or revise the title to align with the main context. For example, “Log-based Software Monitoring: A Systematic Mapping Study.”

2. In abstract, the authors mention “A holistic view of the logging research field is key to provide directions and to disseminate the state-of-the-art for technology transferring.” The term “logging” may not be accurate. “Logging” typically refers to designing logging statements in source codes and the practice that developers conduct logging is “logging practice”. The research field should be “automated log analysis”.

3. The authors mention that “In this paper, we study 108 papers … from different communities (machine learning, software engineering, and systems) ” However, it is unclear how many papers each category contains. The authors are recommended to add a table to show the related statistics.

4. Line 221 (Overview of Research Areas). The term “Log Engineering” is inaccurate and misleading. Could the authors explain why “log engineering” is an appropriate term here? I would prefer “logging”.

5. Table 3. The categorization is inaccurate. For example, “Logging practice” row mainly includes empirical studies. “Implementation of log statements” row mainly includes methodologies on what-to-log, where-to-log, and how-to-log. However, studies in the “Logging practice” row also explores these three topics. Thus, I suggest the authors to use “Empirical studies” instead of “Logging practice”.

6. In “log analysis”, the authors mention several tasks, including anomaly detection and failure prediction. However, it is unclear what are the differences between them. The definition provided in Table 3 looks similar: “Detection of abnormal behaviour” and “Anticipate abnormal behaviour”.
Experimental design: 1. A line of research on what kind of log we should use should be considered, for example:
[ICSE’16] Behavioral Log Analysis with Statistical Guarantees
[ESEC/FSE’18] Using Finite-State Models for Log Differencing
[ASE’19] Statistical Log Differencing

2. I suggest the authors add a section that describes existing open-source tools and datasets for log-based software monitoring.
Validity of the findings: no comment
Additional comments: No additional comments


Instruction: You are one of the reviewers, your task is to write a review for the article. You will be given the title of the article, the number of the round in which the article is located, and your order among the reviewers.

Input:
Title: LOG-BASED SOFTWARE MONITORING: A SYSTEMATIC MAPPING STUDY
Review round: 1
Reviewer: 2

Output:
Basic reporting: This paper details a mapping study about the monitoring of modern software systems. The main result is a classification about logging research areas through the analysis of 108 papers. Then a discussion concerning the findings is reported.

The article is well written, and the authors show competence and demonstrate excellent expertise in the field. I think that behind this paper, there is a certain amount of work that must be acknowledged.
The paper is clear concerning the English language, but the authors mix English UK and English US. For example, in the Introduction Section, they use both behavior (or analyze) and behaviour (or analyse). Please check it in the overall paper.

The Introduction highlights the problem adequately, and the motivations are well justified. Relation with exiting literature is satisfactory and well discussed.
Concerning the paper structure, it does not follow the template suggested by the journal for literal review. In particular, the authors added the Discussion section as recommended by the standard journal template.  However, having two separate sections allow better to highlight the results with respect to their analysis.
Experimental design: The authors used the well-known guidelines from Petersen K. et al. "Systematic Mapping Studies in Software Engineering" for having a mapping study in software engineering. However, they did not follow the guidelines. In particular, they did not define any research questions about the mapping study; they did not report a subsection containing validity threats.
Concerning the period used during the search process, it comes from 1992 to 2019 (line 213). However, in the paper's title, the authors refer to the monitoring of contemporary software, and the starting year is far from being considered modern. So please motivate why they decided to use 1992 as the starting date.
Moreover, it is unclear why they decided to update the survey through a forward snowballing and not through a repetition of the search process only for the considered period. Again, please motivate it.

Please change the sentence at line 89 to better highlight the usage of the previous guidelines described in the paper.
Validity of the findings: The Results section contains the core of the paper, and it is clear and well written. However, as already said in the study design, the results are not discussed for the hypothetical defined research questions. So please, add the RQs and then modify the results section accordingly. Also, The discussion and conclusions sections are clear and well written.
Additional comments: Please see the following from the PeerJ literal review template: Should not be used to acknowledge funders – funding will be entered online in the declarations page as a separate Funding Statement and appear on the published paper

## Minor comments
- overall paper: please hyphenate open source when used as an adjective before a noun, i.e., open source projects --> open-source projects
- line 41: and also highlight --> and also highlights
- lines 228, 229: (2)security (6)model --> add space (2) security (6) model
- line 395: located located --> located
- line 456: to large log data Later, --> to large log data. Later,
- line 488: Results indicates --> Results indicate
- line 527: analysis, Farshchi et al. (2015); Why the reference? the sentence refers to another reference i.e., Juvonen et al. (2015)
- line 594: The propose --> They propose
- line 644: Ulrich et al. (2003) shows --> Ulrich et al. (2003) show
- line 651: has been also possible --> has also been possible
- line 741: allows to obtain --> allows obtaining


Instruction: You are one of the reviewers, your task is to write a review for the article. You will be given the title of the article, the number of the round in which the article is located, and your order among the reviewers.

Input:
Title: LOG-BASED SOFTWARE MONITORING: A SYSTEMATIC MAPPING STUDY
Review round: 2
Reviewer: 1

Output:
Basic reporting: The review is of broad interest and is in the scope of the journal.
Experimental design: Sources are properly cited.
Validity of the findings: The goals set out in the introduction has been supported well.
Additional comments: The authors have addressed all my major concerns.


Instruction: You are one of the reviewers, your task is to write a review for the article. You will be given the title of the article, the number of the round in which the article is located, and your order among the reviewers.

Input:
Title: LOG-BASED SOFTWARE MONITORING: A SYSTEMATIC MAPPING STUDY
Review round: 2
Reviewer: 2

Output:
Basic reporting: The authors made an acknowledgeable effort towards addressing my comments, although a couple of comments could have been addressed more deeply.
Experimental design: Specifically, concerning the comment related to the usage guidelines from Petersen K. et al., the authors state that they used to only derive the classification scheme systematically by using keywording of abstracts. However, some questions naturally arise. What other standard process/guidelines did the authors use? The authors' defined process seems very similar to the Petersen K. et al., except for the definition of the research questions that are missing. Moreover, it is not clear to me the motivation concerning the research questions' absence. Furthermore, the authors used the keywording of abstracts from Petersen K. et al. paper. Why did the authors not provide the process? Therefore, I am not entirely convinced about the authors' answer.
Validity of the findings: Concerning the threats to validity section, in general, it has to contain internal validity, external validity, construct validity, and sometimes conclusion validity. Please modify the threats to validity section accordingly. A good example of threats to validity section is to the following paper: Architecting with microservices: A systematic mapping study (Paolo Di Francesco, Patricia Lago, Ivano Malavolta).
Additional comments: I think that the paper would reach a higher maturity if the above comments would be reworked. Should the editor opt for a further minor revision round in this direction, from my side, it is not needed to receive the final revision.


Instruction: You are one of the reviewers, your task is to write a review for the article. You will be given the title of the article, the number of the round in which the article is located, and your order among the reviewers.

Input:
Title: OPTIMALLY SELECTING THE TOP K VALUES FROM X + Y WITH LAYER-ORDERED HEAPS
Review round: 1
Reviewer: 1

Output:
Basic reporting: The authors present a way to find top k smallest elements from the Cartesian sum X+Y where X and Y are two given sets. For this, they use layer-ordered heaps (LOH). In the results section they compare their method with two other methods: 1)  naive O(n^2 log(n) + k) method and 2) soft heap based method by Kaplan et. al. They see a 295x and 18x speedup respectively.  The authors have also thoroughly analyzed the proof of correctness of the algorithm. Additionally, they have provided the python code for the same.

In my opinion, the paper is well written (there are typos which are listed below) and is organized properly.  All the lemmas and theorems are clearly defined. The manuscript also includes enough background on the topics that are needed to understand the paper. But it would have been easily understandable if an example had been provided. Additionally, the space complexity is not analyzed properly.

Typos: (provided inside " ")
...their method selects only the kth "minimum" value from X +Y in...
It bounds the number of "corrupt" elements in the soft heap....
this can be seen as a more constrained form of "the" heap...
Due to its simplicity and contiguous memory access, it has "a" fast performance in...
used to indicate that this is the "minimum" value in the layer product..
W.l.o.g., there is "no" guaranteed ordering of the form...
The values from every element in each layer product in q "are" generated...
Each (u′, v′) matches "" exactly one layer product (u′−1, v′−1)...
Table 1: N = 4000, K = 2000...17.2 ... 0.422... 0.0409=0.0268+0.0141
Experimental design: The research question is well defined but it is a new optimal algorithm in addition to other optimal algorithms that already exist (for eg. soft heap based approach). The novelty of the paper is the usage of layer ordered heaps.

In the results section, the table is not explained in depth i.e. how different methods compare and why they are giving such results. Also, it would have been interesting to see such comparison for different values of alpha and then compare the average speed up with other methods.

I feel that the authors should provide some brief introduction to the naive algorithm stated in the results section.
Validity of the findings: I think that the manuscript lacks a concluding paragraph that wraps up everything.
Additional comments: No additional comments


Instruction: You are one of the reviewers, your task is to write a review for the article. You will be given the title of the article, the number of the round in which the article is located, and your order among the reviewers.

Input:
Title: OPTIMALLY SELECTING THE TOP K VALUES FROM X + Y WITH LAYER-ORDERED HEAPS
Review round: 1
Reviewer: 2

Output:
Basic reporting: The authors present a new algorithm for the problem of selecting the top k values from the Cartesian sum X + Y. The algorithm is simple and runs in optimal O(n + k) time by making use of Layered-Ordered Heaps. An algorithm with the same time complexity previously existed, but the authors demonstrate through experiments that the algorithm they present is faster in practice. They additionally provide an implementation of their algorithm in python.

Pros:  I believe that the techniques being used here provide a very simple algorithm for solving this problem. Additionally, despite having the same asymptotic time complexity, the new algorithm appears to be significantly faster in practice.

Cons:  Even though I believe that the algorithm is correct, I found many of the statements made in the proofs are hard to follow and feel they require more justification. Also, there are minor issues I take with some of the notation.

The overall structure of the proof appears to be a good way to go about proving the correctness and time complexity of the algorithm. That is, the lemmas seem to be the right ones to prove the overall result. However, the proofs of the lemmas could use additional work. I have described below the main portions of the individual proofs that could use further justification, along with some additional minor notational issues and typos.

Once these revisions are made and the proofs are satisfactorily clear, I feel that the presented algorithm is interesting enough that the work should be accepted.

Line 56: limit should be on u, not i.

Line 80: comma before X_2^(u) + Y_1^(v)

Line 83 and following paragraph: I think "false" and "true" within the tuple are unnecessary and just confused me at first. It seems 0 and 1 could be used immediately. Also, before the same line where the ordering on tuples is introduced, it should be stated that lexicographic ordering is being used. Finally, I'm not sure why (u,v) itself needs to be in the tuple at all.

Proof of Lemma 2.1 :
Line 112: "structures of pops of the form ... " Could this be made more explicit? Is there a better word than "structures", and can some justification be given? This line seems critical, after that I agree with the remaining proof.

Proof of Lemma 2.2: this is fine.

Proof of Lemma 2.3 ;
The mention of maximum and minimum tuples in the statement of the Lemma made me think that the proof was to prove all maximum tuples are in ascending order and all minimum tuples are in ascending order, but what the proof is actually trying to show that all tuples are popped in ascending order.

Line 128: "meaning that for all v >= r <= b". I am confused by this notation.
Overall, I believe this second paragraph could use much more justification. How does at "least one such floor(a,r) must be in H" go-to "all floor(a,r) must be popped before floor(u,v)"?

Similarly, I feel more justification is needed in the third paragraph. How does "at least one ..." lead to floor(a,b) being popped.

Line 141: the select notation is introduced for the first time in this way.

Line 179: Approximately less-or-equal-to should be made more rigorous.

Lines 180 and line 190, "area"? Perhaps just say product.

Line 199: "on the hull" should be "on the heap"

Section 2.4 seems unnecessary.

Very minor: I prefer sentences beginning with a word rather than mathematical notation, as was done in a few places.
Experimental design: no comment
Validity of the findings: no comment
Additional comments: No additional comments


Instruction: You are one of the reviewers, your task is to write a review for the article. You will be given the title of the article, the number of the round in which the article is located, and your order among the reviewers.

Input:
Title: OPTIMALLY SELECTING THE TOP K VALUES FROM X + Y WITH LAYER-ORDERED HEAPS
Review round: 1
Reviewer: 3

Output:
Basic reporting: In the introduction section, I would suggest the author add more practical applications for selecting the top-k problem before explaining the existing solutions to enhance the importance of the work.

Although the structure of the article is in an acceptable format, It would be better if the author provides some examples or figures for illustrations of the construction part or even the definitions to help readers understanding. For instance, in section 1.2 I would suggest adding an example including a figure to illustrate layer-order heaps and show the partitioning for some alpha.

I would suggest adding a pseudo-code including all the important steps from phases so that readers can see all the data structures (with their initializations) and the main algorithms without distracting by details and proofs.
Experimental design: no comment
Validity of the findings: no comment
Additional comments: In this paper, the author presents an optimal algorithm to output the top-k values of X_i+Y_j based on layer-ordered heaps. The solution provided in the paper is not complicated and it is easy to follow as it only uses linear time one-dimensional selection. In addition, the algorithm has fast performance since it uses data structures contiguous in memory and it is cache efficient.

This paper is well written and has been organized properly by providing appropriate sections. The problem is well stated, and the solution is compared properly with the related algorithms in the last section of the paper. The python implementations of all phases of the algorithm have been provided properly. The presented algorithm achieves 295X speed-up over the naive approach and 18X speed-up over the soft heap approach.

As far as being comprehensible, the algorithm and proofs provided are reliable and error-free. There exist sufficient proofs for the correctness of the algorithm as well as for the time and space complexities of each phase.


Instruction: You are one of the reviewers, your task is to write a review for the article. You will be given the title of the article, the number of the round in which the article is located, and your order among the reviewers.

Input:
Title: OPTIMALLY SELECTING THE TOP K VALUES FROM X + Y WITH LAYER-ORDERED HEAPS
Review round: 2
Reviewer: 1

Output:
Basic reporting: I think that the authors have incorporated the changes and the final version of the paper looks good and can be accepted.
Experimental design: no comment
Validity of the findings: no comment
Additional comments: No additional comments


Instruction: You are one of the reviewers, your task is to write a review for the article. You will be given the title of the article, the number of the round in which the article is located, and your order among the reviewers.

Input:
Title: OPTIMALLY SELECTING THE TOP K VALUES FROM X + Y WITH LAYER-ORDERED HEAPS
Review round: 2
Reviewer: 2

Output:
Basic reporting: I find the changes made by the author make the proofs more explicit and easier to verify. One addition that has a significant impact on the paper is the addition of Figure 1. At least for myself, it encourages visualizing this algorithm as processing the values on a grid in a particular order, rather than only contemplating the correctness of inequalities. Conceptualizing the algorithm this way makes the proofs significantly easier to understand.

The remaining comments I have are minor:

All proofs have two QED symbols.

I feel that references to figures, lemmas, phases, should be capitalized. For example, "An illustration of these phases is provided in figure 1." should be "An illustration of these phases is provided in Figure 1."
Experimental design: no comment
Validity of the findings: no comment
Additional comments: No additional comments


Instruction: You are one of the reviewers, your task is to write a review for the article. You will be given the title of the article, the number of the round in which the article is located, and your order among the reviewers.

Input:
Title: OPTIMALLY SELECTING THE TOP K VALUES FROM X + Y WITH LAYER-ORDERED HEAPS
Review round: 2
Reviewer: 3

Output:
Basic reporting: no comment
Experimental design: no comment
Validity of the findings: no comment
Additional comments: In the revised version, the authors have added an example based on my last review and have explained the corresponding algorithm. Regarding my other comment, I have been convinced that their python code is sufficient as it makes their implementation reproducible. The new version is well written and I recommend to be published.


Instruction: You are one of the reviewers, your task is to write a review for the article. You will be given the title of the article, the number of the round in which the article is located, and your order among the reviewers.

Input:
Title: INTROSPECTIVE ANALYSIS OF CONVOLUTIONAL NEURAL NETWORKS FOR IMPROVING DISCRIMINATION PERFORMANCE AND FEATURE VISUALISATION
Review round: 1
Reviewer: 1

Output:
Basic reporting: -	The graphical representation needs to be improved in Figure 1. The arrow sizes are different, and some are not straight. Does arrow length carry any information? If not, they should all be of the same size. Traditionally, the neural network layers are depicted from left to right, such as the schematic shown in Figure 2. Therefore, it is recommended to demonstrate the two CNN architectures in Figure 1 from left to right.
-	The English language should be improved in Lines 117-132 to ensure your points are clearly described in a scientific language.
-	Line 177: The terms normalization and standardization are sometimes used interchangeably, but they usually refer to different things. Normalization usually means to scale a variable to have a value between 0 and 1, while standardization transforms data to have a mean of zero and a standard deviation of 1 (z-score). Hence, it is recommended to use “standardization” instead of “normalization”.
-	I recommend reporting some of the tables presented in the main manuscript to supplementary data so that the key points are highlighted, and additional results are available for an interested reader.
Experimental design: no comment
Validity of the findings: -	One key missing analysis on the performance evaluation is the significance test in comparison with previous architectures such as LeNet and AlexNet in Table 5-10. Reporting p-value will provide additional information about the effectiveness and significance of the proposed approach.
Additional comments: -	This is a very well written manuscript describing a novel approach for improving discrimination performance and feature visualization in Convolutional Neural Networks (CNN). Different parameters are taken into account and the results are reported with sufficient details and discussions.


Instruction: You are one of the reviewers, your task is to write a review for the article. You will be given the title of the article, the number of the round in which the article is located, and your order among the reviewers.

Input:
Title: INTROSPECTIVE ANALYSIS OF CONVOLUTIONAL NEURAL NETWORKS FOR IMPROVING DISCRIMINATION PERFORMANCE AND FEATURE VISUALISATION
Review round: 1
Reviewer: 2

Output:
Basic reporting: - English language should be improved.

- It is necessary to provide more literature review related to the datasets that they used.

- Deep learning (i.e., CNN) has been used in previous works in a variety of fields such as https://doi.org/10.7717/peerj-cs.177 and https://doi.org/10.1016/j.neucom.2019.09.070. Therefore, it is suggested to provide more references in this description.
Experimental design: - How did the authors perform hyperparameter tuning in the models? It should be described clearly. Did the cross-validation technique apply?

- Hardware specification to train model should be described.

- The authors should release source codes for replicating the results.

- There is a need to refer to some works related to measurement metrics (i.e., recall, precision, accuracy, ...) of deep learning models such as PMID: 31362508 and PMID: 32613242.

- In addition to the aforementioned metrics, the authors should report ROC curves and AUC values for prediction models.
Validity of the findings: - The authors should have some validation data on the models.

- It looks like some models contained a lot of overfitting after few epochs (i.e., Fig. 3-4). How did the authors address this critical point?

- Training time should be reported.

- The authors should perform some statistical tests to compare the performance results among different models.

- The authors should compare the predictive performance with previous works that focused on the same datasets.
Additional comments: No comment


Instruction: You are one of the reviewers, your task is to write a review for the article. You will be given the title of the article, the number of the round in which the article is located, and your order among the reviewers.

Input:
Title: INTROSPECTIVE ANALYSIS OF CONVOLUTIONAL NEURAL NETWORKS FOR IMPROVING DISCRIMINATION PERFORMANCE AND FEATURE VISUALISATION
Review round: 2
Reviewer: 1

Output:
Basic reporting: No comment.
Experimental design: No comment.
Validity of the findings: No comment.
Additional comments: My previous comments have been addressed well.


Instruction: You are one of the reviewers, your task is to write a review for the article. You will be given the title of the article, the number of the round in which the article is located, and your order among the reviewers.

Input:
Title: MULTI-STEP RAINFALL FORECASTING USING DEEP LEARNING APPROACH
Review round: 1
Reviewer: 1

Output:
Basic reporting: no comment
Experimental design: no comment
Validity of the findings: no comment
Additional comments: The manuscript presents "Multi-step Rainfall Forecasting using Deep Learning Approach", which is interesting. The paper has the potential to be published in this journal after revision. Appropriate revisions to the following points should be undertaken in order to justify recommendation for publication

1- The language of the manuscript should be checked and more proofreading is necessary. I also recommend that authors should refrain from using long sentences. Instead, they should use clear and shorter sentences in terms of the readability of the manuscript. In addition, please correct the grammatical errors throughout the manuscript.
2- Introduction chapter requires the research purposes in detail. Since they are vague in their present form, please revise them. The authors need to provide a very detailed statement as to how this paper differs from their other papers of similar titles.
3- Full names should be shown for all abbreviations in their first occurrence in texts. For example, CNN, DBN etc. (Noy only in Abstract).
4- Authors should give some comparisons about the usage of data-driven models a little bit more (briefly) in the "Introduction" section. They should mention about the performance of these models, shortly. It is well known fact that most of the statistical models (ANN, ARIMA, SVR, etc.) can predict one day ahead forecast with considerable accuracy. However, their performance reduces significantly when forecasting is made multiple days ahead. In this study the authors considered multi-step ahead forecast. It is important to see the performance of the model when no observed rainfall data were used for the entire validation period to forecast the rainfall data.
5- There are several conceptual/physically based models available for rainfall forecasting, which needs to be discussed in the introduction and discussion section.
6- For readers to quickly catch your contribution, it would be better to highlight major difficulties and challenges, and your original achievements to overcome them, in a clearer way in abstract and introduction.
7- It is mentioned in “Main contribution of this research” that the authors employed DBN for feature extraction whereas NARX network was developed and trained for extrapolating the temporal forecasts. What are other feasible alternatives? What are the advantages of adopting this particular method over others in this case? How will this affect the results? The authors should provide more details on this.
8- More focus should be placed on the application of accurate multi-step rainfall forecasting. You can use specialized articles in this field.
* Emotional artificial neural networks (EANNs) for multi-step ahead prediction of monthly precipitation; case study: Northern Cyprus
* Threshold-based hybrid data mining method for long-term maximum precipitation forecasting
Accurate rainfall forecasting is very effective in predicting runoff, streamflow and flood:
* Emotional ANN (EANN) and Wavelet-ANN (WANN) Approaches for Markovian and Seasonal Based Modeling of Rainfall-Runoff Process
* Hybrid Wavelet-M5 Model Tree for Rainfall-Runoff Modeling
* Data mining based on wavelet and decision tree for rainfall-runoff simulation

9- Please explain that, how were the time series divided?
10- The titles to the figures and tables need to be much improved-more descriptive.
11- Please discuss the results and compare them with previous studies.
12- Please examine reference styles from "Instructions for Authors" in the journal website and arrange your references according to it.


Instruction: You are one of the reviewers, your task is to write a review for the article. You will be given the title of the article, the number of the round in which the article is located, and your order among the reviewers.

Input:
Title: MULTI-STEP RAINFALL FORECASTING USING DEEP LEARNING APPROACH
Review round: 1
Reviewer: 2

Output:
Basic reporting: The focus of this work is direct prediction of multistep forecasting, where a separate time series model for each forecasting horizon is considered and forecasts are computed using
observed data samples.
Experimental design: Forecasting in this method is performed by proposing a deep learning approach, i.e, Temporal Deep Belief Network (DBN). The best model is selected from several baseline models on the basis of performance analysis metrics. The results suggest that the temporal DBN model outperforms the conventional Convolutional Neural Network (CNN) specifically on rainfall time series forecasting.
Validity of the findings: The results are valid.
Additional comments: These are my main concerns:
1: The abstract is poor and it should be rewritten completely. Remove the unnecessary information from the abstract. Stay focus more on your work and contribution.
2: What is meant by one-step, four step and eight step?  Explain this clearly whether this means by hourly, daily, weekly or monthly time step.
3: The introduction and background literature is not enough on rainfall using ML models. The authors should clearly work on it to enrich the literature section. For example the authors can cite the following relevant work.
a): Complete ensemble empirical mode decomposition hybridized with random forest and kernel ridge regression model for monthly rainfall forecasts; Journal of Hydrology 584, 124647, 2020.
b): Multi-stage hybridized online sequential extreme learning machine integrated with Markov Chain Monte Carlo copula-Bat algorithm for rainfall forecasting, Atmospheric research 213, 450-464, 2018.
c): Forecasting long-term precipitation for water resource management: a new multi-step data-intelligent modelling approach, Hydrological Sciences Journal, Volume 65 Issue 16 (2020).
d): An ensemble-ANFIS based uncertainty assessment model for forecasting multi-scalar standardized precipitation index, Atmospheric Research 207, 155-180, 2018.
e): Improving SPI-derived drought forecasts incorporating synoptic-scale climate indices in multi-phase multivariate empirical mode decomposition model hybridized with simulated. Journal of Hydrology 576, Pages 164-184, 2019.
f): Multi-stage committee based extreme learning machine model incorporating the influence of climate parameters and seasonality on drought forecasting. Computers and Electronics in Agriculture, Volume 152, September 2018, Pages 149-16

4: The presentation of the paper should improved. Some figures are hardly readable.
5: The data should be explained clearly. What are the input attributes that are used to predict the rainfall? How many data points are used for model training and testing purposes?
6: The experimental setup and model architectures should be discussed in details. They are very important.
7: The results should be used to explain in some other graphical visualization such as Taylor diagram, scatter plots etc.


Instruction: You are one of the reviewers, your task is to write a review for the article. You will be given the title of the article, the number of the round in which the article is located, and your order among the reviewers.

Input:
Title: MULTI-STEP RAINFALL FORECASTING USING DEEP LEARNING APPROACH
Review round: 2
Reviewer: 1

Output:
Basic reporting: The authors answered to all of my concerns and comments
Experimental design: The authors answered to all of my concerns and comments
Validity of the findings: The authors answered to all of my concerns and comments
Additional comments: The authors answered to all of my concerns and comments


Instruction: You are one of the reviewers, your task is to write a review for the article. You will be given the title of the article, the number of the round in which the article is located, and your order among the reviewers.

Input:
Title: MULTI-STEP RAINFALL FORECASTING USING DEEP LEARNING APPROACH
Review round: 2
Reviewer: 2

Output:
Basic reporting: The authors have successfully revised the manuscript according to my feedback which improves to the level of publishing. I recommend to accept the paper in present form.
Experimental design: The authors have successfully revised the manuscript according to my feedback which improves to the level of publishing. I recommend to accept the paper in present form.
Validity of the findings: The authors have successfully revised the manuscript according to my feedback which improves to the level of publishing. I recommend to accept the paper in present form.
Additional comments: The authors have successfully revised the manuscript according to my feedback which improves to the level of publishing. I recommend to accept the paper in present form.


Instruction: You are one of the reviewers, your task is to write a review for the article. You will be given the title of the article, the number of the round in which the article is located, and your order among the reviewers.

Input:
Title: SEMI-SUPERVISED OBLIQUE PREDICTIVE CLUSTERING TREES
Review round: 1
Reviewer: 1

Output:
Basic reporting: - The proposal is in overall good as well as the organization of the paper. I greatly appreciate the fact of making the code of the tool publicly available through git repository.
- Please, provide vectorial images (eps, pdf, ...) when possible. Furthermore, for most of the figures (such as figures 4 and 5), authors should include a legend for lines/colors to improve their readability.
- Please, be consistent with the notation. In table 2 methods are called "SPYCT-GRAD-SSL", while in Table 3 they are called "SSL-SPYCT-GRAD"
Experimental design: - Authors repeat the experiments 10 times with each configuration. It would be desirable to repeat at least 30 times for each different configuration, in order to obtain more consistent results in statistical tests.
- In Figure 3 as well as in Tables 2-3, authors include the results for all algorithms. These results are for all datasets, each of them averaged among their 10 (hopefully 30 in the future) executions? Please, make it clearer.
- I encourage authors to make raw results publicly available to ease the reproducibility of results in any kind of supplementary material (as a website or in the gitlab repository). At least those averaged results among all executions, not necessary the results of each single execution.
- When studying the learning time, authors only report the results on big datasets. I think it would be a good approach to illustrate the results in a figure (as they do), but results for all datasets should be reported and studied. If the proposed method do not perform faster in smaller datasets but does in bigger, it should be stated (or if it run faster in all/most cases). Besides, I would encourage to include the times of single trees too.
Validity of the findings: - The validity of the results is great; the proposed method perform significantly better than state-of-the-art in some cases, but never significantly worse. Besides, their proposed method run much faster than state-of-the-art.
Additional comments: - The paper is in overall good, but some improvements should be made in the experimental setting to finally accept the paper.


Instruction: You are one of the reviewers, your task is to write a review for the article. You will be given the title of the article, the number of the round in which the article is located, and your order among the reviewers.

Input:
Title: SEMI-SUPERVISED OBLIQUE PREDICTIVE CLUSTERING TREES
Review round: 1
Reviewer: 2

Output:
Basic reporting: In general, the paper is well written and related literature is well cited. Figures and tables are adequate.

For reporting the results, the manuscript only contains ranking diagrams and the number of wins per method. To allow a better understanding of how good your method is, I would like to suggest the inclusion of the numerical values themselves (perhaps in an appendix or supplemental material).

Small remarks: figure 4 is missing a key and the symbol 'w' is used with different meanings (e.g. rule 121 versus rule 129).
Experimental design: The evaluation metrics for multi-target regression (MTR) and hierarchical multi-label classification (HMLC) are somewhat unconventional. What is the advantage of using the coefficient of determination for MTR as opposed to the more common average relative root mean squared error? For HMLC, why exactly was the LRAP measure chosen, has it been used before?

The current evaluation for feature importance is rather unclear. It relies on adding noisy features and comparing their scores against real features, nonetheless this raises several comments.
		a) What exactly do you mean by score?
		b) How different coefficients value affect your score? I assume that the same feature may have very different coefficients depending on the split, can they cancel each other out?
		c) How are these random features generated?

As reported in Kocev 2013, targets with different ranges can affect the heuristic values differently. This is solved by normalizing the targets using their variance beforehand. Does your implementation take that into account?
Validity of the findings: My main concern regarding the validity of the results is the transductive experimental set-up for the proposed approach: if the unlabeled training set is used as test set, it seems to me somewhat an unfair comparison to the supervised setting. In my opinion, either the predictive performance should be compared on an independent test set for both SSL and supervised settings (i.e., inductive) or the learning task should be clearly defined in the method section (e.g., when introducing the notation in the first paragraph).

In the ensemble setting, bagging is used. Indeed, with bagging you will have a time complexity quadratic in the number of features. However, more common ensembles like random forests will also reduce time complexity. How does bagging with oblique splits correspond to random forests with default splits in learning time (and predictive performance)?

For reproducibility purposes, parameter values should be included. Please find a below a list of parameters that I have failed to find in the manuscript:
		- C -> SVM and gradient variant;
		- initial w, initial b and Adam parameters -> gradient variant;
		- Stopping criteria to grow trees -> I could only find the following quote
		"The splitting of the examples (i.e., the tree construction) stops when at least one of the following stopping criteria is reached. We can specify the minimum number of examples required in leaf nodes (at least one labeled example is always required otherwise predictions cannot be made). We can also require a split to reduce the impurity by a specified amount or specify the maximum depth of the tree"
		- Label weights for the ara_interpro_GO dataset: -> I could only find the excerpt:
		"we weighted each label with 0.75^d, where d is the depth of the label",
		nonetheless a label can have multiple values for d since it may have several parent nodes. How do you define the weights here?

I appreciate the effort of providing the source code to your method, nonetheless, to make it even further reproducible, I would suggest including the data partitions as well.
Additional comments: The novelty of this work is rather limited. The proposed method is a direct combination of two already proposed methods in the literature by the same research group, namely oblique predictive clustering trees and semi-supervised predictive clustering trees. However, as I understand it, novelty should not play a role in the evaluation for this journal.

What is the size of trees generated by your method? Since your method allows multiple features per split, I assume that fewer splits should be enough to build a model, nonetheless that should come with a trade-off in interpretability and performance. For instance, how does one oblique split with 10 features in its hyperplane compare with a decision tree with 10 nodes on its path? I recommend the authors to provide a comparison between the number of features on average involved in performing a prediction.

Finally, I have some concerns about the motivation for the paper.
What are exactly the advantages of employing a linear of combination of features rather than a single one? In the manuscript, I found the excerpt: "The potential advantage of oblique splits compared to axis-parallel splits is presented in Figure 1. What do the authors mean exactly? I can notice that linear combinations may result in smoother splits, but how is that superior? Would you mind elaborating more on that?


Instruction: You are one of the reviewers, your task is to write a review for the article. You will be given the title of the article, the number of the round in which the article is located, and your order among the reviewers.

Input:
Title: SEMI-SUPERVISED OBLIQUE PREDICTIVE CLUSTERING TREES
Review round: 2
Reviewer: 1

Output:
Basic reporting: The authors have addressed all my comments in previous review.
Experimental design: The authors have addressed all my comments in previous review. The experimental design is now correct.
Validity of the findings: The validity of the results is great; the proposed method performs significantly better than the state-of-the-art in some cases, but never significantly worse. Besides, their proposed method run much faster than state-of-the-art.
Additional comments: The paper is overall good. I recommend accepting the paper as is.


Instruction: You are one of the reviewers, your task is to write a review for the article. You will be given the title of the article, the number of the round in which the article is located, and your order among the reviewers.

Input:
Title: IMPLICIT DETECTION OF USER HANDEDNESS IN TOUCHSCREEN DEVICES THROUGH INTERACTION ANALYSIS
Review round: 1
Reviewer: 1

Output:
Basic reporting: This paper presents an interesting and relevant study about the operative used user’s hand when using a mobile application. It is well written and easy to read. This could help on issues related to the design of user interfaces (assuring usability and accessibility) of mobile application, and also in their implementation in order to avoid the use of mobile sensors that impact un battery consumption and other related features. To that end, a model based on the use of machine learning techniques to classify users implicitly and dynamically according to their operational hand was applied, obtained a high level of accuracy which makes the result really relevant for the community of software engineering, human-Computer Interaction and mobile design. The paper presents an up-to-date and good background on the topics used on the study and related studies performed in the same domain. The study was performed with a large set of participants (174), following good experimental procedures and statistical analysis. All terms and concepts used in the paper are supported by references.
Experimental design: This paper is completely whiting the scope of PJeer Computer Science.
Section 3 presents the design study from its definition (section 3.1) to the execution. It clears presents from section 3.2 how the procedures from section 3.1 was applied, all variables analyses, collected data, how this data was statistical analyzed and the conclusions obtained suppirted by the data. However, section 3.1 (Design of the test) limits to describe the test procedure (step by step) performed by the participants. It is really important for the replicability of the study, to present in this section the complete research protocol of the study establishing the plan followed in the experiment, that is : the context selection, the hypothesis formulation, variables selection (independent and dependent variables, selection of subjects (if it’s a a probability or a non-probability sample), instrumentation, the statistical test planned to analyze the results with justification, and validation. All this information is presented as the results are described in the following sections, but in a dispersed way. That means the experiment really followed all rules of empirical studies which gives confidence for the reader and the researches that intend to use the results from this paper. My suggestion is just to value the study and allow others to not only replicate it but use the protocol as a basis for other work. Moreover, having a complete research protocol previously with a complete view of the study helps in the understanding of the execution of the study and analysis of the results.
Validity of the findings: The paper presents very relevant results for the community that work on the design of user interfaces for mobile applications. The conclusion is well stated. The goal of the study is recalled and the performed procedures to address the goal is summarized. The main results and originality of the study are also highlighted.
The conclusion (section 7 ) is followed by another section (section 8) that presents the limitation of the work and future works. In fact, the future works is limited to the last paragraph. I suggest that future works be transferred for the section of conclusion and more explored. Moreover, that the limitations be presented before the conclusion. In other words, I suggest that section 8 be divided and limitations be presented before the conclusion section to better discuss the validity of the sudy.

The validity of the study is discussed in the limitation section explicitly explaining some results and the conditions to be replied, However, as an experimental experiment, threats of validity should be analyzed to better discuss the limitations. The way it is presented does not really address all validity criteria (internal validity, construct validity, external validity and conclusion validity) of an empirical study.  A good suggestion about the threats to validity in experimentation can be foun in the book of  C. Wohlin, P. Runeson, M. Höst, M. C. Ohlsson, B. Regnell, and A. Wesslén, Experimentation in software engineering. Springer Science & Business Media, 2012.
Additional comments: It is a very interesting study with impact results for research community and industry that development of mobile application)


Instruction: You are one of the reviewers, your task is to write a review for the article. You will be given the title of the article, the number of the round in which the article is located, and your order among the reviewers.

Input:
Title: IMPLICIT DETECTION OF USER HANDEDNESS IN TOUCHSCREEN DEVICES THROUGH INTERACTION ANALYSIS
Review round: 1
Reviewer: 2

Output:
Basic reporting: This is an interesting article about the analysis of users interaction on touchscreen. The motivation of the paper is significant as well as the structure is well written. The experiment design of the paper is good as well as the verification also shown very good result.
Experimental design: I suggest the author can describe more about the connection between different phase of the experiments.
Validity of the findings: no comment
Additional comments: No additional comments


Instruction: You are one of the reviewers, your task is to write a review for the article. You will be given the title of the article, the number of the round in which the article is located, and your order among the reviewers.

Input:
Title: IMPLICIT DETECTION OF USER HANDEDNESS IN TOUCHSCREEN DEVICES THROUGH INTERACTION ANALYSIS
Review round: 2
Reviewer: 1

Output:
Basic reporting: This reviewed version answer all the issues I mentioned in the first review.
Experimental design: The authors described the design steps performed for the experiment as required in my previous review.
Validity of the findings: The authors reorganised the section to explicitly discuss the limitations of the work, the conclusions and future work.
The paper is much better.
Additional comments: Thank you very much, for the improvement in the paper.
I believe this study will be very useful for the research community.


Instruction: You are one of the reviewers, your task is to write a review for the article. You will be given the title of the article, the number of the round in which the article is located, and your order among the reviewers.

Input:
Title: SELECTION ON X1 + X2 + ⋯ + XM VIA CARTESIAN PRODUCT TREES
Review round: 1
Reviewer: 1

Output:
Basic reporting: The authors describe rather complex algorithms and data structures in an easy but professional language. English grammar and spelling are (to my knowledge) flawless. The authors improve a well-defined, very elemental problem, for which the amount of literature references is sufficient, also because a short recap is provided in the introduction.

However, since there are a few variations of the algorithm (and its "parts") described, it sometimes was not immediately clear to me which exact variation was referred to in later parts of the manuscript.
E.g. in line 70 "Serang's pairwise selection method" is mentioned. I am confident that the method from Serang (2020) is meant here but I think it would be beneficial if these methods would be introduced (e.g. in line 25) with a certain consistent name or phrase that is referred to later.
Similar to "FastSoftTree" (L. 117), for which I did not find any occurrence in the introduction, though.
In general, maybe a little Table in the Methods would help to clarify which combination of the referenced (or newly developed) algorithms/data structures (for storing the tuples, for making new proposals, for building the tree, for making wobbly selections [if applicable]) are used in "FastSoftTree" vs the new "Cartesian product tree".

In line 63 the authors speak of a "neighboring layer". Until I saw the figure, I was wondering if a "diagonally touching" (i.e. u+1, v+1) layer is neighboring. This could be clarified from the beginning.

I really liked figure 1, it helps a lot to understand the concepts. I would, however, suggest adding a sentence to its caption on why "the total number of items (...) is >= 10". Alternatively, another column for the size of the Cartesian product represented by the tuples or a short reminder that the size of each is u*v. This would make it more self-contained.

Since figure 1 helped so much, I was wondering if the authors could add a similar figure for the actual extension to n (e.g. n=3) vectors, that is described in the manuscript. I am myself not sure if it can be broken down into a simple figure or if this is asking too much, so this is a mere suggestion.
Experimental design: The research question itself is relatively simple and well defined here but it is of great importance to a lot of methods. The extensions/improvements to existing algorithms provided in the article is a welcome addition to the landscape of methods and described in detail. The evaluation of the presented algorithm is sufficient. The speed improvement in practice is clear.

It seems like it was not mentioned in the text, how the test data was exactly generated. On bitbucket "srand" is mentioned. It would be good if this would be shortly clarified in the text. The used random seed could be mentioned but results will probably differ on other platforms anyway.

I was also missing a short comment about memory consumption. They are the same in theory but are they (roughly) the same in practice? It does not necessarily need to be evaluated if it is similar/negligible.

Lastly, a small suggestion: Does the number of data points in the Tables already justify a simple line plot? I think it would significantly reduce the time needed for readers to see the large improvements.
Validity of the findings: The source code is provided and I managed to compile and run it. I am confident that the results can be reproduced (through e.g. the seed for random numbers).

Regarding the Discussion of the results, I was struggling a bit to put LL167-189 of the Discussion into context. Which exact problem, observation, or use case does it discuss?

I was also wondering if the composition of the vectors has any impact on the algorithms. Specifically, do long stretches of equal numbers have any impact? Maybe this can shortly be discussed if I am not missing something obvious.

I assume that the speed-up translates to different alphas? It would be nice if the authors could at least provide an argument on why this should be the case, otherwise, it would probably be good to include another alpha to the test data.
Additional comments: Minor potential errors:
- L 115 "cumulative size the layers": There is probably an "of" missing.
- L 118 unclear usage of "both"
- L 134 "(2^2)^2 = 8": Should be 16 (actually depends on the level of the second child I guess but from context, a full tree is assumed here)
- L 148 "(Table 1": Closing parentheses missing
- L 149 and 162 "m^log2(alpha^2)": Is this supposed to be log_2 in subscript? As in L 120?