Instruction: You are one of the reviewers, your task is to write a review for the article. You will be given the title of the article, the number of the round in which the article is located, and your order among the reviewers.

Input:
Title: MODULAR AUTOMATIC DESIGN OF COLLECTIVE BEHAVIORS FOR ROBOTS ENDOWED WITH LOCAL COMMUNICATION CAPABILITIES
Review round: 1
Reviewer: 2

Output:
Basic reporting: The introduction provides useful background on the automatic design of collective behaviours for robot swarms, and sets the research in the context of previous work on AutoMoDe, clearly stating the aims of the article.

The related work section thoroughly reviews relevant literature on inter-robot communication, covering previous attempts to automatically design communication for multi-robot systems and robot swarms, strongly motivating the work presented.

The rest of the article is well-structured, and presents the research in a logical progression. The supplementary material is a valuable addition, particularly the videos of the generated behaviours.

Although the PFSM notation has been covered in previous work, Figures 3 and 8 would benefit from a legend to explain the symbols used, to make this article self-contained.

There are a few grammatical errors and typos throughout - please proofread carefully before final submission.
Experimental design: The experimental design is strong - the swarm is sufficiently large, standard tools (e-puck robots and ARGoS3 simulator) are used, appropriate case study swarm tasks were chosen, and the performance of Gianduja (and variants) is compared against that of AutoMoDe-Chocolate and EvoCom.

The data gathering is rigorous - experiments are carried out in simulation, "pseudo-reality", and on real robots. The methods are described in sufficient detail to be replicated, and source code is available online in a GitHub repository.

It is unclear why GiandujaE was not tested on real robots in Study A - only in simulation and pseudo-reality. The authors state that "After having validated the ability of pseudo-reality to provide reliable evaluations, we will adopt it as the exclusive device to perform the experiments on Study B. Performing these experiments with real robots would be time consuming and would not give any further contribution to the scientific message of the paper". However, the results shown in Figure 2 are part of Study A, so I expected to see the results of real-robot trials for GiandujaE here.
Validity of the findings: There is sufficient replication of each experimental treatment, and non-parametric statistical testing is used appropriately to validate the findings. The conclusions drawn are consistent with the results presented.

I could not find the raw data that the box plots were generated from though - will this be included in the supplementary material?

There are clear differences in the performance between the pseudo-reality and real-robot experiments for Study A (Figure 2). Carrying out real-robot assessment of GiandujaE in Study A, and for all treatments in Study B would strengthen the validity of the findings. However, if this is infeasible, could you comment further on the expected performance on real robots?
Additional comments: This research into the automatic design of communication is a valuable contribution to the field of swarm robotics, and provides critical insight into the effect of increasing the amount of information exchanged between robots.


Instruction: You are one of the reviewers, your task is to write a review for the article. You will be given the title of the article, the number of the round in which the article is located, and your order among the reviewers.

Input:
Title: LEAN THINKING BY INTEGRATING WITH DISCRETE EVENT SIMULATION AND DESIGN OF EXPERIMENTS: AN EMERGENCY DEPARTMENT EXPANSION
Review round: 1
Reviewer: 1

Output:
Basic reporting: The paper proposes to study the expansion of an emergency department. The authors propose to simulate the current emergency department to evaluate the number of resources (beds) required to meet the demands coming from close care centers and to reduce the length of the stay of the patients. The proposal combines DES and Lean Healthcare.
The conceptual model was implemented with the IDEF-SIM language and the computing modeling with Felxim.
Experimental design: Can you better explain the “Design and analysis of experiments” step. What do you mean with “we chose a complete factorial arrangement for screening the most significant decision variables”

Can you give more information about the DE? The name, location, etc.? If not, please explain.
How did you set the waiting times and the delays?
Validity of the findings: The conceptual model was validated with the review of experts. Experts verified if the model matches the real system. Figure 1, is really difficult to read. Can you better explain this figure, the text at the bottom of the figure and perhaps divide the figure? In this figure you have 3 flows, can you detail which process correspond to each flow?
In section “Model Execution and Initial Analysis” how many days did you simulate? Which is the inter-arrival rate of patients? Did they arrive all at the same time?

Figure 2 is not explained. Why you use an alpha =0.5?  Which is the effect of alpha? What does it means an standardized effect of 200 or 20?

In line 318, what are central points and axial points?
What is “R2 fit”?
In line 322 you say “The results indicated that the best combination of factors is:” How did you get these results?
In line 357: “After the analysis and some adjustments, we determined the ideal number of resources” How did you get this ideal number?
Additional comments: In the introduction autos claim that DES has no additional costs. However, it depends on how you define costs. In fact, there is a cost in terms of time, model designs, adjusting the simulation parameters and validation.
“we proposed improvements by the principles of patient continuous Flow”, which ones?
In line 276, what do you mean with: “increasing the amount of time to add value”

In general, this is an interesting work. However, there are several issues not properly addressed in this paper. The inter-arrival speed of patients is not defined. How do you decide to increase the number of resources is arbitrary. I would expect a more methodologic experiment design to decide how to increase the number of resources, etc. And more importantly, it is not clear how you apply Lean healthcare in this work. Additionally, it would be nice to see the Felxim model used for the experiments.

The English language should be improved to ensure that an international audience can clearly understand your text.
Some examples:
the literature defines as the model development process
allowing observing if there is an iteration between  iteration or interaction?
aims to analyses its expansion
and VA, and some interactions-> Remove “and”


Instruction: You are one of the reviewers, your task is to write a review for the article. You will be given the title of the article, the number of the round in which the article is located, and your order among the reviewers.

Input:
Title: LEAN THINKING BY INTEGRATING WITH DISCRETE EVENT SIMULATION AND DESIGN OF EXPERIMENTS: AN EMERGENCY DEPARTMENT EXPANSION
Review round: 1
Reviewer: 2

Output:
Basic reporting: (1)
The study refers to the hospital under consideration in an inexplicably vague way, as ‘Canadian emergency department’.

The details of the hospital (e.g. name, precise location etc.) should be clearly mentioned.
If there are issues to prevent mentioning such info, then compelling reasons should be given.

(2)
The goals and motivation of the study are mentioned in a very broad perspective, rather than well-defined.The authors need to be more specific about the key motivation behind the study. The reader would be wondering was this work initiated by a request from the hospital, related stakeholders, or the authors themselves?

Stating that the goal was to plan the expansion of a Canadian ED is not quite enough. Simulation studies should include clear-cut ‘questions’ of interest. The care process under consideration should be mentioned clearly, as they would not be modeling the whole processes involved.  I'd recommend adding a section as 'Motivation' or 'Questions of Interest'.

(3)
The study gave a good review of literature and good examples of relevant work.
However, the review unfortunately does not give any critique or positioning of this study with respect to that literature.

The authors only mentioned (briefly) that they contribute to the literature by using Design of Experiments. I’m afraid it is not clear at all how this would contribute to literature.

(4)
Data description is an important missing piece. A thorough description of the fields included in the medical/process records should be provided.  A snapshot of a few records may be presented for the purpose of demonstration. The authors may also refer to a data dictionary if available.

(5)
How the authors actually acquired the data?
Did they use a freely accessible dataset?
Did they submit some request to access the data?
Did they need an approval of research ethics ?

(6)
It’s good to see that the authors are aware of the study limitations. However, I am still wondering whether there could be (under-estimated) limitations pertaining to the quality of data used. Based on my experience with healthcare data, this has been always a concern. This is something that should be properly considered in the limitations.

(7)
I am afraid that the quality of language should be reviewed thoroughly. Many grammar and language mistakes plainly exist.
Experimental design: (1)
Most importantly, the study did not mention any convincing reasons behind using the DES approach. There are actually three approaches of simulation modeling: i) System Dynamics , ii) DES , and iii) Agent-based models.

Going with one of those approaches depends largely on the modeller’s view of the world (i.e. the problem under study). From the study perspective, were patients viewed as aggregates sharing similar characteristics (i.e. System Dynamics), or individual entities with particular characteristics (i.e. DES), or agents with particular behaviour (i.e. Agent-Based)? This needs to be clearly explained.

Sally Brailsford provides a very useful comparison on that issue in the study below. Please consider looking at it, especially Section 8.
Brailsford, S.C. and Hilton, N.A., 2001. A comparison of discrete event simulation and system dynamics for modelling health care systems.

(2)
Please describe in detail the entities involved in the simulation model.
In addition to Figure 1, a further explanation of the timing/events in the simulation model would be helpful for the reader.
Validity of the findings: (1)
As a simulation study, I'd have expected more specific details regarding the verification tests used to verify the simulation model. Examples may be: i) Structure-verification test, ii) Extreme conditions test, or iii) Parameter-verification Test.

(2)
It is hard to find a clear interpretation of the simulation output. The study largely reported the simulation runs and parameters, but the interpretation is ill-described I'm afraid. The discussion of results should be linked to the questions of interest (which are already missing).
Additional comments: No additional comments


Instruction: You are one of the reviewers, your task is to write a review for the article. You will be given the title of the article, the number of the round in which the article is located, and your order among the reviewers.

Input:
Title: LEAN THINKING BY INTEGRATING WITH DISCRETE EVENT SIMULATION AND DESIGN OF EXPERIMENTS: AN EMERGENCY DEPARTMENT EXPANSION
Review round: 2
Reviewer: 1

Output:
Basic reporting: The article has largely improved, and I thank the authors for considering the earlier comments thoroughly. There are some issues that still need to be covered as follows.

(1)
The abstract should be more informative by including more details about the case study under consideration. The reader should get more information about the use case, and the key findings of the study.

(2)
For better readability, the literature review could be divided into smaller sections.

Further, the review can include a background section to get the reader familiar with the main tools used in the study. For example, the IDEF-SIM language can be described in that section. This would avoid explaining such issues in the experimental part.
Also, I recommend mentioning some representative examples in literature that successfully applied the IDEF-SIM in similar contexts.

(3)
Data Description should have a separate section. A better presentation is needed to fully describe the dataset acquired.
Summary statistics of the dataset and its fields should be useful.
Figures and tables can be used effectively in this regard.

(4)
The quality of writing has been revised to an adequate level, though there are still several language mistakes.
The abstract particularly needs further revision.
Experimental design: A sufficient explanation of the simulation entities and their characteristics is still largely missing. For example, what are the particular characteristics of patients included in the simulation model.
Were those characteristics learned from the dataset or some domain knowledge ?
Validity of the findings: The major weak point of the manuscript is the lack of a convincing validation.
The validation is a core component of simulation studies. Without a clear-cut validation, the simulation output would be considered with a large degree of skepticism.

The authors already cite an excellent reference in the context of simulation validation, which is (Sargent, 2013).  Please apply any of the validation methods mentioned in Section 3.2 at (Sargent, 2013).
Additional comments: No additional comments


Instruction: You are one of the reviewers, your task is to write a review for the article. You will be given the title of the article, the number of the round in which the article is located, and your order among the reviewers.

Input:
Title: LEAN THINKING BY INTEGRATING WITH DISCRETE EVENT SIMULATION AND DESIGN OF EXPERIMENTS: AN EMERGENCY DEPARTMENT EXPANSION
Review round: 3
Reviewer: 1

Output:
Basic reporting: I’d like to thank the authors for working thoroughly on the previous comments.
The manuscript has been largely improved.
Please consider the points below in the final version.

(1)
It is not entirely clear if the simulation model is simulating the current situation of ED (under full demand), or it predicts a possible scenario in the future after closing the surrounding care centres.
I am wondering if those centres have been already closed?
If yes, would this decision be possibly related to some economic feasibility?
This generally raises key questions about the purpose of the study.

(2)
A short introductory paragraph would be encouraged at the beginning of the Background section.
This may help prepare the reader about the topics reviewed.

(3)
The language has improved, though there are still several mistakes.
For example:
line 324: Patients (of) CTAS level III and IV
line 333: needs (an) exam
line 344: Table 4 describes (these) data.

Also, please use either British or American English consistently.
For example, analyse (British) / Analyze (American)
Experimental design: It seems unrealistic to assume that staff meetings can be completely removed.
Meetings might be reduced to a sufficient level, but it would not be reasonable to assume that the medical staff need no meetings at all.
There are activities that do add value to the patient’s journey, though indirectly.
Validity of the findings: I am concerned with the simulation output. I believe that the simulated scenarios need to strike a balance regarding the improvement before and after applying the proposed procedures.
For example, the percentage of treated patients went from 17% to over 95%, this seems overly optimistic.
The reader would be left wondering how such low percentages could happen in Canada, reportedly as one of the most performing healthcare systems in the world.
Additional comments: No additional comments


Instruction: You are one of the reviewers, your task is to write a review for the article. You will be given the title of the article, the number of the round in which the article is located, and your order among the reviewers.

Input:
Title: A METHODOLOGY FOR PSYCHO-BIOLOGICAL ASSESSMENT OF STRESS IN SOFTWARE ENGINEERING
Review round: 1
Reviewer: 1

Output:
Basic reporting: - Overall, the paper has been written clearly but there are too many typos, misspellings, punctuation problems and inconsistencies that need to be fixed. A few examples:
Line 13: exposere -> exposure
Line 28: research on of stress -> research on stress
Line 36: Wordl … realiced -> World … realised
Line 192: Open double quotation for “unchangeable”, both are close double quotations.
Lines 194-195: This is becomes … -> This becomes …
Line 196: missing space after semicolon and again problem with double quotations.
...
Line 516: Be consistent with the usage of “alphaamylase” or its other form.
...

- The paper provides sufficient background and it is well structured.

- Figures must be revised to have better quality, especially when printed. All the figures (except figure 1) need better labeling and information support in the paper's main text. For example, it would be nice if you add more information in the text about the range of scores in PANAS. Or what is Div. Responsetime in Figure 4 and why does it have a negative median?
Figure 7 (a): It seems that the sample size is a typo.

- The orientation of Tables and their captions need to be fixed.
Experimental design: The Methodology and case studies are nicely designed and aligned with the aim of the study. The split of each stage of the experiment is sound. However, based on the information that the authors provide as the background of the study, important questions come to mind:

- Section 5.1 states that there is a possibility that stress reactions be influenced by medication or pre-existing mental conditions. To what extent can they tell what portion of the stress reactions are due to secondary reasons rather than the task itself? Which one has a larger effect: the task, medications or mental conditions?
(I also could not find any notes in this paper regarding your participants. It would be nice to report if none of your participants had medical or mental conditions.)

- Section 5.2 points to the limitations of using hormone tests, as the hormone system reacts in the range of minutes to hours and cortisol have a daily cycle. In psychological studies, most of the time there are not strict participation requirements, but it can easily become complicated to find software engineers who accept to participate in such a setting.
As the authors are trying to propose a standard framework for the software engineering domain, do they think researchers can follow such a framework?
Finding participants is a challenge itself. Is there a possibility that these conditions make software engineers refuse to participate?

- What if a participant has a health condition that they are not aware of or they don’t want to share with you? Like Addison's disease or Cushing's syndrome which directly affects cortisol?

- Section 6.1 states that in the pilot study, you split the task into two parts of 25 and 20 minutes. In case a participant finishes the first part in less than 10 minutes, can we expect that the saliva test works (as the system responds in 15-20 minutes)?

- As a side note, section 6.1 does not clearly state which samples are missing. Are the same samples missing for NBack correctness in Figures 3 and 4? Are there any overlaps in missing data from PANAS Indicators and NBack correctness?
Validity of the findings: I understand and appreciate the time and energy that the authors invested in conducting this study. Their study design is thoughtful. Yet, analysis and conclusions are not supported. In its current shape, the methodology does not show enough evidence to be capable of being a guiding framework for future studies.

- First case study (pilot study):
How reliable is 4 and 6 samples to draw any conclusions and move to the next case study?
Authors admit that they did not find anything significant in the pilot study but they state that the study is feasible. Is the feasibility of a study enough to move it to the next stage? If they wanted to just check the feasibility, why didn’t they recruit, let’s say, 5 students? What was the point to recruit 46 people and then conclude that the study is feasible?
In my opinion, the huge amount of missing data in the pilot phase, which led to non-significant results, discredits the paper’s contribution. Hence, my suggestion is to revise the paper’s abstract and do not emphasize the pilot study as a case study and a contribution.

- Lines 522-529: This sample size does not grant enough evidence to even run significance tests. Hence, discussions on Lines 524 and 525 seems pointless.

- Lines 541-543: Any significant stress results? Lines 522 and 523 stated that there was not enough data for the n-back test and results may get meaningful with a larger dataset, how come non-significant results led to ignoring baseline and offset but not other stages of the study?

- Line 557: Master students have been recruited for your pilot study and Bachelor students for the second study. Does the task fit both Bachelor and Master students similarly?

- Lines 577-578: Reporting these findings based on "mean" while the standard deviation is quite large and the median is constant does not seem appropriate.

- Section 7.1 does not seem convincing enough. The cost of other equipment, such as eye-trackers or heart rate monitors, is usually a one-time cost and they can be reused for several studies. Moreover, these listed costs can vary from country to country and the laboratories may not be accessible to many researchers.
Additional comments: The following are some suggestions that can enhance the clarity of the paper.

Line 389: Cite a number of such studies here.
Lines 438-439: What is the exact basis of this advice?
Lines 445-446: Cite the most effective papers that shaped your methodology here.
Line 469: In line 460 you stated that the endocrine system responds in 15-20 minutes after the stressor onset. Why in line 469 you suggest 40-120 minutes for longer studies?
Lines 495-497: It is not clear what was the intention by stating mean, median and standard deviation. Boxplots are a standardized way of displaying five number summary (minimum, first quartile, median, third quartile, and maximum). Did you want to mention the content of Table 1?
Line 498: Line 485 reported a total of 46 participants for the pilot study but in line 498 the total number adds up to 50. Which one is correct?
Lines 510-511: What are those 22 and 17 measurements? It would be nice to cite a reference for it. Why are they not equal for the experiment and control group? What do those measurements tell us? What should I learn from Table 2?
Lines 536-540: Are there any other reasons rather than reducing time and money? If so, please clarify.
Lines 557-561: Add an explanation of why there are missing samples in Figures 5 and 6.


Instruction: You are one of the reviewers, your task is to write a review for the article. You will be given the title of the article, the number of the round in which the article is located, and your order among the reviewers.

Input:
Title: A METHODOLOGY FOR PSYCHO-BIOLOGICAL ASSESSMENT OF STRESS IN SOFTWARE ENGINEERING
Review round: 1
Reviewer: 2

Output:
Basic reporting: The use of the English language is mostly clear and professional with few exceptions which could be easily fixed with a further round of proofreading.
Regarding style, some sentences can be simplified (e.g., Section 1, lines 31–37).

The paper reports enough background to be understood by a technical audience in the field of software engineering not familiar with physiology and psychometrics. An ad-hoc section (Section 3) is devoted to cover the basic required concepts.
Relevant prior literature for software engineering is reported in Section 1 and Section 2. However, I find the second part of Section 1 (starting from line 66) a bit disconnected from the rest. Given its contents, it feels it should belong to Section 2. I understand these paragraphs are necessary for motivating the need for interdisciplinary research, but the gap should be made more clear (at the moment, these paragraphs only enumerate previous interdisciplinary work).

Although it follows a customary order for SE papers, the structure of the article can be improved by moving Section 2 after Section 6 (i.e., the results of the two studies). This way, previous results on stress can be better contrasted with the results reported in this paper (given that the authors agree that reporting such contrast can be of interest). Similarly, factors of interest—e.g., the cost for running a study reported in Section 7—can be easily compared by, for example, formulating a hypothetical scenario in which physiological techniques are used in previous work.

Boxplots comparing different treatments (i.e., groups) should be reported side by side, for a better comparison, in a single plot.
The boxplots axes labels are currently unreadable.
Tables can be arranged so that they do not need to be reported in landscape layout.
The information regarding the results of the statistical test should be reported in the text, rather than in Table 2 caption.
In my opinion, the data is made available to the best extent possible (Appendix A).

The paper, in its current form, is self-contained. The study reported (including a pilot) and the references cited contribute to its relevance with respect to its goals.
Experimental design: The research is relevant to the field of software engineering. However, research questions are not formally defined. A gap is identified in Section 1 but (as commented in the section above) it should be made more clear based on the cited references.
Contributions are clearly stated. The first one is supported clearly in the paper (i.e., Section 4). The second is phrased as "a robust methodology to detect and measure stress in controlled experiments that is tailored to software engineering research."
I have two main objections regarding the latter, i) I understand a methodology to be robust if it consistently (e.g., under several runs) detects what it is supposed to detect. This is not the case according to the (only two) studies reported; and, ii) it is not clear how the methodology (which really is an experimental design + a methodology for stress detection) is specific for software engineering experiments. One could just replace the experimental tasks with tasks specific to another discipline.

The study design seems solid and results are reported for both studies. The authors also reflect on the activities undertaken with respect to ethical conduct and data protection.

The methodology is explained to a good level of details (including figures and annexes) which would aid replication. Moreover, the authors reflect on the cost of replicating the methodology in terms of money, effort, and required skills.
However, the evaluation method should be more precise. For example, "The methodology has been supported by two case studies implemented as controlled experiments..." these are actually controlled experiments (base + replication) and not case studies. I suggest to report them under another lense—i.e., as replications—following the accepted guidelines and reporting standards in software engineering research (see the work of Carver, Vegas, Juristo). Also, the methodology is "evaluated" rather than "supported" by an experiment/case study/etc.
Validity of the findings: The paper does not include a formal "threats to validity" section. However, the authors identify are report some threats in Section 6 and Section 7. I recommend grouping the considerations on the threats to the validity of the experiments in an ad-hoc section (especially in the case the authors will follow my previous recommendation to restructure the paper according to the reporting standard mentioned above).

The paper reports detailed results regarding the experiments as well as interesting lessons learned grounded in their experiences. However, these are targeted to researchers, whereas no implications of practitioners are provided.
Data is reported in the paper in tabular form, and material attached as a supplement to this submission. I believe that, upon acceptance, these materials will be made public.
Additional comments: I am adding minor comments here:
- Please avoid restating the abstract verbatim in the introduction
- The abstract should mention the evaluation methodology and report of a glimpse of the results
- I am not sure the citation style is correct. (Author (Year)) seems odd
- Lines 63–65 "For detecting stress, research in software engineering has focused on machine learning and data mining approaches, wearable technologies, and ad-hoc questionnaires so far (Suni Lopez et al., 2018; Brown et al., 2018; Meier et al., 2018), but we still see some research gaps here." Please, cite the work more precisely (who reported on machine learning, who did so on questionnaires, etc).
- Line 139. "As the difﬁculty of a task can be a stressor, brain waves are an interesting indicator to measure. However, this kind of study needs specialized equipment and the results are controversial." Why are they controversial? These need to be back up by a solid argument and (possibly) reference to existing evidence.
- Line 159. "While all previous studies contributed to our understanding, there are indications that software engineering has been avoiding using robust and validated methodology for stress detection and psychological issues in general, thus threatening the validity and reliability of studies." This is also a strong claim which should be at least better discussed. Can you report an example of such evidence from Graziotin et al. 2015b?
- Line 161. "Most of these studies use non-standard, ad-hoc, and non psychometrically validated questionnaires to assess the stress reaction, either by self-report or a number of questions aimed to derive the personal stress level." Same as above. Please, indicate more precisely which studies use non-standard questionnaires (with examples).
-This is a wish, and not necessarily a critique. It seems that stress can be related to the workplace (e.g., the work culture) and/or the individual's personality. Perhaps, this can be elaborated a bit more in Section 3.
- Line 478. "Salutogenesis is a well-being theory which we have previously proposed for enhancing the interaction of software developers with their tooling (see Ostberg and Wagner (2016) and Ostberg et al. (2017) for details on Salutogenesis)." This can be explained in a sentence without having the reader look for the reference for the basic definition.
- The last two references (World Health Organization) should be footnotes.


Instruction: You are one of the reviewers, your task is to write a review for the article. You will be given the title of the article, the number of the round in which the article is located, and your order among the reviewers.

Input:
Title: A METHODOLOGY FOR PSYCHO-BIOLOGICAL ASSESSMENT OF STRESS IN SOFTWARE ENGINEERING
Review round: 2
Reviewer: 1

Output:
Basic reporting: 'no comment'
Experimental design: 'no comment'
Validity of the findings: 'no comment'
Additional comments: No additional comments


Instruction: You are one of the reviewers, your task is to write a review for the article. You will be given the title of the article, the number of the round in which the article is located, and your order among the reviewers.

Input:
Title: RECONFIGURABLE MONITORING FOR TELECOMMUNICATION NETWORKS
Review round: 1
Reviewer: 1

Output:
Basic reporting: Overall language is good but with some issues though. Several examples:
Line 6: Seems -> Seem (plural form)
Line 9: of is redundant
Line: 42: medical -> medicine
The sentence in lines 57-58 is not clear and should be rephrased.
Lines 89-90 and 100: in dynamics -> dynamically

If possible, the article should be reviewed by a professional English teacher or native speaker.

Regarding other aspects, the authors provided plenty of references, background context is well described and sufficient. Generally speaking, the article is well organized and contains necessary raw data and figures. Though in the introduction authors mention sections, but it's unclear where each section starts. It'll be easier to navigate through the article if authors will add this.
Experimental design: The article meets the scope and aims of the journal. Research questions are well defined in the article introduction and then well described in the article.

Unfortunately, authors do not provide a detailed description of how to replicate their method in some end-user devices. However they provide some source code with the actual implementation of the monitoring method.
Validity of the findings: Authors provided the some examples of real-life problems that could be solved by implementing their method.

There is a clear connection between research questions, which was defined and the outcome of the work.
Additional comments: No additional comments


Instruction: You are one of the reviewers, your task is to write a review for the article. You will be given the title of the article, the number of the round in which the article is located, and your order among the reviewers.

Input:
Title: RECONFIGURABLE MONITORING FOR TELECOMMUNICATION NETWORKS
Review round: 1
Reviewer: 2

Output:
Basic reporting: The text contains clear professional English. The list of references is sufficient. The structure of the article is appropriate. The images have a low level but sufficient informative value. The symbol table needs to be corrected.
Experimental design: The research question is meaningful but not sufficiently defined. Research does not have sufficient bases. The approach presented is exciting and original, but is not adequately substantiated. There is no comparison with other similar methods.
Validity of the findings: The article does not show the evaluation of the present approach. Were data analyzed to build the current situation? How. Compare your approach with available non/dynamic approaches - process mining, dynamic modelling and network optimization.
Additional comments: No additional comments


Instruction: You are one of the reviewers, your task is to write a review for the article. You will be given the title of the article, the number of the round in which the article is located, and your order among the reviewers.

Input:
Title: RECONFIGURABLE MONITORING FOR TELECOMMUNICATION NETWORKS
Review round: 2
Reviewer: 1

Output:
Basic reporting: Authors addressed all my feedback
Experimental design: Authors addressed all my feedback
Validity of the findings: Authors addressed all my feedback
Additional comments: Authors addressed all my feedback


Instruction: You are one of the reviewers, your task is to write a review for the article. You will be given the title of the article, the number of the round in which the article is located, and your order among the reviewers.

Input:
Title: RECONFIGURABLE MONITORING FOR TELECOMMUNICATION NETWORKS
Review round: 2
Reviewer: 2

Output:
Basic reporting: The presented text is not significantly different from the first version of the text. The list of references is sufficient. The structure of the article is appropriate. Images have remained poor quality, for example, figure 2 contains grey remnants of previous captions, and new captions overlap object boundaries.
Experimental design: The text still does not contain a clear justification for the choice of the method in the proposed system. The novelty of the technique and the trivial list of inappropriate ways are not sufficient conviction of the uniqueness of the approach.
Validity of the findings: The text still does not contain an analysis of the proposed approach - the main argument is the synthesis of new programs and many undefined parameters. What is the purpose and benefit of your proposal? Is it more time-efficient, has lower operating costs, set up? Does it have a lower computational complexity? How is the Entropy of the system? Have any optimizations been used?  Nothing like this is discussed in the text.
Additional comments: Your approach is interesting, but you need to add a theoretical justification for its application. Convince readers about the efficiency and bonuses of your design. Perform design analyzes, compare case study results, discuss limitations and disadvantages. In this state, the text is insufficient.


Instruction: You are one of the reviewers, your task is to write a review for the article. You will be given the title of the article, the number of the round in which the article is located, and your order among the reviewers.

Input:
Title: RECONFIGURABLE MONITORING FOR TELECOMMUNICATION NETWORKS
Review round: 3
Reviewer: 1

Output:
Basic reporting: Thanks to the authors for editing the text. The article is now of higher quality; the text is clear and has a suitable structure; literary references are sufficient.
Experimental design: The research question and the justification for the research are already more evident.
Validity of the findings: The text now contains more information on the methods and procedures applied in this article. In the current state, the reader can already sufficiently evaluate the presented research.
Additional comments: Thanks to the authors for the revisions made and for submitting an interesting article.


Instruction: You are one of the reviewers, your task is to write a review for the article. You will be given the title of the article, the number of the round in which the article is located, and your order among the reviewers.

Input:
Title: IDENTIFICATION OF PREDICTIVE FACTORS OF THE DEGREE OF ADHERENCE TO THE MEDITERRANEAN DIET THROUGH MACHINE-LEARNING TECHNIQUES
Review round: 1
Reviewer: 1

Output:
Basic reporting: The article uses clear and technically correct text. Sufficient background and context was provided. The structure of the article is in the standard, good form. All relevant results and explanation was included and proofs are described sufficiently. I found only a typo on page 7 in the reference to the Figure 2 (should be reference to Figure 1) and in the last line on page 7 question marks are visible instead of the reference.
Experimental design: no comment
Validity of the findings: no comment
Additional comments: The article is interesting and technically well designed. The results of this article are relevant and as authors said are in accordance with what has been previously published. Any typos was found besides two errors in references.


Instruction: You are one of the reviewers, your task is to write a review for the article. You will be given the title of the article, the number of the round in which the article is located, and your order among the reviewers.

Input:
Title: IDENTIFICATION OF PREDICTIVE FACTORS OF THE DEGREE OF ADHERENCE TO THE MEDITERRANEAN DIET THROUGH MACHINE-LEARNING TECHNIQUES
Review round: 1
Reviewer: 2

Output:
Basic reporting: The study “Identification of predictive factors of the degree of adherence to the Mediterranean diet through machine-learning techniques” by Alba Arceo-Vilas, Carlos Fernandez-Lozano and co-authors reports the performance of 4 machine learning algorithms in classifying subject based on their adherence to the Mediterranean diet with a multi-domain feature-set comprising a total of 38 variables and 3 sub-set of 32, 16 and 4 variables.
Experimental design: 1. It’s not clear which reference method was used to compare the results of the 4 algorithms. In the “Materials and Methods” section, the questionnaire used in the PREDIMED study is cited and, if this was indeed the reference, it should be specified.
2. Line 123-126. This reviewer did not understand sample size calculation.
3. One of the predictor of the 16-feature model is subscapular circumference. It is unclear what it refer to and it is not detailed in the “Materials and Methods” section.
3. The type of bio-impedance analysis (BIA) reported is not suited to research or even clinical evaluation beside the evaluation of body weight. Total-body BIA can be used used in the clinical setting for evaluation of total body water and thus estimation of fat free mass, but several assumptions are made in parameters not closely linked with impedance, limiting its applicability in research. Some degree of collinearity can be expected from measurement derived from the same measured parameters and indeed the feature selection process seems to have filtered most of BIA related parameters. Three of them remains, bone density, basal metabolic rate and activity metabolism. None of those parameter can be adequately obtained from a BIA measurement (activity metabolism is a vague definition, but to the best of this reviewer knowledge isn’t included in the parameters adequately measured or estimate by BIA). In an unsupervised fashion, it may be interesting to understand why those of all BIA measurement were related to the Mediterranean diet and not other more closely related to impedance. Nevertheless, it is the reviewer opinion that data reduction should also be based on the understanding of the methodology used and known literature data.
4. Beside BIA measurements, other parameters in the 16-parameter model seem highly related: circumference of the arm and brachial area, waist-hip ratio and circumference of the waist.
4. While performance was on par or lower than the selected model, detailing the 4, 32 and full feature model could still provide insight in the feature selection process. At minimum the full feature model should be reported.
5. Line 387. It’s not clear how the results from this study have served to “quantify the importance each factor has in the determination of the yield”. The authors report at lines 382-384 that “The best model obtained was an SVM with an RBF kernel as a decision function. The importance of each one of the predictors cannot be studied because a nonlinear SVM is like a black box and the internal mapping function is unknown. Furthermore, the weight vector cannot be explicitly computed.”
5. Some other limitations should be noted: * the PREDIMED questionnaire provide an operative definition of Mediterranean diet, food consumption recollection would have been a better reference method * no reference method were used for the assessment of body composition * no biochemical parameters were included * …
The methodological limitations in the assessment of nutritional status greatly hinders the quality of the results. If the authors are not able to address these issues, the study should only be regarded as a proof of concept on the applicability of machine learning techniques in biomedicine.
Validity of the findings: In this reviewer opinion, the application of machine learning techniques in the field of biomedicine can provide valuable insight in several disciplines, including nutrition research.
Additional comments: A value of this study is being one of the first to investigate this topic and possibly providing guidance for other researchers willing to use machine learning in future investigations.


Instruction: You are one of the reviewers, your task is to write a review for the article. You will be given the title of the article, the number of the round in which the article is located, and your order among the reviewers.

Input:
Title: IDENTIFICATION OF PREDICTIVE FACTORS OF THE DEGREE OF ADHERENCE TO THE MEDITERRANEAN DIET THROUGH MACHINE-LEARNING TECHNIQUES
Review round: 1
Reviewer: 3

Output:
Basic reporting: This manuscript is written in a scientific and professional manner, but there are several sections where readability could be improved and some details that should be addressed.
The manuscript is self-contained and effectively addresses the initial hypothesis. Authors have provided a file with the data.
The introduction section provides enough information to contextualise the research presented in this manuscript. However, the structure of this section should be improved as there are different elements that seem to be misplaced (check comments for the authors below).
The methods, results, discussion and conclusions sections require some editing for some aspects that should be corrected.
The number of references is adequate and the references are valid.

Comments to be addressed by the authors:

•	Sentence starting in line 40 reads “The assessment of nutritional status is one of the best indicators, being a methodology that must include three important aspects: a global assessment, a study of the dimension and a study of body composition Ravasco et al. (2010).” What for is this an indicator? Maybe this shouldn’t be in a new paragraph but just continuing the previous one.
•	Paragraphs between lines 69 and 76 seem are difficult to contextualise in their current location. These two paragraphs seem to be disconnected from the sections above and below (the construction of the sentence implies that there is some relationship with something that has been previously stated but it is not the case here.
•	Section between lines 77 and 104 seems to be strangely arranged as authors introduce the work conducted in this manuscript followed by a section describing machine learning techniques (ML) (lines 86-98) and then return to describe the objectives of this work. The logical structure would probably be the introduction of the ML techniques followed by the aims and description of the methodology applied in this manuscript
•	Sentence starting in line 114 is confusing, it should be revisited.
•	Paragraph starting in line 221-227 is difficult to read and confusing, what do they mean in this context with reducing the bias in the dataset? Are they referring to the use of different feature selection techniques?  This should be clarified. Equally there is some redundancy in the explanation of the cross-validation approach used.

•	There are problems with figure numbering. There is no reference in the text to figure 1. Probably line 325 refers to figure 1.
•	Figure captions should be better explained and should contain a better description of the contents of the figure and should provide explanations about what are the different axis in the figures.
•	Lines 334 and 335 refer to figure 2b but they probably refer to figure 1. This should be corrected.
•	In line 341 there is a call to figure ??
•	Line 345 needs to be corrected as it does not make sense.
•	References are not properly formatted in the text as they are not fully enclosed between parenthesis, in the reviewed manuscript only the year was between parenthesis.
•	There are a large number of references that are in Spanish. It would benefit the manuscript if the authors could provide supporting evidence in the literature in English.
Experimental design: The paper is well designed and effectively develops and applies different machine learning techniques to assess the initial hypothesis.
The methodology is well described and should suffice to replicate the results.
The methods described in the manuscript are rigorously applied.
There are however some aspects that should be addressed:
•	In line 187 what was the criteria used to define the ex-smokers?
•	In line 253 authors mention that there is a multiple testing correction. They should state the methodology used for the correction and should refer later one to adj. p-values in the manuscript if they are presenting adjusted/corrected p-values.
•	In line 315 authors refer that variables were “scaled to the example” What did they mean? Were they trying to explain that they standardised the data using a z-score? - This should be clarified.
•	Results of the Shapiro-Wilk test should be included. (line 345)
•	What was the confidence level used for the Bartlett test? (line 348)
Validity of the findings: The conclusions are well supported with the data and methodology described in the manuscript.
There are some comments that should be addressed:

•	In line 353, there is a conceptual error as Null hypothesis are not accepted they cannot be rejected which is different.
Additional comments: No additional comments


Instruction: You are one of the reviewers, your task is to write a review for the article. You will be given the title of the article, the number of the round in which the article is located, and your order among the reviewers.

Input:
Title: IDENTIFICATION OF PREDICTIVE FACTORS OF THE DEGREE OF ADHERENCE TO THE MEDITERRANEAN DIET THROUGH MACHINE-LEARNING TECHNIQUES
Review round: 1
Reviewer: 4

Output:
Basic reporting: This paper has a potential to be accepted, but some important points have to be clarified or fixed before we can proceed and a positive action can be taken.We here summarize this points:1. It is really unclear to me the why  authors take into consideration only few factor in study  of mediterranean diet through machine-learning . This is an important point before we can start to reason about the proposed method seriously. We need to understand precisely if the proposed mechanism is able to face with the following:-different origin may  have different food intake ranges and different medication depending on the other factor of body type
Experimental design: Give details regarding follow-up examinations.  The authors used only 14 question for survey.  Was  any other past medical history  obtained  from  two population   group?
why use small data size ? did you try any other kernel beside RBF?
Validity of the findings: The authors stated there was no higher statistically significant with comparison previous such study . The authors should state the exact  number of  factors  which significant  for mediterranean diet. The author should give clear idea for factor which consider as positive aspect and which consider as negative aspects
Additional comments: The manuscript was poorly written and many grammatical error.The study must use large data size when you apply RBF kernel. The study must  provide previous study examples of diet factor and compare the data.


Instruction: You are one of the reviewers, your task is to write a review for the article. You will be given the title of the article, the number of the round in which the article is located, and your order among the reviewers.

Input:
Title: IDENTIFICATION OF PREDICTIVE FACTORS OF THE DEGREE OF ADHERENCE TO THE MEDITERRANEAN DIET THROUGH MACHINE-LEARNING TECHNIQUES
Review round: 2
Reviewer: 1

Output:
Basic reporting: This new version of the article has successfully addressed the issues previously identified by this reviewer.
The new text included in the manuscript has improved it however there are aspects that need to be addressed as there are sections that remain difficult to read or confusing.
There are changes that did not appear highlighted in the text
Experimental design: The experimental design has been improved providing some relevant additional data such as the power of the analyses fo the given sample that was analysed and extended explanations about the results of some of the statistical analyses used throughout the manuscript.

As it was stated in the previous review the paper was well designed and contains enough information to reproduce it.
Validity of the findings: No comment
Additional comments: The explanation for the calculations of the sample size needs to be clarified, not as much in the methodology, although including the formula used would be a great addition, as in its explanation as it is confusing for the reader.

The explanation about the measurement of the subscapular skin fold is repeated (lines 163-168) and then the old explanation remains at 174-176.

Placement of the new text for the adherence to the Mediterranean diet could be improved. It would make more sense if it follows lone 206, as it provides additional information about how the test is built. In this regard there is some confusing information in the new text as line 227 mentions 16 items but in the rest of the text lines 205 and 230 refers to 14 items. Which is correct?

There should be coherence in the use of digits or words for numbers (i.e. in line 205 the use of “14” and in line 220 “fourteen”)

English grammar should be thoroughly revised as there are some errors throughout the text.


Instruction: You are one of the reviewers, your task is to write a review for the article. You will be given the title of the article, the number of the round in which the article is located, and your order among the reviewers.

Input:
Title: IDENTIFICATION OF PREDICTIVE FACTORS OF THE DEGREE OF ADHERENCE TO THE MEDITERRANEAN DIET THROUGH MACHINE-LEARNING TECHNIQUES
Review round: 2
Reviewer: 2

Output:
Basic reporting: The paper is well written. The Introduction and Background sections provide useful information for the readers.
Experimental design: Such novel kind of experiments are practically possible for predictive factors of the degree of adherence to the Mediterranean diet. Even the multi city datasets, such as the ongoing patients based nutrients intake destining project, are far from meeting the number of nutrients requirement  patients needed for a congestion control study.
Validity of the findings: some information presented is need to be rectify. For example, in table.1, the authors made two major group one is below 65 and above 65, in two groups data must shows with percentage.
Additional comments: The paper entitled "Identification of predictive factors of the degree of adherence to the Mediterranean diet through
machine-learning techniques" studies degree of adherence to the Mediterranean diet with help of SVM model to check the performance, Also identify role for RBF kernel as a precise decision function in predication.It also proposes a new algorithm view to asses Mediterranean diet with machine learning methods.


Instruction: You are one of the reviewers, your task is to write a review for the article. You will be given the title of the article, the number of the round in which the article is located, and your order among the reviewers.

Input:
Title: A NOVEL FULLY CONVOLUTIONAL NETWORK FOR VISUAL SALIENCY PREDICTION
Review round: 1
Reviewer: 1

Output:
Basic reporting: - The paper is well written. The language is clean and understandable.
- The referenced works are insufficient.  For example: "Deep Gaze I: Boosting Saliency Prediction with Feature Maps Trained on ImageNet", "A Deep Multi-Level Networkfor Saliency Prediction", "Predicting Eye Fixations using Convolutional Neural Networks ", "Large-Scale Optimization of Hierarchical Features for Saliency Prediction in Natural Images" papers are related this work too. Some of them also contains multi-scale inference as in the proposed work.
- The tables are readable but the given figures are not. Figure 1, 2 and 3 are not clear and zooming makes them worse. It would be better to fix them.
- Both Table 4 and 5 belong to MIT1003 dataset. MIT300 experiment is missing.
Experimental design: - The authors propose a deep bottom-up saliency model but using "This testing revealed that the most salient objects in the image are detectable by the proposed model." , "...proposed model clearly predicts most of the salient objects within the given images." statements misguide the readers by making them think if it will be compared with salient object detection methods or analyzing the relation between the bottom-up saliency and objectness. The proposed model is a bottom-up saliency model not salient object detection. The authors should fix the mentioned statements to avoid confusion or they should propose a salient object method and compare their results with the related salient object detection methods.
- The methods in comparison tables are not sufficient. It does not contain the state-of-the-art methods as SalGAN, Deep Gaze I - II, ML-NET etc. To understand the contribution of the proposed method, the comparison tables should include the mentioned models' results.
- MIT300 results are missing.
- The results are not analyzed properly. The reasoning behind the given scores is not explained.
Validity of the findings: - The results are not discussed.
- The motivation is not clear. The experiments in Table 7 is well-thought but it gives the impression that the authors decide to model based on the experimental results only instead of a scientific reason.
Additional comments: No additional comments


Instruction: You are one of the reviewers, your task is to write a review for the article. You will be given the title of the article, the number of the round in which the article is located, and your order among the reviewers.

Input:
Title: A NOVEL FULLY CONVOLUTIONAL NETWORK FOR VISUAL SALIENCY PREDICTION
Review round: 1
Reviewer: 2

Output:
Basic reporting: 'no comment'
Experimental design: 'no comment'
Validity of the findings: 'no comment'
Additional comments: This paper proposes a new deep learning model based on a Fully Convolutional Network (FCN) architecture. The proposed model is trained in an end-to-end style and designed to predict visual saliency. The model is based on the encoder-decoder structure and includes two types of modules. The proposed video saliency detaction idea is interesting. However, the descriptions of introduction and methods could be further improved, and some important experiments should be added. Many major confused concerns should be modified and clarified as follows.
1) The abstract is too long and redundant. The authors had better write the part of the method clearly and concisely.
2) In Section 1. Introduction, the authors lack a sufficiently comprehensive review of the related work. The authors should add a new subsection “Related works”, and they should they should carefully introduce these related works about saliency detection as follows:
*Saliency-aware video object segmentation, IEEE TPAMI
* Inferring salient objects from human fixations, IEEE TPAMI
*Correspondence driven saliency transfer, IEEE TIP
* Deep visual attention prediction, IEEE TIP
* Video Saliency Detection Using Object Proposals, IEEE TCYB
* Revisiting video saliency prediction in the deep learning era, IEEE TPAMI
* Video Salient Object Detection via Fully Convolutional Networks, IEEE TIP
3) In Section 2. Material and methods, Figure 1. Architecture of the proposed model should be reproduced, and the authors should add more details to explain the architectures of their network.
4) In Section 3. Experimental results, the authors should add more training and implementation details for better presentation.
5) In Section 3. Experimental results, the comparison results are not enough. The authors should add more experimental comparisons on more benchmarks (like [ref1] Revisiting video saliency prediction in the deep learning era, IEEE TPAMI).
6) The computational speed of the proposed algorithm is not mentioned, it is better to add one table to list the computational statistics with the state-of-the-art methods.
7) There are still some typos and grammar errors, and the authors should do a more careful proof-reading for the reversion.


Instruction: You are one of the reviewers, your task is to write a review for the article. You will be given the title of the article, the number of the round in which the article is located, and your order among the reviewers.

Input:
Title: A NOVEL FULLY CONVOLUTIONAL NETWORK FOR VISUAL SALIENCY PREDICTION
Review round: 2
Reviewer: 1

Output:
Basic reporting: The improvements are very good. There are some typos in the manuscript. Some of them: In introduction section: "... human eye movement and visual attention [8]" reference should be fixed, "...performance of Visual saliency" should be "visual saliency", "Semantic Segemnation ", "34. Experimental results " section "...are described in detial." should be fixed. The authors should review the manuscript for other typos.
Experimental design: According to the MIT300 results, SALGAN, ML-NET outperforms the proposed approach. The authors should include these methods for other datasets, too. SalGAN and ML-NET  implementations are currently avaliable, the authors should generate results for other datasets by using them. Also ML-NET provides generated saliency maps for MIT1003 dataset, which authors should definitely include this results.
The analysis of each module is beneficial to observe their contribution to performance. But the authors should do a fair comparison as mentioned previously.
Validity of the findings: No comment
Additional comments: No additional comments


Instruction: You are one of the reviewers, your task is to write a review for the article. You will be given the title of the article, the number of the round in which the article is located, and your order among the reviewers.

Input:
Title: DISCRETE NATURAL NEIGHBOUR INTERPOLATION WITH UNCERTAINTY USING CROSS-VALIDATION ERROR-DISTANCE FIELDS
Review round: 1
Reviewer: 1

Output:
Basic reporting: •	Clear and unambiguous, professional English used throughout.
The article is written in English and uses clear, unambiguous, technically correct text. However, sometimes the manuscript is written in first person. I suggest avoiding the use of “I” or “We”. Instead writing “I present an approach” use “this article presents an approach” (example of the lines 62/63).

•	Literature references, sufficient field background/context provided.
The article includes sufficient introduction and background to demonstrate how the work fits into the broader field of knowledge.

•	Professional article structure, figures, tables. Raw data shared.
The structure of the article is conform to an acceptable format of ‘standard sections’.
Figures are relevant to the content of the article, of sufficient resolution, and appropriately described and labeled.
All appropriate raw data are available in accordance with our Data Sharing policy.

•	Self-contained with relevant results to hypotheses.
The submission is ‘self-contained,’ represents an appropriate ‘unit of publication’, and includes all results relevant to the hypothesis.
Experimental design: •	Original primary research within Aims and Scope of the journal.
The article is strongly related do geoprocessing and GIS, which are not presented at the Journal`s Aims & Scopes. However, GIS is an useful tools to Environmental Sciences. I suggest the author to write an paragraph that demonstrates the strong relationship of interpolation techniques with the environmental sciences.

•	Research question well defined, relevant & meaningful. It is stated how research fills an identified knowledge gap.
The submission clearly defines the research question, which is relevant and meaningful. The knowledge gap being investigated is identified, and statements are made as to how the study contributes to filling that gap.

•	Rigorous investigation performed to a high technical & ethical standard.
The investigation was conducted rigorously and to a high technical standard.

•	Methods described with sufficient detail & information to replicate.
Methods were described with sufficient information to be reproducible by another investigator.
Validity of the findings: •	Impact and novelty not assessed. Negative/inconclusive results accepted. Meaningful replication encouraged where rationale & benefit to literature is clearly stated.
The article is not a ‘pointless’ repetition of well known, widely accepted results.

•	All underlying data have been provided; they are robust, statistically sound, & controlled.
The data on which the conclusions are were made available in an acceptable discipline-specific repository. The data is robust, statistically sound, and controlled.

•	Conclusions are well stated, linked to original research question & limited to supporting results.
The conclusions are appropriately stated, also connected to the original question investigated, and are limited to those supported by the results.
Additional comments: Please, consider to add the equations for MAE (lines 189 and 191) and MAD (line 200).
Please reference the other spatial interpolation uncertainty methods which estimate error using a global model fitted to all the data simultaneously (lines 221 to 222).


Instruction: You are one of the reviewers, your task is to write a review for the article. You will be given the title of the article, the number of the round in which the article is located, and your order among the reviewers.

Input:
Title: DISCRETE NATURAL NEIGHBOUR INTERPOLATION WITH UNCERTAINTY USING CROSS-VALIDATION ERROR-DISTANCE FIELDS
Review round: 1
Reviewer: 2

Output:
Basic reporting: This paper presents the use of cross-validation error-distance ﬁelds to assess the uncertainty attached to estimates provided by natural neighbour interpolation. The paper appears technically sound and is well illustrated; the manuscript however includes multiple typos (e.g., lines 146, 149, 208) and would need a careful reading.
Experimental design: My main comment relates to the virtual geography experiments:
1) the use of MAE as performance criterion simply indicates that the average prediction errors decrease with the number of data and the strength of the spatial autocorrelation; it doesn’t directly tell me whether the error distance field accurately portrays where predictions errors are large or small. I would recommend that the correlation coefficient between modeled and actual prediction errors be computed and plotted versus n and h, as for the MAE.
2)  the virtual geography experiments were limited to the convex hull around the data points and it would be useful to explore how well the approach performs in extrapolation situation.
Validity of the findings: The virtual geography experiments were limited to the convex hull around the data points and it would be useful to explore how well the approach performs in extrapolation situation.
Additional comments: This paper presents the use of cross-validation error-distance ﬁelds to assess the uncertainty attached to estimates provided by natural neighbour interpolation. The paper appears technically sound and is well illustrated; the manuscript however includes multiple typos (e.g., lines 146, 149, 208) and would need a careful reading. My main comment relates to the virtual geography experiments:
1) the use of MAE as performance criterion simply indicates that the average prediction errors decrease with the number of data and the strength of the spatial autocorrelation; it doesn’t directly tell me whether the error distance field accurately portrays where predictions errors are large or small. I would recommend that the correlation coefficient between modeled and actual prediction errors be computed and plotted versus n and h, as for the MAE.
2)  the virtual geography experiments were limited to the convex hull around the data points and it would be useful to explore how well the approach performs in extrapolation situation.


Instruction: You are one of the reviewers, your task is to write a review for the article. You will be given the title of the article, the number of the round in which the article is located, and your order among the reviewers.

Input:
Title: DISCRETE NATURAL NEIGHBOUR INTERPOLATION WITH UNCERTAINTY USING CROSS-VALIDATION ERROR-DISTANCE FIELDS
Review round: 1
Reviewer: 3

Output:
Basic reporting: The paper is well written and structured clearly.
Below are listed a few minor points in roughly the order in which they occur in the text.

Line 146 – “recognise that the over data cells” should be “recognize that the other data cells”
Line 149  Sentence should start with “To” rather than “The”
Line 211 spelling of  neighbor.

Figure 6. Since the MAE seems to be more interesting than the MAD, it might be preferable to plot the MAE as the colors and the MAD on the z axis. It is easier to judge the value of the colors than the value along the z axis.

Figure 6.  Are the numbers on the color scale correct? In experiments.py, colors are broken into values from 0 to 0.02 (rather than 0 to 0.2) and listing of values in nniMAEMAD variable show values up to about 0.02 rather than 0.2. Tick values on legend are set separately rather than being tied to values in the array.

Supplemental material.
I was able to download and run the python scripts provided with little trouble.
I had to comment out the lines pertaining to the use of LaTex (see below). These lines were not critical to producing the figures and the python scripts reproduced the figures in the manuscript without them.

# Check if LaTeX is installed to plot with consistent math font
from distutils.spawn import find_executable
if find_executable('latex'):
    mpl.pyplot.rc('text', usetex=True)

Including these lines generated the following error.
!Latex Error: File `typelec.sty’ not found

In experiment.py line 116 changed
Dnni.NNI(xyz,C) to dnni.nni(xyc,c)
As the discreteNaturalNeighbourInterpolation.py has a function called nni, not NNI.
Experimental design: The evaluation is a bit cursory and a few suggestions are given for expanding it.

The authors run 100 experiments for each combination of h (amount of correlation) and n (number of samplers). Each experiment has a different underlying simulated geometry and placement of samplers as both are generated with random processes. The MAE is computed for each of the 100 experiments and a median MAE and a median absolute deviation, MAD, are reported in Figure 6. It isn’t clear to me why the results from the 100 experiments are not just combined to calculate one value of MAE. There is no discussion provided about the MAD.  MAD seems to be most strongly dependent on n (rather than h) which could be because the convex hull will tend to include less points for small n and thus each experiment represents a smaller sample size and so there will be more variation between the samples. Some discussion is needed if this metric is included.

Line 210-211 ‘indicate that the cross-validation error-distance field provides more accurate and precise estimates than the natural neighbour interpolation field”   The range of values was 0 to 1. Whereas the range of errors was about 0 to 0.25.  Consequently this statement is misleading as MAE is being used to compare datasets with different scales.

Lines 212-215.  The summary is not particularly useful as is. All units are arbitrary which makes the evaluation dataset general but statements such as “when the number of data points n>= 40…. And spatial autocorrelation h>=1”  are not useful in any other context. Can the authors construct a dimensionless quantity (perhaps number of samplers divided by area multiplied by autocorrelation distance squared) and look at performance as a function of that?

Line 214-215 “should perform well”.  The criteria for good performance should be stated.

Lines 223-225 “… the error estimates may be slightly higher than the real errors…”  It may be worthwhile to include a measure of bias for the estimated error. A quick look at the full distribution of one of the experiments error difference (errorDifference variable in experiments.py) does show a distribution that is skewed positive.  On the other  hand the distribution of the actual error seems symmetric about 0, showing that the interpolation method itself does not produce estimated values which are biased.
Validity of the findings: no comment.
Additional comments: The manuscript describes a method for estimating uncertainty in values obtained from natural neighbor interpolation. The focus is on use for two-dimensional geospatial data as the evaluation dataset is produced using an algorithm designed to produce a simulate landscapes.  However, the technique itself seems quite general. The background and methodology are clearly described. The authors provide all code necessary to recreate their results in the supplemental material. The code is well documented and easy to run for someone with some knowledge of python. The evaluation using the simulated landscapes is adequate to show that the method is of interest. The evaluation is a bit cursory and a few suggestions are given for expanding it. I recommend the paper for publication with some minor revisions.


Instruction: You are one of the reviewers, your task is to write a review for the article. You will be given the title of the article, the number of the round in which the article is located, and your order among the reviewers.

Input:
Title: FORECASTING BITCOIN CLOSING PRICE SERIES USING LINEAR REGRESSION AND NEURAL NETWORKS MODELS
Review round: 1
Reviewer: 1

Output:
Basic reporting: Review of the paper :
Forecasting Bitcoin closing price series using linear regression and neural networks models (#44798)

Sergio Focardi

1. BASIC REPORTING
Given the popularity gained by cryptocurrencies, forecasting Bitcoin prices is an important task. Therefore we consider this paper an important contribution.

However, the authors should address a number of points as follows
a.	The paper includes rather naïve statements such as on line 165 when they write “To facilitate the understanding of this analysis we introduce the concept of unit root.” Unit root is a standard concept in econometrics, the authors should eliminate statements such as the one above. Simply mention techniques that are well known
b.	Not clear why they compare forecasts of Bitcoin with those of three stocks. They should explain why they compare these forecasts
c.	They divide the entire periods in partially overlapping subperiods. Now the idea that markets go through different regimes with different levels of predictability is has been exploited in many papers. However, how to determine the length of each subperiod is a critical issue, even more so given the limited length of series. They choose 200 time steps for each period. Why? Did they try different periods? This would be anticipation of information and it would weaken results. The authors should explain how they choose the length of the subperiods
Experimental design: 2. EXPERIMENTAL DESIGN

They should explain how they choose the length of the subperiods. This is a critical issue that the authors should clarify.
Validity of the findings: 3. VALIDITY OF THE FINDINGS

The findings are credible. Results are not very robust given the limited length of time series. However, this is a problem typical of any study of Bitcoin prices and should be accepted with the warning that future investigations might yield different results.

They should exhibit all results avoiding statements such as the one on line 364 “For the sake of brevity we only show the best results obtained on a speciﬁc time window….” Cherry picking past results is not a rigorous way of conducting econometric studies.
Additional comments: 4. General comments
The paper is interesting but suffer from a certain lack of rigor. The authors should try to avoid useless explanations of well known methodologies and try to be precise about critical issues on testing


Instruction: You are one of the reviewers, your task is to write a review for the article. You will be given the title of the article, the number of the round in which the article is located, and your order among the reviewers.

Input:
Title: FORECASTING BITCOIN CLOSING PRICE SERIES USING LINEAR REGRESSION AND NEURAL NETWORKS MODELS
Review round: 1
Reviewer: 2

Output:
Basic reporting: In the experimental settings, I did not find any error; evaluation metrics are given and methods are described. Time regimes could be better formalized; for example a pseudocode could be given in addition to the current writing.
Experimental design: Experimental setting must be defined with more rigor. For example, instead of "We performed several experiments" the authors can tell what epoch and batch sizes were tried.
Validity of the findings: The results seem valid. The MAPE values are similar to what we have found in our Bitcoin price prediction experiments (2016 MAPE 0.003829 for the first 300 days). However note that this manuscript is a bit behind state of the art models that use graph metrics in addition to price and volume.
Additional comments: In this manuscript, the authors use past price and volume information to  predict Bitcoin price. Their first attempt shows that there is no useful signal in the data; the price walks randomly. Next, the authors use shorter periods and make predictions for each period.

In addition to  BTC, the authors predict three share prices. Using the shares gives a sense of how different BTC is. However, we also have other currencies, such as ether, litecoin and even tokens on platform blockchains, whose price could be predicted. The experiments should be expanded with at least one more cryptocurrency, ideally more. Data. with the same format, can be taken from coinmarketcap.

More comments below:

The abstract is a bit colloquial. Verbs such as "turn out to be" can be avoided.
30: It was possible to demosntrate can be given in active voice : ... we demonstrate...
43: please update the volume (from May 2019) with current values.
45: researcher names are missing in the reference entry, so they are shown as D. or R. in the text.
66: it turned out-> From this analysis we show that ...

Related work does not explain how these works are different from the proposed method. Specifically, how did they perform, what were the weaknesses in them? Some contrast sentences are needed. A major issue is that this manuscript does not consider related work that use onchain metrics other than the volume. For example an early work is "Using the bitcoin transaction graph to predict the price of bitcoin by A Greaves, B Au - No Data, 2015" (I have never met or communicated with these authors).

125: all extracted from ... Does this mean that the authors extracted themselves? Please consider using active voice to avoid such questions. It would be useful to give a URL for every website mentioned.

134: Bitcoin's place in total market capitalization is known as "Bitcoin dominance", currently over 60%. This number should be given with a citation such as https://coinmarketcap.com/charts/

164: references for DF tests and AC plots are complete books. At least, these should be cited with the page number that explains these concepts. Citing a whole book which is beyond a paywall is not helpful.

168: why having a unit root causes inference problems. The sentence needs to be expanded.

187 and 207 and 219: software without authors have blank names in citations, but these libraries have proper publications that you can cite (e.g., pandas: a foundational Python library for data analysis and statistics W McKinney - Python for High Performance and Scientific Computing, 2011).

250: every regime is 200 points wide-> is each point a day?

Table 4 and 6 results are surprising; between left and right panels, results differ by once cell only.

Conclusion: the first sentence can be shorter. What is meant by kindness of a method?


Instruction: You are one of the reviewers, your task is to write a review for the article. You will be given the title of the article, the number of the round in which the article is located, and your order among the reviewers.

Input:
Title: AN EXPERT STUDY ON HIERARCHY COMPARISON METHODS APPLIED TO BIOLOGICAL TAXONOMIES CURATION
Review round: 1
Reviewer: 1

Output:
Basic reporting: The paper as a whole is well structured and entirely readable (I spotted two typos, at Line 637 ‘curing’ should be ‘comparing’, Line 725 ‘legibly’ should be ‘legible’). The introduction gives adequate coverage to both the taxonomic problems/tasks and the visualisation techniques that are plausible solutions. The literature is well covered – occasionally a citation in the text is missing a year (e.g. for Lin et al’s web-based tool). There is only one figure and unfortunately it’s heavily pixelated – I can tell which of the four scenarios is which but the detail in all is obscured – this needs to be redone. Both the raw study results data and the test visualisations and example taxonomies used have been made available via a github account.
Experimental design: The research question is which visualisation is best suited to display the typical relationships between a pair of trees (taxonomies), where the mapping is not always a simple one-to-one relationship, but can involve renamings, splits and aggregations, exclusions and new additions. This is obviously relevant and meaningful to the taxonomists that must frequently understand and occasionally create such relationships between taxonomies.

The methods are described with sufficient detail, in conjunction with the test visualisations and taxonomies, to recreate the study, and the study itself is carried out with a reasonable number of participants (12) and with tasks that representative users consider important.  The experiment design of recording user accuracy in performed tasks (along with audio recording), followed by Likert-style user preference questionnaires and open-ended questions is standard and therefore not a weak point. Details on how long each stage took and the sequencing of the test are supplied.
It appears from the text that rather than have users perform all the questions on each visualisation in turn, all the visualisations were tested on each question in turn? (The order of the visualisations was randomised for each question to avoid learning effects.) It’s unusual in that I haven’t seen it done like this before, but that isn’t the same as saying it’s worse. It would be good if the authors mentioned why they took this particular approach.

There are a couple of possible design flaws in the study. Firstly the study assigns a base taxonomy and one of four artificially generated taxonomies per visualisation condition. Even though the differences between the four generated taxonomies are small, the fact that one is always tied to a particular visualisation type means it could be a slight confounding factor i.e. how much difference in user performance was due to Visualisation A being better than B, or because Visualisation A always used taxonomy 1 and Visualisation B always used taxonomy 2? The authors do pick up on this in their results analysis, but think it maybe because the artificial dataset always associated with the matrix visualisation was slightly smaller, whereas the real problem is that this dataset is always associated with the matrix visualisation. It would have been perhaps preferable to cycle through combinations of visualisation type and artificial dataset when presenting the users with their tasks, rather than just vary the ordering of the visualisation type.
Secondly, with the agglomeration visualisation, colour is overloaded by being used for both the type of relationship a taxon is involved in between two taxonomies, and for indicating which taxonomy it belongs to. The authors again themselves acknowledge this later on.

One other choice I’m unsure about which isn’t revealed until the results stage are the labels for the Likert-type scale. The question is termed “How good do you think each method is in order to perform task t?”, whereas I think ‘effective’ would have been a better adjective than ‘good’. It leads to the unfamiliar situation where the middle term in the five-point scale, which I’d normally expect to be neutral (i.e. ‘average’ or ‘moderate’) is in fact labelled ‘good’.
Validity of the findings: The statistics used to investigate the data gathered from the user study appears to have undergone careful consideration – appropriate non-parametric tests for the Likert-style data and an appreciable effort made in coding the open-ended questions.

In the results section, the matrix condition isn’t mentioned in the last five of the satisfaction level bullet points (line 500-519) which seems curious – is this on purpose or a mistake?

The coding of the open questions / feedback reveals four categories and importantly details about why the users have these objective and subjective preferences e.g. animation and scrolling were seen as frustrating / inefficient so impacted on the impression of those visualisations that relied heavily on those.

The authors’ final findings in the discussion are that while effectiveness was difficult to distinguish between visualisation condition, the users had a clear subjective preference for the edge drawing method. This is something that often occurs in visualisation experiments, there is often little or no difference in performance between alternatives, but very clear preference.

The conclusions are clearly stated, with the clearest outcome being the subjective preference for the edge drawing condition, and ideas for future improvements to the study are made, along with recognition of some of the flaws of the current study.
Additional comments: Possibly, one of the more interesting points within the paper that the authors didn’t detail much was the generation of the artificial data sets. They describe using an artificial taxonomy generator (Line 333) that works on a seed taxonomy. It would have been interesting to see more information on this as artificial data generation is an important topic within data science at the moment and has been touched on in the visualisation field previously, especially for graphs/networks.

In short, there are issues with some of the initial experimental design choices (1.colour in agglomeration, 2.non-neutral middle value in questionnaire, 3.fixed pairing of data to visualisation type), that could be considered to threaten the validity of the results. However, the analysis itself is thorough and the rest of the paper to my mind is sound and doesn’t need much change apart from the points in the review above. I've marked this down as a major revision simply because I'd like to see the changes, even though the amount of text

So what the authors need to do is
1. convince that these design choices do not significantly alter the outcome of the study – in essence counter their own findings in the “threat to validity” section - i.e. would the agglomeration condition still have been difficult to use if marking the two taxonomies in a different way (glyphs, font style etc), given that the distinction between the two taxonomies is always going to be marked with a much less powerful perceptual variable than in the edge drawing, animation or matrix conditions (position getting replaced by line thickness (bold text) or glyph (shape))? Beyond this the alternative is to re-run the study which is obviously a much, much larger undertaking.
2. Generate a much clearer Fig 1.


Instruction: You are one of the reviewers, your task is to write a review for the article. You will be given the title of the article, the number of the round in which the article is located, and your order among the reviewers.

Input:
Title: AN EXPERT STUDY ON HIERARCHY COMPARISON METHODS APPLIED TO BIOLOGICAL TAXONOMIES CURATION
Review round: 1
Reviewer: 2

Output:
Basic reporting: Please see General Comments
Experimental design: Please see General Comments
Validity of the findings: Please see General Comments
Additional comments: This paper describes an experiment to understand differences among different visualization designs for tasks in understanding differences between a pair of biological taxonomies. The paper provides a rare example of a true expert level study for these kinds of tools. While the findings are not surprising, the mere fact that the authors have followed through on such a student should be commended, and provides a contribution to show that it is possible.

I feel this paper has a number of problems with exposition that, if fixed, would make it considerably stronger. On one hand, the paper is good enough that I was able to get the message and be convinced of its value. On the other hand, this took a little more effort, and I may not have gotten the full impact of the results.

1. (small) In the abstract and introduction: be clear the degree to which this is understanding existing designs. This is a little hard because while the broad categories of the designs have been presented, their application to the specific task and their specific implementation do matter.

2. (big) The paper is very cautious to talk about its narrow domain goal. This is refreshing - the authors are cautious not to over-claim. However, the paper never helps me understand the specific challenges of the domain. What is unique about this domain that it requires special consideration that may not apply more broadly? I actually think the insights of this paper may extend more broadly.

3. (small) The introduction made me wonder the degree to which there were good tools for the specific problem (or the general one). Other than research prototypes, have things made it into actual practice? This issue is addressed a little in the related work section, however, even there I am left feeling like the design of tools for the specific challenges of biological taxonomy comparison have not been addressed yet.

4. (small) Around line 56 - I feel like the writing is conflating documentation with the data. This could be made clearer.

5. (big) Around 71-72, the table, and other places in the paper: I feel the paper could better connect the tasks and categorization into existing task frameworks (e.g., Munzner, Gleicher, Schulz et al). Now, the categorization seems arbitrary and unmotivated. It also doesn't help that the categories are only explained later.

6. (small) I misread things such that I didn't realize the whole paragraph at 74 was talking about the pattern identification task (and that the later highlights were kinds of patterns to identify - except then the ideas of summary and overview are introduced). This paragraph is mixing ideas together.

7. (big) @107 - why aren't the methods in use? if they aren't in use, why do we believe they are worthy of study?

8. (big - general comment with small specific) @125 - One experimental design choice the authors made was to create their own implementation of all the methods. This has pros and cons (the methods are uniform, but they may not all be best-of-breed) that should be discussed. Also, the uniformity may not be fair (@128): it could be that some designs make it easier to add interactive features. I don't think this is a problem: but the experimental design choice should be more fully discussed.

9. (small, but connects to big #5) @169 - it would be nice to connect these frameworks to the present work, rather than just citing them.

10. (small) @215 - maybe this is a wording thing, but quantifying subjective response is still subjective

11. (big) The format of the manuscript placing the figures so far away exacerbates the problem that there are not enough of them: please give more an better figures to help the reader. In particular, the features of the program are hard to understand as described as text, and the numerical results are hard to keep track of without graphs.

12. (big/small, connects to #2) As I understand it, a problem in taxonomy comparison is that the taxonomies are huge and have many unfamiliar parts (even to experts). While this discussed as a limitation, it would be useful to know how useful/realistic the kinds of scales considered in the experiments are.

13. (huge) The experimental results only presented in as numbers text are impossible to appreciate. Please present them as a tables and figures. (standard practice is to provide both). It's ironic that a paper on visual comparison provides no visual comparisons to help the reader.

14. (small/big) The paper never really states the hypotheses for the experiments, which makes interpreting the results (what did they show?) a bit harder. Post-hoc, it is wrong to introduct hypotheses, but it might be good to phrase the research questions in the design and results to help the reader appreciate what the results are showing.

15. (small) It would be worthwhile to include tests for order effects.


Instruction: You are one of the reviewers, your task is to write a review for the article. You will be given the title of the article, the number of the round in which the article is located, and your order among the reviewers.

Input:
Title: AN EXPERT STUDY ON HIERARCHY COMPARISON METHODS APPLIED TO BIOLOGICAL TAXONOMIES CURATION
Review round: 1
Reviewer: 3

Output:
Basic reporting: This paper presents the results of a study comparing four alternative visualization idioms for biological tree comparison. The study tries to focus on biological data and tasks, in contrast with other papers in the field which focuses more on general tasks.

The paper is well written, with clear and professional language.

The organization of the paper could be improved though, as the results are buried at the very end of the paper and should be highlighted right at the beginning in the abstract and introduction. Doing so could greatly improve the impact of the paper. Overall, the purpose of the paper is presented too late into the document, I'll suggest highlighting it right on the first paragraph of the introduction.

The journal format greatly affects the readability of these type of papers. Having tables and figures at the end is very detrimental for understanding a visualization paper. The paper shouldn't be published on this format (clarifying that this is the journal's fault and not the author's ). Having said that, the paper didn't include any figure illustrating the visualizations created, the interface, the type of questions, how the visualizations represented the chosen data, etc. This makes it very difficult to judge the results. The main visualization conferences even encourage a video and online demo so that the results are more reproducible, I strongly suggest these to be added.

The paper does a good job reporting the state of the art. It provides good comparisons with previous works, and leverages previous works by others to strengthen the results. However the choice to use the "ten main curation tasks" from a self referenced work is worrisome, this should be acknowledge as a treat to validity and justified in the paper.

For the related work and problem definition, I would also suggest clarifying the type of trees that are used on the study. Not all visualization techniques can be used with all types of trees described on the related work. The trees used for this study seems to be non aggregated trees, with a fixed hierarchy, without numeric attributes, and only labels on the leaves. This categorization is described on my work (already cited) Guerra-Gomez et al. 2012
Experimental design: The experiment motivation is sound, it is well justified and convinced me of the importance of the study. The research question is also well defined, and the methods applied make sense overall. The paper also includes a very good section acknowledging many of the limitations of the study and its treats of validity. The biggest of which was in my opinion the very small number of nodes involved in the tasks (less than 120). This contradicts the purpose of the experiment of finding the best type of visualization for biocomp tasks. Are bio scientists used to analyzed such tiny trees? if so this needs to be explained in the experiment design and more importantly this limitation needs to be mentioned when reporting the results. This is very important because the matrix visualization was viable because of the size of the dataset, and the results suggest that the matrix was the most effective idiom, therefore it should be clarified that this only applies for very small trees as the ones used. On the other hand, I applaud the decision of choosing an unfamiliar taxonomy for the tasks.

The paper doesn't describe how were the participants recruited, or if they were compensated for their participation.

It is also important to notice that randomization of the tasks, with such small number of participants won't be enough to counter learning effects.

It is very remarkable that participants have such broad domain experience, this adds value to the experiment. However, I would like to know what types of visualizations they have experience with, which could help to explain the results.

Many of the decisions made with the implementation were unfounded in my opinion. Examples of this are the choice of color hues and properties which could perceptually favor one task over the other. Not having any way of looking at the interface only make things worse
Validity of the findings: The results, despite being somewhat constrained in a way limited, are important, novel and impactful for the communities. The statistical methods seemed valid as well, although lots of discussions have being rising lately about the dangers of using p-values. Using confidence intervals could be a good addition to the paper, although I will leave this up to the authors and the journal.

The conclusions are well stated, but are very difficult to understand and process. They should be better summarized and presented in a more conclusive way at the very begining of the paper, if not in the abstract itself.

As I mentioned before, having the tables at the end of the paper is extremely detrimental for the reading experience. Moreover the tables are very difficult to interpret, there aren't any legends and they could be great improved by highlighting which differences were significant and which were the best alternatives for each task. Legends should help the reader by explaining if higher values are better or worse. The paper needs figures illustrating the 4 visualization alternatives, the main interface, and examples of how the chosen dataset + tasks were visualized. A video would be even better.

Neither the data, nor the prototype was shared, which greatly affects reproducibility. Many decisions made with the prototype seemed unjustified to me, and should be better defended, such as the choice of colors. Moreover, the paper claims to have left many interactions out of the prototype because "...the software... was not intended to be a final product". I strongly disagree with such statement, even the visualization mantra states that "zoom and filter" are critical for these type of applications, they aren't just bells and whistles that vendors can choose to add or not. Not having these interactions greatly change the scope of the results. This needs to be addressed in the paper
Additional comments: Good paper overall. I think it presents a valuable (yet constrained) results that is relevant for the community. The paper requires in my opinion major revisions that would make it significantly better, but these can be covered in another review cycle.