Instruction: You are one of the reviewers, your task is to write a review for the article. You will be given the title of the article, the number of the round in which the article is located, and your order among the reviewers.

Input:
Title: SELECTION ON X1 + X2 + ⋯ + XM VIA CARTESIAN PRODUCT TREES
Review round: 1
Reviewer: 2

Output:
Basic reporting: I found the basic reporting of this paper to be largely professional and clear. However, I have collected a list of points in the paper that were unclear or needed more references.

Issues of clarity and correctness:
Lines 20, 21, 45, 49, 68, 108, 109: I suggest specifying what computational resource the asymptotic complexity notation you use is referring to. (Examples: in Line 20, replace “takes O(n^2 \log(n))” with “takes O(n^2 \log(n)) time” and in line 21, replace “no O(n^2) algorithm” with “no O(n^2) time algorithm”.)
Line 44-45: The phrase “is able to be created in” suggests an algorithm (i.e. upper bound), but instead you use Omega, suggesting a lower bound. Perhaps the LOH can be created in O(n) time and this is optimal? Please clarify.
Line 80: Since there are m input arrays, wouldn’t it take O(mn) time to LOHify all the input arrays? Please clarify.
Line 109: Capitalize “lemma”
Line 115: replace “the cumulative size the layers” with “the cumulative size of the layers”
Line 148: Right parentheses needed after “(Table 1”
Line 177: The authors state that “the sequence of k values must grow exponentially”. I believe they mean that in order for the online retrieval of top-k values to take the optimal amount of time, the values k_1, k_2, … must grow exponentially. Please clarify whether this is the case. As I understand it, the k_j values are given to the algorithm at runtime, so they don’t necessarily have to grow exponentially, as stated. But they do need to grow exponentially for the algorithms given in the paragraphs at 170 and 178 to be optimal. I believe this is an important distinction that should be made.

Issues of literature references and context:
Line 29-30: What algorithm is faster for small k? Is there a trivial, faster solution for small k? This may be worth clarifying in any revisions.
Line 50: Here the authors describe “Serang’s method of selection on A+B”. If I understand correctly, “Serang’s method” refers to the algorithm presented in Serang (2020). Please clarify whether this is the method being summarized in this section with a citation. It would be useful to readers to be able to know which text to reference in order to better understand Serang’s method.
Line 63: What are the “neighboring layer products” mentioned here? I understand that the procedure described in this section is presented fully in another paper. However, I believe the authors should either give a more complete summary of Serang’s method of selection on A+B or else state explicitly that they are summarizing and make a note of where full details can be found.
Experimental design: I have found the experimental design of this paper largely satisfactory with the exception of a few specific questions about the experimental method and results that should be clarified in revisions.

1. How were the sets X_1, X_2, …, X_m initialized for the experimental evaluations given in Table 1 and Table 2? This information should be given for replication purposes and because it could impact the performance of algorithm implementations.

2. What are the units used in Table 1 and Table 2 for experimental runtime? I assume seconds, but please state it somewhere.
Validity of the findings: The authors found that their algorithm implementations were faster at selection than FastSoftTree, and in particular that Wobbly version of their Cartesian product tree implementation yielded significant speedups over their standard algorithm when m is small and k is large. They also give a detailed comparison of possible approaches to the problem of online computation of top k values in a soft heap and describe why these approaches fail to produce optimal algorithms. However, I find the argument given in the paragraph at 185 about the “layer ordering property” to be vague and too ambiguous to evaluate scientifically. Please consider clarifying your description of the "layer ordering property" and how it pertains to selection on X_1 + ... + X_m. If the layer ordering property is a more speculative aspect of the Discussion, please identify this in your paper.
Additional comments: No additional comments


Instruction: You are one of the reviewers, your task is to write a review for the article. You will be given the title of the article, the number of the round in which the article is located, and your order among the reviewers.

Input:
Title: SELECTION ON X1 + X2 + ⋯ + XM VIA CARTESIAN PRODUCT TREES
Review round: 2
Reviewer: 1

Output:
Basic reporting: I agree that the more detailed introduction/separation of the referenced algorithms suffices. A table should not be necessary.
Thank you a lot for the additional figure, it is everything that I hoped for. I also agree that there is no need to cover more complex scenarios.
Experimental design: The additional clarifications that were added helped a lot and addressed all my concerns. It should also be easier to replicate now.
I understand the concerns about transforming the table into a line plot and see the benefits of the current state.
Validity of the findings: The discussion was significantly improved. It is easier to follow and discusses the few points that were in my opinion missing before.
Additional comments: Potential minuscule errors, if they can be addressed before potential publication:
L31: "et al." not in italics like in the rest of the block (just for consistency)
L36: "Kreitberg" instead of "Kreitzberg"?


Instruction: You are one of the reviewers, your task is to write a review for the article. You will be given the title of the article, the number of the round in which the article is located, and your order among the reviewers.

Input:
Title: A PRODUCT QUALITY IMPACTS OF A MOBILE SOFTWARE PRODUCT LINE: AN EMPIRICAL STUDY
Review round: 1
Reviewer: 1

Output:
Basic reporting: The paper is technically sound and well-written. There is a typo in section four’s title “An project’s…” (line 239).
The background and literature review provided is wide and enough to understand the context in which this work is framed. Also, the paper is well organized.
Experimental design: The experiment’s design is interesting, however, I think a section for the SPL development process (the development of core assets, variability points, etc.) with deeper details would be beneficial to understand both the approach and the experiment’s results.
Validity of the findings: The results are well discussed and all the data supporting them provided, however, I have a few questions that need to be justified in the paper:

- Why the Android app as a SPL? Could the findings be different if the SPL was developed for the iOS app? Why?
- Is there an estimate on how much time, resources and effort the change of methodology from manual development to a SPL consumed?
Additional comments: The paper provides an interesting experiment regarding the introduction of a SPL within the mobile applications’ domain. A visual example of the workflow followed regarding the application engineering process would be appreciated. I would like to know if the authors plan to carry out usability experiments with both apps to test if the SPL introduction had any impact for the user.


Instruction: You are one of the reviewers, your task is to write a review for the article. You will be given the title of the article, the number of the round in which the article is located, and your order among the reviewers.

Input:
Title: A PRODUCT QUALITY IMPACTS OF A MOBILE SOFTWARE PRODUCT LINE: AN EMPIRICAL STUDY
Review round: 1
Reviewer: 2

Output:
Basic reporting: No comment.
Experimental design: My main concerns about this paper are in the experimental design:

1) Why the decision of making a comparison using two different operating systems and programming languages? The results may be biased by the important differences between these two systems. For me, it would have more sense to develop the SPL-based and the non-SPL versions of the system in the same operating system, e.g. Android. In this case, it would be much easier to compare the results and to identify the benefits of using an SPL. Even if the Android project were using both approaches in different periods of the project the results are difficult to be compared.

2) I do not completely understand how in RQ1 the lines of code can be comparable. Even if the application is the same, the operating system and the programming language are different and therefore this has to be also taken into account in order to do the comparison. Once again this RQ would have much more sense if the comparison were between the same applications, being developed with and without a SPL approach.

3) There is not enough information in the paper about the system under development. The description of the system is too informal. A more formal and detailed description of the system requirements, design, etc... would be needed.

4) Figure 2 is not a representation of an SPL. Which notation are you using? Which is the mandatory/optional part? Which are the constraints? I mean an SPL implies certain phases that are not considered in the paper or at least not documented. For instance, there is a differentiation between the Problem Space and the Solution Space. In the Problem Space the variability is modeled using traditional a feature model. This feature model includes optional features, mandatory features, groups and constraints. From this feature model, the number of different valid products can be counted and configurations for each product are automatically generated. In the solution space the code of each feature is developed and connected somehow with the features in the feature model. In case you are using another approach, this should be well documented in the paper.
Validity of the findings: The findings would need to be better organized and better explained.

Organization. There are too much textual information when it would be more readable to have several tables organizing the information.

Explanation. I do not completely understand how the reasoning about the LoC reveals that there is a benefit of using an SPL. I have the same impression with the answer to the other RQs. I do not completely see how the analyzed results help to conclude that the use of a SPL has benefits.
Additional comments: I think the motivation for this paper is very interesting. However, I am not convinced about the experimental design and the findings for this work (as indicated in my previous comments).

In summary;
- There is a lack of information about how the SPL has been defined. What does authors understand by applying an SPL to their project? I think that this should be clarified because Figures 2 and 3 are not a representation of a traditional SPL. Thus, authors need to make clearer the approach they have followed to define their software product line.
- The criteria used to quantify the benefits of using a SPL need to be reviewed. For instance, in some implementations of a SPL the number of lines of code can be considerably increased but this is because the automatic generation of code for the different products introduces some benefits that compensate for the increment in the lines of code. Since here it is not clear how the SPL has been developed it is very difficult to reason about the benefit regarding the number of line of code. The same can be said for the other RQs.


Instruction: You are one of the reviewers, your task is to write a review for the article. You will be given the title of the article, the number of the round in which the article is located, and your order among the reviewers.

Input:
Title: A PRODUCT QUALITY IMPACTS OF A MOBILE SOFTWARE PRODUCT LINE: AN EMPIRICAL STUDY
Review round: 2
Reviewer: 1

Output:
Basic reporting: The paper shows an interesting approach to study the impact of the introduction of the SPL paradigm in the development process of a mobile app. The work is organized and well-written.
Experimental design: With the revised manuscript, the experimental design is more clear.
Validity of the findings: The discussion and limitations are detailed and provide clear explanations regarding the obtained results.
Additional comments: Thank you for the detailed changes and explanations on the revised manuscript. The paper has been clearly improved.


Instruction: You are one of the reviewers, your task is to write a review for the article. You will be given the title of the article, the number of the round in which the article is located, and your order among the reviewers.

Input:
Title: A PRODUCT QUALITY IMPACTS OF A MOBILE SOFTWARE PRODUCT LINE: AN EMPIRICAL STUDY
Review round: 2
Reviewer: 2

Output:
Basic reporting: No comment
Experimental design: I would like to thank the authors for working on all the comments in my previous review. In this review, I went over all my previous comments and the author's response and I will comment on them.

Response #01 and #06. Comparison between and Android and IoS applications.
The response given by the authors to this comment made me understand much better the motivation and value of the work. However, I think that the authors explained this motivation much better in their responses #01 and #06 than in the paper itself. In the paper, this description is split between section 3 and section 4 and I think it should be all explained in the same section.

Recommendation to improve the paper: check again how the work is motivated in the paper and try to move some of the discussion in responses #01 and #06 to the paper. The information is in the paper but slightly reduced and organized and explained in a different way and I think it is much more clear in their response. The research method discusses the two mobile applications but they are not introduced yet (they are introduced at the beginning of section 4). I would move the description before section 4.1 to section 3. The reader needs to understand the context of what has been developed before describing the research method.

Response #7. Line of Codes comparison.
I kind of understand why the authors monitor the line of codes in both projects, but still do not completely understand how the number of line of codes, by themselves, can provide authors with such amount of information: whether products are comparable, that functionalities are the same, that using an SPL does not imply additional changes. For instance, it may easily happen, based on my experience, that for the same functionalities the number of lines of codes may simply be completely different and that doesn´t mean that the two systems offer different functionalities.

Response #9. SPL representation.
Well, I have to say that I disagree that the SPL is "something conceptual and does not imply an implementation". It is not only a concept, and it is not only to have a set of components that are added/removed from a system. Software Product Line Engineering is much more than this and provides clear phases of how to develop a software product line and clear artifacts that have to be defined/implemented in each phase, etc.
Said that I understand that authors have better explain now the mechanism that Android developers used to implement the variation points in the SPL.
Validity of the findings: Comment to Response #10. Better organize the findings.
I was not aware that the journal requested more text than tables or graphical elements. I any case, in my opinion, it is important that the results are presented and discussed in a format that helps to understand the conclusions. I still consider that the discussion about the obtained result is difficult to follow and it is very easy to be lost in the details.

Recommendation: Better explain some sections. For instance, in sections 5.4 and 5.5 authors merely refer to the table or figures, but an explanation of the results and the meaning of that results are missed. In general, when a table or figure is included this improves the readability so I completely agree with them, but they need to be referenced and explained in the text.
Additional comments: - Pre- and post questionnaires and their responses should be included as Annexes.

- I still have problems understanding the results of your experiments.


Instruction: You are one of the reviewers, your task is to write a review for the article. You will be given the title of the article, the number of the round in which the article is located, and your order among the reviewers.

Input:
Title: A SYSTEMATIC REVIEW OF AUTOMATED PRE-PROCESSING, FEATURE EXTRACTION AND CLASSIFICATION OF CARDIOTOCOGRAPHY
Review round: 1
Reviewer: 1

Output:
Basic reporting: The manuscript presents a systematic review on CTG.The techniques,datasets , limitations and recommendations are presented using 50 articles.
Overall English language needs improvement.
A detailed review with more techniques might have been included.
Experimental design: .A detailed inclusion of classification techniques can be done.
Validity of the findings: Tables and figure names are inaccurate.
Table 2 and 3 shows the same title.
Review of Feature extraction techniques should be extensive.
Additional comments: No additional comments


Instruction: You are one of the reviewers, your task is to write a review for the article. You will be given the title of the article, the number of the round in which the article is located, and your order among the reviewers.

Input:
Title: A SYSTEMATIC REVIEW OF AUTOMATED PRE-PROCESSING, FEATURE EXTRACTION AND CLASSIFICATION OF CARDIOTOCOGRAPHY
Review round: 1
Reviewer: 2

Output:
Basic reporting: The article is written in English, the text is technically correct in general. However, it contains some unsuccessful expressions and errors, for example:
•	Line 93-94: If deceleration is perceived with contraction, possibility indicates with hypoxia - apparently meant it probably indicates hypoxia.
•	Line 99-100: However, the interpretations made by humans are inconsistent namely about the traces and variability of inter and intra high observer - it is not clear what the authors wanted to say.
•	Similar examples are found in lines 67, 85, 92, 107, 108, 131, 179-180 etc.
The article is well structured. The authors should be commended for the successful nomenclature and content of the tables, which are very helpful in understanding the complex structure of the material.
The article is within the scope of the journal.
As far as I know, there have never been such full reviews on this topic before, so the proposed article is of great interest and value.
The Introduction adequately introduces the subject. However, since the article is interdisciplinary, it should be explicitly indicated who exactly it is addressed to - IT specialists, obstetricians, or both.
Experimental design: The survey methodology is consistent with a comprehensive, unbiased coverage of the subject. The review is logically organized, sources are adequately cited.
Authors use two types of classification of the reviewed articles. The second type of classification (Datasets, Techniques of Validation, Performance Measure, Motivation, Challenges, Recommendation) is quite clear and logically comes from the structure of the reviewed articles.
As for the first classification (the authors call it either taxonomy or stages), it is not clearly indicated. For example, the abstract (lines 41-44) says: “First stage discussed the proposed method which presented steps and algorithms in the pre-processing stage, feature extraction and classification as well as their use in CTG (20/50 papers). The second stage included the development of system specifically on automatic feature extraction and CTG classification (7/50 papers).” These expressions are very similar to each other and are not very suitable for taxonomy.
Moreover, lines 208-209 for the same articles (20/50) provide a different basis for the same classification: “The general context of all articles (20/50) in this category is about achieving high performance or increased in accuracy of CTG”.
Authors should be encouraged to define the basis for classification used more carefully and reproduce it similarly in all sections of their review.
Validity of the findings: Conclusions are well stated.
The fragment stating the meaning of the taxonomy of articles (lines 322-351) appears to be the authors' personal reasoning and does not follow from the articles reviewed. According to the journal's requirements (“Speculation is welcome, but should be identified as such”), this fragment should be labeled as speculation.
Essentially, the main conclusions of the review are contained in tables developed by the authors, which provide a comprehensive picture of the state of research in the subject area of cardiotocography. Such clear and compact presentation of the material should be really welcomed.
Additional comments: The article is of undoubted interest and after the elimination of pointed remarks can be recommended for publication in the journal


Instruction: You are one of the reviewers, your task is to write a review for the article. You will be given the title of the article, the number of the round in which the article is located, and your order among the reviewers.

Input:
Title: A SYSTEMATIC REVIEW OF AUTOMATED PRE-PROCESSING, FEATURE EXTRACTION AND CLASSIFICATION OF CARDIOTOCOGRAPHY
Review round: 1
Reviewer: 3

Output:
Basic reporting: This article reports the work that aimed to provide a systematic review to describe the achievements made by the literature from the years 2013-2018, summarizing findings that have been found by previous researchers in feature extraction and CTG classification, to determine criteria and evaluation methods to the taxonomies of the proposed literature in the CTG field and to distinguish aspects from relevant research in the field of CTG.
Experimental design: The study design is sound and detailed.
Validity of the findings: The findings are well summarized. However, I found it quite shallow how the authors reported those findings. For instance, I expected some details on what are the features mostly used, what are the best-reached performance metrics.  Currently, the article reads like a collection of the existing research, however, it can't act as a stand-alone for an adequate literature review of the topic.
Additional comments: Section 2.3 is redundant
Why 5 years?

Section 3 expected details on the best features

Placement of figures and tables could be enhanced

4.4.3. Motivation related to guidelines missing details

4.5.1 shallow


Instruction: You are one of the reviewers, your task is to write a review for the article. You will be given the title of the article, the number of the round in which the article is located, and your order among the reviewers.

Input:
Title: A SYSTEMATIC REVIEW OF AUTOMATED PRE-PROCESSING, FEATURE EXTRACTION AND CLASSIFICATION OF CARDIOTOCOGRAPHY
Review round: 2
Reviewer: 1

Output:
Basic reporting: The revised article meets all the requirements of the journal
Experimental design: The revised article meets all the requirements of the journal
Validity of the findings: The revised article meets all the requirements of the journal
Additional comments: The revised article meets all the requirements of the journal


Instruction: You are one of the reviewers, your task is to write a review for the article. You will be given the title of the article, the number of the round in which the article is located, and your order among the reviewers.

Input:
Title: A SYSTEMATIC REVIEW OF AUTOMATED PRE-PROCESSING, FEATURE EXTRACTION AND CLASSIFICATION OF CARDIOTOCOGRAPHY
Review round: 2
Reviewer: 2

Output:
Basic reporting: This article reports the work that aimed to provide a systematic review to describe the achievements made by the literature from the years 2013-2018, summarizing findings that have been found by previous researchers in feature extraction and CTG classification, to determine criteria and evaluation methods to the taxonomies of the proposed literature in the CTG field and to distinguish aspects from relevant research in the field of CTG.
Experimental design: The study design is sound and detailed from the initial submission
Validity of the findings: I laud the authors for the rebuttal and the revised manuscript.
They managed to answer all my concerns and I think the submission is ready for acceptance.
Additional comments: No additional comments


Instruction: You are one of the reviewers, your task is to write a review for the article. You will be given the title of the article, the number of the round in which the article is located, and your order among the reviewers.

Input:
Title: A CONFLICTS’ CLASSIFICATION FOR IOT-BASED SERVICES: A COMPARATIVE SURVEY
Review round: 1
Reviewer: 1

Output:
Basic reporting: The authors provide a review article on BAS conflicts on the level of IFTTT programming.
The paper is mostly well-written and well-structured. Minor typos can be found in the bibliography, e.g. some references have lower-upper-case issues, such as "iot" instead of "IoT".
Experimental design: Clear design. Suitable literature basis in the domain; but lack of related literature. The the paper has some flaws, see (3).
Validity of the findings: The survey results provided by the paper are mostly valid. However, the major flaw of the work is the following: The authors try to describe some essential (mostly semantic) programming issues which are studied in compiler design and formal language verification / language-based security since decades. If the authors would have used the terminology and knowledge from these domains, most of their work would not be new at all, since surveys/review papers in these domains cover them in more detail and with a clearer description.

Some of the findings are moreover trivial, such as condition inconsistency conflict (CTC).

Other conflicts are not stated correctly: especially the "privacy conflict" mixes very different terms such as privacy, confidentiality, danger (safety) and integrity.

The "CIC" condition is not necessarily always wrong and too simplified.

The "Time-based Conflict" is what computer scientists call a race condition and there are tons of work about this ... since decades.

The "Useless Rule Conflict" is also unsuitable. A typical interpreter/compiler would throw a syntax error there. There is no clear semantic provided.

Due to the lack of formal specifications of the author's provided rules and above-mentioned issues, I propose to reject this paper.
Additional comments: I encourage the authors to carefully study work on compiler design, race conditions, side channels in building automation etc. when revising the paper. The whole research domain that you describe would highly benefit if results from other CS domains are considered.


Instruction: You are one of the reviewers, your task is to write a review for the article. You will be given the title of the article, the number of the round in which the article is located, and your order among the reviewers.

Input:
Title: A CONFLICTS’ CLASSIFICATION FOR IOT-BASED SERVICES: A COMPARATIVE SURVEY
Review round: 1
Reviewer: 2

Output:
Basic reporting: - The writing is clear and understandable though further proofreading is required.
- The font sizes in the figures should be more optimized for print readers.
- Technical keywords are well-introduced.

- However, the title indicates it is specific for BAS while the target papers include generic IoT papers. The authors should clarify the difference in terms of conflict categorization.
Experimental design: This is a comprehensive survey paper on conflict detection for IoT/BAS applications and systems. The reviewer appreciates the authors' contribution here, and gives several comments to improve the study further.

- The process to
Validity of the findings: The authors' other paper (Ibrhim et al. 2020) is highlighted in the last row of Table 2. While it is not unfair to cite their papers, the authors should summarize the basic rationale of why their paper is standing out in terms of their own criteria. The review founds the summary of the technology but not much insight about why it can beat the other technologies.

The reviewer recommends the authors to review the coverage of the papers again especially the arguments that many papers ignore "system policies". System policies are often expressed in other automation services rules. And even further, DepSys paper definitely models system policies as "requirement dependencies".

For IRON paper, it's unfair to argue the proposed tool is "not suitable for non-technical users" given most of the rule detection technologies remain at the algorithm level while the paper proposes more tools for usability.

It's a big burden for readers to validate certain techniques' lack of conflict type coverages. Thus, the reviewer recommends the authors to exemplify several papers to explain why they lack in covering certain types of conflicts in explaining Table 2.

While the authors conclude in the abstract by recommending hybrid model checking, the reasoning is weak in the body. The authors may allocate a couple of paragraphs to explain that.
Additional comments: This paper survey a comprehensive list of papers about conflict types and detection, and then propose categorization. The proposed categories are very helpful for evaluating existing and future frameworks.


Instruction: You are one of the reviewers, your task is to write a review for the article. You will be given the title of the article, the number of the round in which the article is located, and your order among the reviewers.

Input:
Title: A CONFLICTS’ CLASSIFICATION FOR IOT-BASED SERVICES: A COMPARATIVE SURVEY
Review round: 2
Reviewer: 1

Output:
Basic reporting: The authors have addressed my comments.

- decrease size of Fig. 2 so that it matches the size of the other figures, incl. font size
Experimental design: no changes to last review
Validity of the findings: findings are valid; authors have addressed my concerns.
Additional comments: all comments of my previous review have been addressed.


Instruction: You are one of the reviewers, your task is to write a review for the article. You will be given the title of the article, the number of the round in which the article is located, and your order among the reviewers.

Input:
Title: A CONFLICTS’ CLASSIFICATION FOR IOT-BASED SERVICES: A COMPARATIVE SURVEY
Review round: 2
Reviewer: 2

Output:
Basic reporting: The authors provide a sort of systematic review article to classify conflicts in IoT apps with a main focus onto ECA and IFTTT programming.
The paper is quite well-structured but some minor revisions should be enacted.
Some typos:
line 19: tools that capable -> tools capable
line  29: extend -> extent
line 59: for the sake of this work -> what do you mean with this statement?
line 83: intentions interactions -> please clarify
line 85: work as follows -> work are as follows
figure 1: open -> turn on
lines 134 and 141: remove the initial questions
line 143: through -> done by
lines 187-188 are not clear, please re-write them
line 196 remind -> remaining
line 228 techniques are different -> remove different
line 228 analysis among these differences are: -> analysis. Among these differences THERE are:
line 244 which responsible -> which is responsible
line 406 using Feature -> using feature
line 457 In (...) provided -> In (...) the authors provided
line 472 (...) authors explained -> (...) explained
line 580 the mean reasons -> the main reasons
line 592 may be lead -> may lead
line 598 as following -> as follows
line 816 starting checking -> starts checking
line 817 when not such conflicts exist, it forwarding -> when such conflicts do not exist, it forwards
various lines: lack of -> missing of
line 857 in proposed -> in the proposed
Experimental design: The introduction is quite dispersive...I would focus it more on the content of the paper.
Lines 116-140 should be better elaborated with no repetitions
In the searched dataset you did not consider IEEEXplore, is it considered in Google Scholar?
In the review of current conflict's classification, a critique of the current state of the art is missing, moreover I would have classified somehow the reviewed papers, not simply summarizing them
In figures 4 on the acronyms was not defined, moreover, what is the aim of designing Venn diagrams with no superimpositions? with separate spheres? Moreover, the link between the figures and the preceding equations for each conflict is not clear.
The discussion does not present quantitative evaluations, I would introduce some graphs about the temporal trend of publications, the venues, and the categories of conflicts in time and in venues
Validity of the findings: The survey results provided by the paper are valid but more quantitative results and graphs should be inserted to quantify the conclusions drawn by the authors.
Additional comments: No additional comments


Instruction: You are one of the reviewers, your task is to write a review for the article. You will be given the title of the article, the number of the round in which the article is located, and your order among the reviewers.

Input:
Title: A CONFLICTS’ CLASSIFICATION FOR IOT-BASED SERVICES: A COMPARATIVE SURVEY
Review round: 3
Reviewer: 1

Output:
Basic reporting: Some proofreading is still necessary, e.g., do not used "we've" but "we have" in formal writing.
Introduction has been improved.
Experimental design: The drawing of separated Venn diagrams is not yet clear to me.
However, further quantitative investigations have been added.
Validity of the findings: New quantitative results are enough.
Additional comments: No additional comments


Instruction: You are one of the reviewers, your task is to write a review for the article. You will be given the title of the article, the number of the round in which the article is located, and your order among the reviewers.

Input:
Title: IMPROVEMENT OF DEEP CROSS-MODAL RETRIEVAL BY GENERATING REAL-VALUED REPRESENTATION
Review round: 1
Reviewer: 1

Output:
Basic reporting: Authors presented the paper "Improvement of deep cross-modal retrieval by generating real-valued representation".

In the paper , author have proposed the framework IDCMR that preserves both intra-modal and inter-modal similarity.

From the abstract , it's not very clear the intent and the main purpose of the paper.
It's written in the abstract that "generates real-valued representation. ", so please mention the properties, whose real-valued representation is generated by model.
Experimental design: In the Introduction section, it is mentioned that "different modalities to provide comprehensive information about any concept or topic.". So mention the different modalities here.

Figure 1 represents the modality from both the benchmark dataset or it shows other info as it's written in the introduction that "difference between them is shown in Figure 1" . So what is that difference?

In the Problem Formulation Section : "semantic similarity between different modalities are preserved in a common sub-space". What is sub-space here and explain the intent behind writing the word preserved as it's not very clear.?

What is the dimension of the similarity matrix as it seems from the problem formulation , i varies from 1...n and j varies from 1 ...C, so the size should be n x C , but then under what condition , Sij = 0 ?
Validity of the findings: No Comments
Additional comments: Good Number of models have been compared in the result section and the author has established the effectiveness of the proposed method.

From the Results and Discussion : It is not very clear why the proposed model performs well against other existing models so please explain this as well.

Elaborate the conclusion section more and clearly state 2-3 future scope of the work.

An improvement in overall improvement in quality of English writing is required at some places especially the "Introduction" and "Material and Method" section.


Instruction: You are one of the reviewers, your task is to write a review for the article. You will be given the title of the article, the number of the round in which the article is located, and your order among the reviewers.

Input:
Title: IMPROVEMENT OF DEEP CROSS-MODAL RETRIEVAL BY GENERATING REAL-VALUED REPRESENTATION
Review round: 1
Reviewer: 2

Output:
Basic reporting: The article more or less satisfies the standards. There may be few undefined terms in the equations which may be completed. The article requires several corrections in its formatting. Many of the equations are ill formatted. Many figures should be better presented.
Experimental design: The article addresses a focused research question. It aims to generate a common representation vector that captures semantic similarity of inter-modal and intra-modal data points. Experiments are rigorously designed. Reported results a re well discussed.
Validity of the findings: The experimental validity of the findings are well established. Authors propose a deep representation techniques for images as well as text in a common subspace. Goal is to preserve semantic as well as content similarities. The method presented is a step in this direction. A neural architecture is adopted for this purpose.
Additional comments: The paper attempts to obtain a common subspace representation of two modalities namely, text and image. The common representation treats multi-modal objects as pairs and generates an unified vector representation. A neural architecture is proposed for this purpose. The architecture is an adoption of Glove and other popular neural embedding techniques.

The embedding optimise intra-modal similarity as well as inter-modal semantic proximity. A category side information is used to obtain the corresponding loss function. Stochastic gradient descent with batch normalization is used to train the network.

Results are presented for a number of benchmark cross modal retrieval data sets like Xmedia and Flickr. Standard evaluation measures are compared. The proposed method do perform comparably with state of art techniques.

In general the paper addresses and important problem and proposes an effective solution. Some of the points may be addressed by the authors.

1. Some of the recent works may be referenced. For example none of the techniques that embed relation between the entities beyond category invariance has been cited. Here is a recent reporting on this topic.
Y. Zhang, W. Zhou, M. Wang, Q. Tian and H. Li, "Deep Relation Embedding for Cross-Modal Retrieval," in IEEE Transactions on Image Processing, vol. 30, pp. 617-627, 2021

2. A number of embedding techniques like glove, word2vec has been discussed. However, there are a number of studies studying their relative efficacy. This should be taken into account while narrowing down on the embedding used for the cross modal purpose.

3. The paper formatting is extremely poor. A thorough reformatting should be done. Many of the equations are difficult to read due to formatting issues. Some of the terms are undefined.

4. Many of the figures require quality improvement too.


Instruction: You are one of the reviewers, your task is to write a review for the article. You will be given the title of the article, the number of the round in which the article is located, and your order among the reviewers.

Input:
Title: IMPROVEMENT OF DEEP CROSS-MODAL RETRIEVAL BY GENERATING REAL-VALUED REPRESENTATION
Review round: 1
Reviewer: 3

Output:
Basic reporting: Language needs improvement.
There are some typos which needs to be removed and a thorough check for grammatical mistakes is necessary.
Check line spacing. Different for different sections.
The Figures are unclear. This may be due to conversion from one format to PDF but this needs to be dealt with.
Instead of providing the already available mathematical models of the SG, BoW etc, it will be useful to provide some analysis on these methods and the maths behind them.
The algorithm for IDCMR should be provided in algorithm format.
Experimental design: It is not clear how the common subspace (Line 264)for image and text modalities is generated. The explanation and discussion/ mathematical analysis to that end is necessary.
The sigmoid activation (line 272) typically demonstrates exploding gradient behaviour. How is that handled? Can any other activation be more useful instead?
On line 218 : ‘In this work, real-valued coordinated representation is generated, which
219 preserves intra-modal and inter-modal semantic similarity’. Please provide a reference or solid reasoning for this.
The title states about real valued representation, however, it is not so clear how it is achieved in the experimentation.

It may be good idea to provide a few sample examples of the output of the IDCMR framework.
Please specify the hyper-parameters which are used in the final model for individual dataset.
How is the data split carried out ?
It is necessary to carry out cross validation.
The uploaded code has files for DCMR, however the results show multiple other models? Have the authors experimented and obtained the results reported in Fig 6,7,8? If not and these baseline results are taken directly from some standard papers, it is necessary to provide the references.
Validity of the findings: On line 113-115 Authors state “ Traditionally, words are mapped into vectors using bag-of-words (BoW) but fail to preserve semantic similarity between words.”
Where do the recent word embeddings such as GloVe, BERT, Word2Vec fit into this?

Fig 4: are these results obtained by the authors? If not, please provide the references for these values.

Is there any other method for evaluation other than mAP computation? This methods appears inadequate and does not support the claims.

Are there any disadvantages of the proposed method?
Conclusion is weak and I suggest the authors to elaborate more to bring out the contribution and novelty of the work.
Additional comments: The paper has good potential. The paper can be improved; provided the comments are addressed by the authors.


Instruction: You are one of the reviewers, your task is to write a review for the article. You will be given the title of the article, the number of the round in which the article is located, and your order among the reviewers.

Input:
Title: IMPROVEMENT OF DEEP CROSS-MODAL RETRIEVAL BY GENERATING REAL-VALUED REPRESENTATION
Review round: 2
Reviewer: 1

Output:
Basic reporting: No comments
Experimental design: I am generally satisfied with the responses of the authors.
Validity of the findings: I am satisfied by the responses of the authors
Additional comments: None


Instruction: You are one of the reviewers, your task is to write a review for the article. You will be given the title of the article, the number of the round in which the article is located, and your order among the reviewers.

Input:
Title: ORDINAL LOSSES FOR CLASSIFICATION OF CERVICAL CANCER RISK
Review round: 1
Reviewer: 1

Output:
Basic reporting: The writing is unambiguous and easy to follow. Background and related work are clear and rather detailed. "Fully automatic knee osteoarthritis severity grading using deep neural networks with a novel ordinal loss" is suggested to add to the related work.
Experimental design: Considering the ordinal nature of the pap smear cell classification, the authors propose a non-parametric ordinal loss to promote the output probabilities to follow a unimodal distribution.
Validity of the findings: The authors experiment with the proposed loss on the Herlev dataset on multiple CNN architectures. In addition, the authors compare several other losses. Experiments show the effectiveness of the proposed methods.
Additional comments: The authors propose a novel ordinal loss for the pap smear cell classification. They focus on promoting the unimodal distribution of the output probabilities, which is a good insight into the ordinal classification problem. The experiments and evaluations well demonstrate the idea.


Instruction: You are one of the reviewers, your task is to write a review for the article. You will be given the title of the article, the number of the round in which the article is located, and your order among the reviewers.

Input:
Title: ORDINAL LOSSES FOR CLASSIFICATION OF CERVICAL CANCER RISK
Review round: 1
Reviewer: 2

Output:
Basic reporting: no comment
Experimental design: no comment
Validity of the findings: The authors have presented an extensive empirical evaluation of their proposed method, as well as several baselines. While the results are indeed decent, it is not clear to me whether the complexity of the method would hold up to some stronger baselines (more on this soon). For example, the proposed loss is a summation of many piece-wise terms to enforce unimodality (linear in the number of classes) and involves two hyperparameters, whereas it seems to me like there are much simpler ways one could enforce 'semi-unimodality'.

To elaborate on my 'stronger baselines' point, it seems like the main reason why this loss was proposed is because we do not necessarily want a distribution that is purely unimodal (like in the case of PU). Perhaps that is partly because (1) the conditional probability distribution should not be modelled by a unimodal distribution; and/or (2) using PU (i.e. a binomial distribution) would be too constraining since the variance cannot be easily controlled.

To address point (2): a variance-controlled version of the binomial distribution does exist -- called the Conway-Maxwell Binomial (CMB) [2,3] -- which has a variance-controlling term. That means that your network could be modified to have two outputs: (p, v) = f(x), and then you can maximise the log-likelihood of a CMB distribution. (A more heuristic version of this was proposed in [1], but it's essentially CMB.)

Secondly, to address point (1): why not just infer a mixture distribution between a regular softmax (CE) distribution and a unimodal (PU) one? For instance, suppose your network was modified to have two outputs: p_s(y|x) and p_u(y|x), where p_s denotes a regular softmax distribution and p_u the unimodal one, you could simply (for some alpha in [0,1]) construct a mixture distribution between the two: p(y|x) = alpha*p_u(y|x) + (1-alpha)*p_s(y|x). alpha could either be a hyperparameter to tune, or you might even be able to get away with making it a learnable parameter as part of the network. This would make for a somewhat interesting method, since a high value of alpha would put more weight on p_u, essentially acting as a strong regulariser on the distribution.

Thirdly, the more competitive version of the simplest baseline (CE) would be to do a post-hoc label prediction based on the 'expectation trick' found in [1]. Essentially, for some p(y|x), if we assign to each class an integer label [1, 2, ..., K], we take the expected value of this distribution by computing the dot product between p(y|x) and [1, 2, ... K], and round the result to the nearest whole integer. This basically uses all of the probability mass in p(y|x) to make a prediction.

In summary, I would compare your proposed technique to:
- A more competitive CE using the expectation trick
- Use a parametric unimodal (PU) method using the CMB distribution
- Experiment with using a mixture distribution between p_u and p_s

[4] also proposes a solution to (2), using the Wasserstein distance. [5] may also be worth reading.

---

Other less significant points:
- While it was appreciated that the authors tried out a vast range of architectures, perhaps it would make for a better presentation if the number of learnable parameters was stated for each of these architectures. You could then explore performance vs # parameters. It seems like the dataset you have used is extremely tiny, and having excessively large networks could degrade generalisation performance here. If it saves you computational resources, I don't think some of these architectures are strictly needed in the analysis: for instance AlexNet and VGG, which were superceded by ResNets (and for good reason).
- It would be interesting to explore the case where you don't start off with pre-trained ImageNet weights. While I would expect such a network to very easily overfit, it can be controlled with sufficient regularisation (weight decay), and also allow you to explore the effect of having a severely constrained distribution (i.e. PU) in a 'low data' setting.

[1] Beckham, C. & Pal, C.. (2017). Unimodal Probability Distributions for Deep Ordinal Classification. Proceedings of the 34th International Conference on Machine Learning, in PMLR 70:411-419

[2]: https://en.wikipedia.org/wiki/Conway%E2%80%93Maxwell%E2%80%93binomial_distribution

[3]: https://github.com/christopher-beckham/conway-maxwell-binomial/blob/master/implementation.ipynb

[4]: Hou, L., Yu, C. P., & Samaras, D. (2016). Squared earth mover's distance-based loss for training deep neural networks. arXiv preprint arXiv:1611.05916.

[5]: Liu, X., Fan, F., Kong, L., Diao, Z., Xie, W., Lu, J., & You, J. (2020). Unimodal regularized neuron stick-breaking for ordinal classification. Neurocomputing.
Additional comments: No additional comments


Instruction: You are one of the reviewers, your task is to write a review for the article. You will be given the title of the article, the number of the round in which the article is located, and your order among the reviewers.

Input:
Title: ORDINAL LOSSES FOR CLASSIFICATION OF CERVICAL CANCER RISK
Review round: 1
Reviewer: 3

Output:
Basic reporting: no comment
Experimental design: Only averages of 5 folds are given, in order to show the robustness I suggest to provide results of multiple experiments (e.g. 10), then averages and variance / standard deviation or box plots. And in addition I suggest to perform statistical significance tests on the prosed and tested algorithms.
Validity of the findings: The proposed algorithm cost functions are straightforward. It would be a surprise if ordinal classification can benefit from these cost functions in general. Only a rigorous statistical evaluation of the proposed cost functions based on 10 or more data sets utilising more complex statistical evaluation (e.g. Wilcoxons test) in comparison with the other approaches could prove the strength of the proposed algorithm.
Additional comments: see Part 2 and Part 3.


Instruction: You are one of the reviewers, your task is to write a review for the article. You will be given the title of the article, the number of the round in which the article is located, and your order among the reviewers.

Input:
Title: ORDINAL LOSSES FOR CLASSIFICATION OF CERVICAL CANCER RISK
Review round: 2
Reviewer: 1

Output:
Basic reporting: no comment
Experimental design: no comment
Validity of the findings: no comment
Additional comments: Revision well resolved raised issues.


Instruction: You are one of the reviewers, your task is to write a review for the article. You will be given the title of the article, the number of the round in which the article is located, and your order among the reviewers.

Input:
Title: ORDINAL LOSSES FOR CLASSIFICATION OF CERVICAL CANCER RISK
Review round: 2
Reviewer: 2

Output:
Basic reporting: Regarding the unimodal papers cited: currently [2] is called "Poisson unimodal" (PU). This is a mischaracterisation because [2] also explores the binomial distribution. Furthermore, although [1] only considers the binomial distribution (BU), [3] (same authors as [1]) wrote a more recent paper which considers both binomial and Poisson. Therefore, I would suggest that you simply refer to both styles (binomial/poisson) as simply "Unimodal" and cite [2] and [3] together. E.g. from lines 133-147 you simply say something like "Unimodal (da Costa et al (2008), Beckham & Pal (2017)) constrains the output of ..."

Also, the difference between [2] and [3] is simply that [2] is exploring binomial/poisson in the context of deep learning (rather than classical ML), [2] also proposes using a learnable softmax temperature term to control the variance of the distribution (which I mentioned in my first review as the Conway-Maxwell {Binomial,Poisson} distribution). Therefore, in lines 141-147 you should explicitly state whether you are learning or tuning such a temperature term or not. If you're not doing this, then your implementation of PU/BU is going to be more faithful to [3] than it is to [2], and that should be mentioned.

[1]: Da Costa, J. P., & Cardoso, J. S. (2005, October). Classification of ordinal data using neural networks. In European Conference on Machine Learning (pp. 690-697). Springer, Berlin, Heidelberg.
[2]: Beckham, C., & Pal, C. (2017). Unimodal probability distributions for deep ordinal classification. arXiv preprint arXiv:1705.05278.
[3]: da Costa, J. F. P., Alonso, H., & Cardoso, J. S. (2008). The unimodal model for the classification of ordinal data. Neural Networks, 21(1), 78-91.

---

I appreciate that uncertainty estimates have been added to the experiments, though now I have various issues with how they are being presented.

In line 271 you say "In all the Tables of results, the best results for each loss are in bold", followed by "Furthermore, the other results are compared to the best result and a statistical test is used with the hypothesis of them being the same or not". This makes it sound like the bold numbers are simply referring to the "best" value (i.e. lowest error, or higher accuracy), with statistical testing being something else added to the table. When I look at the table, for each row (with each row denoting an architecture), I see multiple bold results. The issue here is that, in most ML papers the bold number traditionally means the "best" value, whereas here I am seeing that it is referring to a whether that result is statistically significant or not. But to make matters even more confusing, the statistical test seems to be done with respect to the *best* loss. For instance, for each row, rather than doing t_test(baseline, loss1), t_test(baseline, loss2), etc... it looks like you are doing t_test(loss1, best_loss), t_test(loss2, best_loss), etc. It would be much easier to interpret the result if each t-test was performed *with respect* to the CE baseline.

I would suggest the following changes to the table:
- Instead of using boldface to determine statistical significance, format the table similar to how it is done in [4]'s Table 2, i.e. a black dot next to the number (x +/- y) indicates whether that result is statistically significant *with respect* to the baseline (CE) loss.
- To also be consistent with most deep learning papers, use boldface to highlight the *best* result (i.e. lowest MAE, or highest accuracy, etc.). You could constrain the "best" result to only be amongst the ones that passed the t-test. This means that for each row in the table, only one of the losses is going to be in bold.

You also need to make it clear in the text if these table numbers are averages from the validation folds or test folds I am assuming that in your k-fold cross-validation setup, for each iteration, 1/k folds is the valid set (for tuning HPs) and another 1/k folds is the 'test set'. That means that when the k-fold cross-validation is completed you will have results for a total of k validation folds and k test folds. Can you confirm that this is the case? (Make this clear in lines 228-236.) *

(*Some experimental setups may opt to have a held-out test set that is completely isolated from the cross-validation loop, but I assume you are not doing this because of how little data you have.)

[4]: Frank, E., & Hall, M. (2001, September). A simple approach to ordinal classification. In European Conference on Machine Learning (pp. 145-156). Springer, Berlin, Heidelberg.

---

Minor comments:

- I would have preferred if the expectation trick result was made a part of the main text rather than relegated to an appendix table, but I won't make a big deal out of this.

---

Typos, etc:

- Figure 5a: Lin et al is mentioned twice
- Line 204: "512 dense layer" --> "512-unit dense layer", same for 256 dense layer
Experimental design: See my comment in Section 1 about being more clear in the text about how the cross-validation is being performed.
Validity of the findings: no comment
Additional comments: no comment


Instruction: You are one of the reviewers, your task is to write a review for the article. You will be given the title of the article, the number of the round in which the article is located, and your order among the reviewers.

Input:
Title: ORDINAL LOSSES FOR CLASSIFICATION OF CERVICAL CANCER RISK
Review round: 3
Reviewer: 1

Output:
Basic reporting: no comment
Experimental design: no comment
Validity of the findings: no comment
Additional comments: Thanks for addressing my concerns. Good work.


Instruction: You are one of the reviewers, your task is to write a review for the article. You will be given the title of the article, the number of the round in which the article is located, and your order among the reviewers.

Input:
Title: COMPACTING OBLIVIOUS AGENTS ON DYNAMIC RINGS
Review round: 1
Reviewer: 1

Output:
Basic reporting: This paper investigates the possibility/impossibility of Compact Configuration Problem (CCP) and Colored Compact Configuration Problem (Colored CCP) of anonymous and oblivious agents in anonymous ring networks.  The CCP requires that agents initially deployed arbitrarily should be located in a continuous segment.  In the ColoredCCP, each agent has its own color, and agents with the same color should be located in a continuous segment.  The paper clarifies the conditions of solvability of these problems.

The results presented in the paper are interesting and valuable sufficiently for publication in PeerJ COmputer Science.  The paper is well-organized and easy to read.

The followings are minor comments.

lines 75-81 (mobile agents within dynamic graphs):
The authors should mention that there are two different models concerning the relationship between the adversary and agents.  One is that the adversary determines the missing edges first and then agents determine their actions, which is the model considered in this paper).  The other is that the adversary determines the missing edges after agents determine their actions.

line 124: "E:N \rightarrow V \times V" should be "E:N \rightarrow 2^{V \times V}".

line 136: Color 1 is considered as max_color but it is confusing since 1 is the minimum among the color set [1..c].  It seems better to use "min_color" instead of "max_color". Also, see the comment to line 420.

line 142: The usage of "blocked" is a little bit confusing for the following reason, and it seems better to make a notice to avoid confusion.  Each agent determines its action based on the (global or local) snapshot that indicates whether the edges incident to the current node is missing or not.  It seems strange that an agent chooses a missing edge as the direction of movement to be blocked.  I guess that "blocked" is introduced for simplicity of algorithm description.  If so, it seems better to make a notice.

line 156: "Round" is defined here, but it is used in several places before (lines 123, 124, 127, etc.).

lines 163, 164: It seems better to replace "\mathcal{G}" with "the footprint G" or simply "G".  A dynamic graph \mathcal{G} is a sequence of graphs so "v \in \mathcal{G}" is confusing.

lines 188-196 (Proof of Theorem 1): The proof contains some unclear points.  (1) I cannot understand what the "states" imply.  Are they "configurations"?  (2) I cannot understand what configuration the sequence, for example, "000100111", represents.  I guess that "0" (resp. "1") represents that the node is empty (resp. occupied by an agent).  But we consider here two-colored agents and the information of colors seems missing in the sequence.

around Theorem 1: It is better to say something (or possible/impossible/unknown) for the case of c=2 and h=3.

line 212: The term "disconnected" is undefined.  It seems to be replaced with "not compacted".  But, it is redundant since any periodic configuration is not compacted.

line 230: It is not clear what local view is represented by [{0,1},{1,1},{1,1}].  I guess that the first, the second, and the last multisets describe the states of nodes at distances 1, 2, and 3 respectively.  But the multiset is not appropriate since it cannot distinguish the node in the clockwise direction from that in the counter-clockwise direction.

line 255: The paper says "with only two non-neighbors agents in the ring, the configuration is a palindrome".  Does it hold for the configuration [010001]?  It is not a palindrome configuration with two agents. (A configuration is a palindrome if some cyclic rotation of this sequence is a palindrome, which is ww^R or wcw^R (where w^R is a reverse of string w, and c is a symbol).

line 290: S_1 and S_2 should be the "maximal" full segments.

Algorithm 1, the definition of S_1 and S_2: S_1 and S_2 should be the "maximal full segments" rather than "non-empty segments".

Algorithm 1, case (a): The direction of movement is not clear from the phrase "all agents not in S_1 move towards S_1".  The same for line 317.

line 326: "empt" should be "empty".

Figure 7:  It is not a palindrome configuration.  Please see the comment to line 255.

Algorithm 2: "neighbors nodes" (three places) should be "neighbor nodes" or "neighboring nodes".  The same for Algorithm 7 (three places).

line 355: The paper says "if only one of them exists, then the configuration cannot be palindrome".  Does it hold for configuration [110010011]?  Here the node of the middle point of the sequence is occupied by agent a.  This configuration satisfies the condition of (a) (the axis of symmetry passes through at least one edge), but it is a palindrome.  This configuration also satisfies the condition of (b), so it can be considered in (b).

Algorithm 2, (b), 3: "there must be two (full) segments of equal size" should be "there must be two maximal (full) segments of equal size".

line 368: "S^− and S^+ be the two segments" should be "S^− and S^+ be the two maximal full segments"

line 375: "C>2" should be "c>2" (a lower case letter).

lines 392, 393: "with color FC" should be "with color max_color".

line 397: "each color but FC" should be "each color but max_color"

line 402: "a color different from FC" should be "a color different from max_color".

line 419: Here the color set is [0,c-1], but it was defined as [1,c] in Section 2 (line 133).

line 420: "A color c_i" should be "A color i". The same for Definition 4 (two places).

line 420: "c_0=max_color" should be "0=max_color".  But it seems strange and the term "max_color" should be "min_color".

Algorithm 4:  It is not clear what "the first" implies in the phrase "the first node in FC".  I understand we can determine a specific agent in FC and v_f as the node the agent is now located at.  It seems better to modify the phrase.

Algorithm 4, Step 1:  The paper says "FC is now used to compute c−1 rally points", but actually c (rather than c-1) rally points (including rp_0) are determined using FC (notice that rp_0 is called the first “rally point”).  It is confusing.

Lemma 3: The paper says "the system reaches a configuration with c−1 almost compact rally lines".  But the configuration can contain compact rally lines.  If a compact rally line can be considered as an "almost" compact rally line, the claim holds; otherwise, the claim does not hold.  It is not clear from Definition 3.

Algorithm 5, Pattern 1: There are three cases.  The first and third both consider the case where there is a missing edge on p.  So it seems better to exchange the first and the second cases to improve readability.

Figure 8: It should be explained that the bold circle represents a rally point.

line 465: "thus" should be "Thus".

Lemma 5:  When FC forms an almost compact line, then there may exist "two" almost compact lines, which is not correctly placed.

Algorithm 6: It is not clear how to define the distance from L_2 to L_3 when L_3 forms an almost compact line.  The distance should be defined from the rally lines of L_2 and L_3.

Algorithm 6: "colored FC" should be "colored max_color".

Subsection 5.2: Section 2 assumes h>2 (line 135) but here the case of h=2 is considered, which is confusing.

line 506: "e_2" should be "e_3".

line 532: The paper says "these three compact lines will intersect".  But it is possible that two compact lines will intersect around one of the edges, and another two compact lines will intersect around the other (antipodal) edge.  Any case of them violates the requirement of ColoredCCP, so the theorem can be proved.

line 564: "C=2" should be "c=2" (a lower case letter).

line 565: "we presented the case for c = 2 colors; here, we analyse the case with more two colors" should be "we presented the case for c > 2 colors; here, we analyse the case with two colors".

line 567: "case" should be "cases".
Experimental design: no comment
Validity of the findings: no comment
Additional comments: no comment


Instruction: You are one of the reviewers, your task is to write a review for the article. You will be given the title of the article, the number of the round in which the article is located, and your order among the reviewers.

Input:
Title: COMPACTING OBLIVIOUS AGENTS ON DYNAMIC RINGS
Review round: 1
Reviewer: 2

Output:
Basic reporting: This paper considers compaction for oblivious agents on dynamic ring networks. This manuscript is almost well written and easy to understand except several parts (stated later). It also has sufficient references for this research and is self-contained.
Experimental design: This research is original one for mobile agents on dynamic rings and combination of mobile agents and dynamic rings is interesting. Also the results presented in the manuscript are almost tight in the sense that the possibility results are given for the cases except the impossibility results for these two problems CCP and CCCP.
Validity of the findings: Although the impact and novelty given to distributed algorithmic field are not so large, the result is useful for researchers working on mobile agents and fault-tolerant ring networks. Conclusions are well stated and contain several interesting open questions.
Additional comments: This paper considers Compact Configuration Problem (CCP) and its colored version (CCCP) on dynamic rings, which changes their topology changes continuously at unpredictable locations and unpredictable times. The  authors show the impossibility and possibility results for CCP and CCCP on dynamic rings. That is, the results in this paper provide a full characterization of the solvable instances for both CCP and CCCP.

This manuscript is well written and easy to understand except several parts and so this manuscript can be accepted if the following conditions are satisfied.

(1)	Since the relation between the impossibility results and possibility ones is very complicated, some tables which show these relations  and clarify the conditions of the impossibility and possibility are necessary and should be inserted.
(2)	Since also in the impossibility results there are several cases holding without the assumption of dynamic rings, these are stated explicitly.
(3)	The proof of Theorem 3: In Figure 2, a_1-a_8 have the same view. Why the authors differentiate a_3-a_6 and a_1,a_2,a_7,a_8?  It should be clarified.
(4)	The sentences of the beginning of Section 6 is incorrect. In the previous section, the case for c>2 is presented?  Also Algorithm 7 (b) treats the case of c=3 but Theorem 13 shows the result for c=2. Something is wrong. The authors should correct this section.
(5)	The following terms should be defined: (local) view and its notation (line290). Although the symmetricity of configuration is defined, but the symmetricity of a set of agents is not defined.
(6)	Some typos: line 135, h > 2 should be h \geq 2? The caption of Figure 3 should be Proof of Theorem 4.


Instruction: You are one of the reviewers, your task is to write a review for the article. You will be given the title of the article, the number of the round in which the article is located, and your order among the reviewers.

Input:
Title: COMPACTING OBLIVIOUS AGENTS ON DYNAMIC RINGS
Review round: 2
Reviewer: 1

Output:
Basic reporting: This paper investigates the possibility/impossibility of the Compact Configuration Problem (CCP) and the Colored Compact Configuration Problem (Colored CCP) of anonymous and oblivious agents in anonymous ring networks.  The CCP requires that agents initially deployed arbitrarily should be located in a continuous segment.  In the ColoredCCP, each agent has its own color, and agents with the same color should be located in a continuous segment.  The paper clarifies the conditions of solvability of these problems.

The results presented in the paper are interesting and valuable sufficiently for publication in PeerJ Computer Science.  The paper is revised carefully with considering the previous review comments, and is now can be accepted for publication at PeerJ Computer Science.
Experimental design: The subject of the paper is within Aims and Scope of the journal and attractive to many readers of the journal..
Validity of the findings: The presented results are new.  The paper is well written and easy to understand due to the detailed and precise proofs.  The conclusion is also well described.
Additional comments: No additional comments


Instruction: You are one of the reviewers, your task is to write a review for the article. You will be given the title of the article, the number of the round in which the article is located, and your order among the reviewers.

Input:
Title: COMPACTING OBLIVIOUS AGENTS ON DYNAMIC RINGS
Review round: 2
Reviewer: 2

Output:
Basic reporting: Since the conditions the reviewer asked in the reviews are satisfied, this article is accepted.
Experimental design: There is no comment about the current article.
Validity of the findings: The article satisfies the reviewer's conditions.
Additional comments: No additional comments


Instruction: You are one of the reviewers, your task is to write a review for the article. You will be given the title of the article, the number of the round in which the article is located, and your order among the reviewers.

Input:
Title: AN ARCHITECTURE FOR NON-LINEAR DISCOVERY OF AGGREGATED MULTIMEDIA DOCUMENT WEB SEARCH RESULTS
Review round: 1
Reviewer: 1

Output:
Basic reporting: The research study described in the article is very well structured and focuses on evaluating the benefits of aggregating "multimedia" information by "verticals". The formal definition of "vertical" (page 7) includes video snippets, news snippets, image snippets, and Web snippets. In this context, I believe that the usage of the term "multimedia" is somewhat misleading. The actual study processes the metadata retrieved from a Web document, which is represented by a Web page, a video, news etc. The metadata of each Web document includes the title, the description, the date, and the URL (a thumbnail if present.) Although this information is retrieved from documents that include multimedia content (e.g. videos, images) the authors use mainly the text-based information related to each Web resource. For this reason, I would suggest to reconsider the usage of the term "multimedia" and maybe substitute it with an expression that represents more the metadata of various Web resources.
Experimental design: The design of the experiment is outlined in great detail. Considering the challenge of the evaluation, the authors always support their choices either providing specific information (e.g. average query length that was used, page 12) or referencing other research studies.
The architecture of the experiment is based on well-known clustering techniques as well as standard software libraries.

The authors chose 2 users from different backgrounds (i.e. education and computer science) to conduct the evaluation of their experiment. I believe that this evaluation should be conducted by more than 2 human experts.

On page 5 the authors mention about “disjoint components” as one of the problems on the Google SERP page. I think that this aspect should be given more attention, especially during the design of the evaluation.  Both the user experience and the user interface play an important role to influence the discovery techniques of the users related to “relevance”. This aspect is partially mentioned in the conclusions.
Validity of the findings: Although the techniques used in the study are not new, their application along with the evaluation process of the study are very interesting.
The value of accuracy (i.e. 99%) in the findings outperforms by far other approaches that are mentioned.

This experiment uses snippets, multimedia documents, and multimedia documents clusters. As shown in table 7, it seems to outperform other related studies when applied to discovery techniques. These findings provide a new, solid baseline for future experiments focusing on information exploration and discovery by search engines users.
Additional comments: No additional comments


Instruction: You are one of the reviewers, your task is to write a review for the article. You will be given the title of the article, the number of the round in which the article is located, and your order among the reviewers.

Input:
Title: AN ARCHITECTURE FOR NON-LINEAR DISCOVERY OF AGGREGATED MULTIMEDIA DOCUMENT WEB SEARCH RESULTS
Review round: 1
Reviewer: 2

Output:
Basic reporting: - The paper is clearly written. However, the paper is a little bit long and wordy.

- The authors provide sufficient literature review and background.

- The structure of paper is well organized, figures look fine. However, texts in tables are not consistent. Several tables have large font size (larger than the font size of content).

- The results support the hypothesis proposed by authors.
Experimental design: - The authors propose a method for aggregating search result by using semantic analysis. In particular, the authors enhance the search engine shortcomings in information exploration and discovery activities by augmenting non-linear information seeking patterns. From the initial search results, the proposed method represents information in the semantically high-dimension space followed by performing clustering, semantic analysis, summarization, then linking information by a graph.

- The authors conduct their experiments on ORCAS dataset. The experiments are measured based on the clustering stability measures and clustering accuracy based on human experts and novices.

- The experimental results have shown the improvement the proposed method compared to prior methods.
Validity of the findings: - The proposed method sound reasonable and interesting. The authors address the realistic, necessary problem in current search engines. The experimental results have shown the performance of the proposed method. Also, the experimental results partly support the hypothesis and address the problem proposed by the authors.

- However, the paper lacks novelty. The method is incremental since the author incorporate prior techniques to be the proposed method.

- The authors claim that they will provide theoretical background to foster exploration and discovery activities. However, these look like explanations rather than the theoretical proofs. The authors should conduct more ablative experiments to prove their claim in the proposed method section.
Additional comments: - The paper should present in a concise form. In the architecture design, the authors should rewrite this section; It is hard to follow this section.

- The mathematical symbol should be edited. For example, the vector or matrix should be bold (use \mathbf).

- The authors should modify tables, the font size should be consistent with the font size of content.


Instruction: You are one of the reviewers, your task is to write a review for the article. You will be given the title of the article, the number of the round in which the article is located, and your order among the reviewers.

Input:
Title: A REVIEW OF MICROSCOPIC ANALYSIS OF BLOOD CELLS FOR DISEASE DETECTION WITH AI PERSPECTIVE
Review round: 1
Reviewer: 1

Output:
Basic reporting: The paper presents a needed survey on a trending topic in the domain of Bioinformatics.
Experimental design: The article is well-designed, However, a few shortcomings are mentioned to the authors.
Validity of the findings: Its a survey paper, a few new references are mentioned to be added.
Additional comments: The paper presents a needed survey on a trending topic. Overall, the paper is fine, however, following issues needs to be addressed.
1.	I could see a few linguistic issues here and there. The authors are advised to have an independent round of proofreading
2.	Availability of datasets is always an issue. The authors must add a section listing the available public datasets for the problem at hand with their URLs.
3.	Fig. 4 is too generic plus it gets blurred with zooming. Kindly revise.
4.	The authors have missed a few recent and important referenced. Pls. consider adding following
4a.	Recent advances in metabolomics in neurological disease, and future perspectives
4b.	Orienting conflicted graph edges using genetic algorithms to discover pathways in protein-protein interaction networks
4c.	Gene encoder: a feature selection technique through unsupervised deep learning-based clustering for large gene expression data
4d.	Optimizing the DNA fragment assembly using metaheuristic-based overlap layout consensus approach
4e.	Quantifying and optimizing visualization: An evolutionary computing-based approach
4f.	On the efficient representation of datasets as graphs to mine maximal frequent itemsets
4g.	Coronavirus disease pandemic (COVID-19): challenges and a global perspective


Instruction: You are one of the reviewers, your task is to write a review for the article. You will be given the title of the article, the number of the round in which the article is located, and your order among the reviewers.

Input:
Title: A REVIEW OF MICROSCOPIC ANALYSIS OF BLOOD CELLS FOR DISEASE DETECTION WITH AI PERSPECTIVE
Review round: 1
Reviewer: 2

Output:
Basic reporting: see below
Experimental design: see below,
Validity of the findings: see below
Additional comments: Deshpande focus on microscopic analysis of blood cells for disease detection using AI. This is an interesting and important paper, several aspect need clarification:
1.	In figure two, please do not use the word agranulocytes, this is not a good way to discriminate white blood cells, please follow the WHO classification, book published 2017.
2. For staining such as Romanowsky stain: please also prefer already in this paragraph that laboratories are used to their respective homebrew technics and therefor, as peripheral blood differential is aphenotypical method, technicians and doctors but also machine learning tools may have problems to translate their experience for other laboratories.
3.	The paragraph starting with line 138 is not clear, what the authors want to say here? Where do they get this information from? Please also mention that normally machines are doing cell counting of peripheral blood and any unknown result by flagging will start  the process of preparing peripheral blood smears. Please also refer to the aspect that (line 146) that there may be  several technics to take the pictures not only "segmentation.
4.	Figure 3 is only an idea what is going on, how ever one should really try to paint the lymphocytes and the monocytes a little bit more according to reality, as showing here they look much too similar.
5.	The sentence line 160 is quite funny, please delete the sentence "it is found to be…" as AML is already killing patients several days after diagnoses, the authors see problems in future areas of interest to devide white and red blood cells on smears, for sure this is a problem, however the reviewer does not see the problem to devide those, but to approach those in different ways with respect to machine learning.
6.	It would be also helpful to understand the expected audience and readers for this paper: bioinformations, haematologics, machine learning experts, technicians in laboratories with less experience or personal …, please speculate for the future perspectives in "conclusions" about the unmet medical needs and what machine learning can contribute for better diagnostics and better treatment and also to guide algorithms that follow the investigation of peripheral blood in the next 5 to 10 years.


Instruction: You are one of the reviewers, your task is to write a review for the article. You will be given the title of the article, the number of the round in which the article is located, and your order among the reviewers.

Input:
Title: A REVIEW OF MICROSCOPIC ANALYSIS OF BLOOD CELLS FOR DISEASE DETECTION WITH AI PERSPECTIVE
Review round: 2
Reviewer: 1

Output:
Basic reporting: Fine
Experimental design: Fine
Validity of the findings: Fine
Additional comments: The authors have addressed my concerns.


Instruction: You are one of the reviewers, your task is to write a review for the article. You will be given the title of the article, the number of the round in which the article is located, and your order among the reviewers.

Input:
Title: SOFT BIGRAM DISTANCE FOR NAMES MATCHING
Review round: 1
Reviewer: 1

Output:
Basic reporting: The aligment of the formulas must be improved.

Soft-bisim is not described
Experimental design: no comment
Validity of the findings: In related work, drug names are used to explain some names-matching measures. However, the Soft-Bisim measure is not used in the experimentation.

The dataset is interesting because it has different languages and domains, but the experimentation and conclusion do not provide more information about it.
Additional comments: No additional comments


Instruction: You are one of the reviewers, your task is to write a review for the article. You will be given the title of the article, the number of the round in which the article is located, and your order among the reviewers.

Input:
Title: SOFT BIGRAM DISTANCE FOR NAMES MATCHING
Review round: 1
Reviewer: 2

Output:
Basic reporting: This paper is exceptionally difficult to understand.

The recurrence relation at 78 differs, for no obvious reason, from the standard one found in most literature, This should be:

Lev(0, 0) = 0.0
Lev(i, 0) = Lev(i-1,0) + w['del'] when i > 0
Lev(0, j). = Lev(0,j-1) + w['ins'] when j > 0
Lev(i, j)  = MIN(Lev(i-1, j-1]  + (w['sub'] if x  != y else 0),
                      Lev(i-1, j) + w['del'],
                      Lev[i ,j-1[ + w['ins'] ) when i > 0 and j > 0

In other words, the base cases of the first cell, the rest of the first column and the first row need to be clearly defined, and the main minimization needs to be defined with precision.

The definition at 86 is similar, and the authors correctly state that the only difference is the addition of an extra path to the minimization, to allow transposition to have a different score from the combination of an insertion and a deletion.

The mentions of Crt (not in the formula, but introduced at 99) are mysterious. Not sure what is meant here. The sentence starting at 90 is hard to understand: I can't make out what the N is in the claim that something is O(N^3) and the reference to an Oracle database is extremely unexpected.

I can't see the relationship between the formulae at 100 and 101 to the much clearer algorithms and presentation in Kondrak 2005 (which I accessed via semantic scholar at

@inproceedings{Kondrak2005NGramSA,
  title={N-Gram Similarity and Distance},
  author={Grzegorz Kondrak},
  booktitle={SPIRE},
  year={2005}
})

In particular T_{i,j}^n is undefined.

The paper needs to provide a more complete and self-contained explanation of what the proposed algorithm is. Even after re-reading the literature references, I am still not sure, and I think other readers will have the same problem.

Examining the code published on Github did not help me.

If asked to guess, what I think is happening is that the authors are adjusting some of the weights in the definition of BI-DIST and combining it with some distance measure.

The language of the paper is mostly professional, but leaves many important details unstated.

The literature references are appropriate. The summaries of what is in the literature are mostly accurate, but again frequently unclear.

The hypotheses of the paper are not clearly stated, because the experiments are not clearly described in the paper. It is possible that literature references may contain details of data preparation and experimental hypotheses, but this should not be necessary. The paper should contain sufficient detail that the reader can understand what the hypotheses are and against what data they are being tested. This material should be comprehensible even before the reader goes to the related literature. As written, the paper is insufficiently self-contained

The formal presentation of algorithms is confusing, with undefined symbols and unfamiliar notation. The authors are urged to go back to the literature that they rely on, then use notations and algorithm presentations that are more clearly similar to those in the prior work.
Experimental design: This is original primary research in line with the goals of the journal.

Experiments are reported at 153 and following. It is not clear from the paper which datasets are used and what metrics are being reported. This should be more self-contained. The authors need to include brief descriptions of the datasets used, the metrics that are used to evaluate performance and the way in which the reports of performance relate to identifiable knowledge gaps. For example, such a description could say "In previous work, a similar algorithm was applied to drug names in English documents, but our new dataset deals with personal names in Urdu. To our knowledge, there have been no previous reports of good performance on this combination of language and task. Our system achieves a balanced F1 of 0.93,"

The investigation is clearly performed to a high ethical standard. The technical quality of the investigation is unclear, because insufficient details are given on the datasets, the partitioning of the data into training, development and test partitions, or of the precise algorithms used.

The unclarity of the paper means that the results will be difficult or impossible to reproduce. Sadly, the source code on github is not much help here, because it is mostly undocumented.
Validity of the findings: The discussion of the findings is relatively good, with an intriguing suggestion for further work at 227

The underlying data has not been presented in an orderly fashion. Robustness, statistical soundness and appropriate controls are difficult to assess. The claims of superiority in 153-218 are too vague, and need to be supported by clearly reported tests of significant difference. The summary statement of improvement at 225-226 is not warranted by any clear evidence in the paper.
Additional comments: Overall, this paper reports work that could be interesting, but the presentation is so unclear that it needs a complete rewrite before it is ready for publication.

I believe that the idea of tuning the weights in a suitable string distance or string similarity function is fundamentally sound, but the suggestion at 63-65 that the work already uses an evolutionary algorithm seems to conflict with the rest of the paper, where what is described is an informal search for a good set of weights. It is fine to say (at 227) that a GA might help, but on its own the comment at 63-65 seems either premature or insufficiently supported by the remainder of the paper.


Instruction: You are one of the reviewers, your task is to write a review for the article. You will be given the title of the article, the number of the round in which the article is located, and your order among the reviewers.

Input:
Title: SOFT BIGRAM DISTANCE FOR NAMES MATCHING
Review round: 2
Reviewer: 1

Output:
Basic reporting: The authors have addressed all the concerns that I raised in my previous review. The paper is now comprehensible and explains what was done and why.
Experimental design: Technically and ethically sound. The impediments to replicability are gone.
Validity of the findings: The results are trustworthy.
Additional comments: No additional comments


Instruction: You are one of the reviewers, your task is to write a review for the article. You will be given the title of the article, the number of the round in which the article is located, and your order among the reviewers.

Input:
Title: UTILITY METRIC FOR UNSUPERVISED FEATURE SELECTION
Review round: 1
Reviewer: 1

Output:
Basic reporting: The writing of the paper requires significant revisions, particularly in technical aspects.
Experimental design: The experiments lack the comparison with state-of-the-art unsupervised feature selection (dimensional reduction) methods, i.e., Graph-based methods, Extended Sammon projection and wavelet kernel extreme learning machine for gait-based legitimate user identification, Proceedings of the 34th ACM/SIGAPP Symposium on Applied Computing, 1216-1219. Etc.
Validity of the findings: It's a good point that the authors have provided the code for readers to prove the claims, which is quite helpful for the community to use the technique for future studies.
Additional comments: The writing of the paper requires significant revisions, particularly in technical aspects. The experiments lack the comparison with state-of-the-art unsupervised feature selection (dimensional reduction) methods, i.e., Graph-based methods, Extended Sammon projection and wavelet kernel extreme learning machine for gait-based legitimate user identification, Proceedings of the 34th ACM/SIGAPP Symposium on Applied Computing, 1216-1219. Etc. It's a good point that the authors have provided the code for readers to prove the claims, which is quite helpful for the community to use the technique for future studies.


Instruction: You are one of the reviewers, your task is to write a review for the article. You will be given the title of the article, the number of the round in which the article is located, and your order among the reviewers.

Input:
Title: UTILITY METRIC FOR UNSUPERVISED FEATURE SELECTION
Review round: 1
Reviewer: 2

Output:
Basic reporting: In this manuscript the authors presented U2FS a novel method for automated feature selection which is important step of dimensional reduction in machine learning. The U2FS is an unsupervised feature selection method which belongs to spectral/sparse learning category. The authors designed new utility metric which is the basis for the U2FS algorithm and is computational efficient. They also explored the use of radial basic function (RBF) kernel and k-nearest neighbours (KNN) algorithms in the first structure extraction learning stage of feature selection.
The main advantages of proposed method are, comparing to other state-of-the art methods, it does not require tuning of the parameters and has less computational time, while its performance is comparable to them in terms of accuracy. Also authors published their method, implemented in Matlab code, as a free publicly available source via github repository.
The article is written in clear and correct English language. The sections Introduction, Related work and Methods provide sufficient and detailed information and relevant literature about the article’s topic. Manuscript has standard structure, and figures and tables are in appropriate format and resolution. Formulas and algorithms of the introduced metrics and methods are provided in details. All results are relevant to hypothesis, with major suggestion that proposed algorithm should be compared to more state-of-the-art methods and some additional evaluation measures could be used.
Experimental design: Research question is well defined and relevant. The methods are described in details with sufficient information to be reproducible. There is a major suggestion about the investigation, that the novel method should be tested with additional evaluation measures and compared to a wider set of state-of-the-art methods. Also authors should emphasize that the analysis included only methods within the same category as the investigated novel U2FS method.
Validity of the findings: The URLs of the datasets used in the research are provided correctly and code for generation of simulation sets and implementation of the method is given on github repository. The minor suggestion is that Matlab code for generation of simulation samples for two sets of Moons and Clouds is missing from the github repository. The experiments are well and in details preformed and conclusions are well stated. However regarding the conclusion about the comparability of novel method with state-of-the-arts in terms of accuracy in the section Results (page 12 and Table 5), my major suggestions are to use additional evaluation measure e.g. error rate, beside the classification accuracy, and to include in the benchmark part more state-of-the-art methods (of the same type and subcategory as U2FS, e.g. NDFS [Li Z, Yang Y, Liu J, Zhou X, Lu H. Unsupervised feature selection using nonnegative spectral analysis. In Twenty-Sixth AAAI Conference on Artificial Intelligence 2012 Jul 14.], DSRMR [Tang C, Liu X, Li M, Wang P, Chen J, Wang L, Li W. Robust unsupervised feature selection via dual self-representation and manifold regularization. Knowledge-Based Systems. 2018 Apr 1;145:109-20.]) beside the RJGSC and MCFS methods to compare U2FS with.
Additional comments: The major suggestion:
1.	The evaluation of the proposed U2FS method by comparing to other state-of-the-art methods in section Results (page 12 and Table 5) should be expended to more methods (of the same type and subcategory as U2FS, e.g. NDFS [Li Z, Yang Y, Liu J, Zhou X, Lu H. Unsupervised feature selection using nonnegative spectral analysis. In Twenty-Sixth AAAI Conference on Artificial Intelligence 2012 Jul 14.], DSRMR [Tang C, Liu X, Li M, Wang P, Chen J, Wang L, Li W. Robust unsupervised feature selection via dual self-representation and manifold regularization. Knowledge-Based Systems. 2018 Apr 1;145:109-20.]) beside the RJGSC and MCFS methods to compare U2FS with.

The minor suggestions:
1.	In the section Results in the evaluation analysis, beside the classification accuracy, additional evaluation measure e.g. error rate could be used.
2.	The URL of the github method could be more explicit given within the manuscript text itself, because it is the valuable result of the work.
3.	The README file on the github could be more informative, e.g. basic description of the U2FS method, requirement list if any, example of basic use of the code, etc.
4.	The data i.e. Matlab code for generation for simulation samples for two sets of Moons and Clouds is missing from the github repository.
5.	In the section Results, in the ‘comparison with state-of-the-art methods’ part (page 12), it should be emphasized in the manuscript that analysis included only methods within the same category as investigated novel U2FS method.


Instruction: You are one of the reviewers, your task is to write a review for the article. You will be given the title of the article, the number of the round in which the article is located, and your order among the reviewers.

Input:
Title: UTILITY METRIC FOR UNSUPERVISED FEATURE SELECTION
Review round: 2
Reviewer: 1

Output:
Basic reporting: N.A
Experimental design: N.A
Validity of the findings: N.A
Additional comments: Thank you for addressing the comments. I have no further comments. Good luck!


Instruction: You are one of the reviewers, your task is to write a review for the article. You will be given the title of the article, the number of the round in which the article is located, and your order among the reviewers.

Input:
Title: UTILITY METRIC FOR UNSUPERVISED FEATURE SELECTION
Review round: 2
Reviewer: 2

Output:
Basic reporting: NO COMMENT
Experimental design: NO COMMENT
Validity of the findings: NO COMMENT
Additional comments: 1.The sample code in the manuscript code will encounter dimension errors. Please check the experiment code again.
2.The comparison method and the data set used can be increased, which will make the article more professional and persuasive.
3.The logic in the manuscript is clear, the grammar is accurate, and the format is relatively standardized.
The formula is best placed in the center to look more tidy.


Instruction: You are one of the reviewers, your task is to write a review for the article. You will be given the title of the article, the number of the round in which the article is located, and your order among the reviewers.

Input:
Title: SUBSET-BASED STEREO CALIBRATION METHOD OPTIMIZING TRIANGULATION ACCURACY
Review round: 1
Reviewer: 1

Output:
Basic reporting: The paper has been well organized. All literature, figures, and tables support the article. Only a typo as inline 333 (in fig Figure12...) is found.
Experimental design: The research has been designed with many dimensions. The experiment covered the hypothesis.
Validity of the findings: no comment
Additional comments: This paper proposed the method to perform the optimized calibration of the stereo vision system.


Instruction: You are one of the reviewers, your task is to write a review for the article. You will be given the title of the article, the number of the round in which the article is located, and your order among the reviewers.

Input:
Title: SUBSET-BASED STEREO CALIBRATION METHOD OPTIMIZING TRIANGULATION ACCURACY
Review round: 1
Reviewer: 2

Output:
Basic reporting: This article is very interested that try to propose the optimization for stereo-camera imaging calibration method along with the OpenCV implementation. The explanation is clear through the article. However, in the literature review, the stereo imaging setups are widely used in 3D reconstruction. Is there any application else that required the stereo imaging setups? If any, please explain to give more reason why we need to develop the new  stereo calibration optimization method.
Experimental design: This article gives a well-defined research question that try to optimize the setup of the stereo imaging calibration. The experimental design and arrangement is well defined.

The suggest is in figure 4 to give this image gain more understanding, please include the 3 steps: 1) The total set of image pairs, 2) identify corners, and 3) store indices of image pairs  that mentioned in line 212-214 along with computational graph for the process of image points preparation.
Validity of the findings: - In the part of "Principle component analysis". Why this article apply only one clustering algorithm that is K-Means? Can we use another clustering algorithm?

- For the experimental result. There is only the result of the proposed method. Is possible to show the result of the comparison of traditional method v.s. the proposed method of stereo calibration optimization.
Additional comments: The figure 1, figure 2, figure 13, and figure 14 are not mentioned in the contexts, please verify.