Instruction: You are one of the reviewers, your task is to write a review for the article. You will be given the title of the article, the number of the round in which the article is located, and your order among the reviewers.

Input:
Title: UNCERTAINTY MODELING PROCESS FOR SEMANTIC TECHNOLOGY
Review round: 1
Reviewer: 2

Output:
Basic reporting: The paper presents a methodology for building an ontology in the presence of uncertainty in the domain. The methodology is applied on a small subset of the Brasilian public procurements domain. The methodology is based on existing software development processes, encompassing requirements collection, analysis and design, implementation and test disciplines, across inception, elaboration, construction and transition phases.

The paper has as goal to fill the gap on modelling ontologies under uncertainty. However, I believe it is not clear in the paper what are the main differences concerning the modelling process when one has or has not an uncertain domain, i.e., besides making use of a language suitable for handling uncertainty, the paper should clearly present the specificities in the overall disciplines and phases that make possible to engineering an ontology representing uncertainty. Otherwise, the proposed methodology is the same as the ones handling traditional ontologies and what changes is only the implementation using a probabilistic ontology language.
Also, along the paper there are lots of citations to the previous work of the authors stating that more details and better explanation can be found there. I believe this harms the paper to be self-contained.
Thus, in a revised paper of the paper,  regarding the overall method, I would like to see:
 - how the uncertainty of the domain makes the phases and disciplines to be different from a standard ontology engineering modeling, as this is the gap that the paper proposes to fulfill; in addition what are the particularities of the UMP-ST process and the POMC compared to the traditional software development process.
- in case it is not possible to highlight how the overall process changes when facing uncertainty, I suggest the authors direct the paper to the development of a probabilistic ontology in the Brazilian procurement fraud domain. But, in this case, it is necessary to point out the contributions of the current paper compared to the previous ones by the same authors.
- the reasons for choosing the PR-OWL instead of others existing in the literature.
- how this paper contributes over the previous ones, and possibly related work concerning probabilistic ontology modeling (even if they are from the same authors)
Experimental design: The case study of the paper is a quite relevant domain, with the potential of helping in the decrease of corruption in Brazil. The UMP-ST is instantiated so that a probabilistic ontology can be built from this domain.
Some explanations about the domain are required, particularly, it should be clear in the paper what are the meaning of procurement, bidders, suppliers, etc, in the Brazilian context.
Following each discipline, please, try to answer the following issues in a revised version of the paper:
 - in the current paper, only a small subset of the requirements is tackled, while the whole set is presented in the previous paper. Please, make clear how this subset was chosen, and how many requirements are there, and how this choice affects the overall process and the final developed probabilistic ontology, providing some examples.
- when defining the entities, its properties and relationships, the UML language is used, and the authors claim that this is because of the popularity of UML. However, I believe that nowadays OWL and its tools and editors are also quite popular among the ontology community. This also brings the necessity of specifying the rules separately, as UML has no support for that. Maybe there was a failed attempt of using OWL editors, but the popularity is not a hard claim for choosing UML, from my point of view.
- it is stated in the paper that everyone is obliged to file his/her TaxInfo, but  this is a bit strong. It should be said that any person relevant to this domain are obliged to do that, not every Brazilian citizen.
-  In page 8, in Implementation section, it is stated that PR-OWL2 is not currently implemented.  At the end of this page, it is said that PR-OWL2 plugin is used. This is a little confusing. I believe this can be clearer if the authors simply explain the limitations of PW-OWL 1 in this section and the modifications that were introduced in order to attend such limitations. Then, PR-OWL2 can be left to the Future work section.
- Resident nodes, input nodes and the difference among them need to be explained in the paper. Actually, I believe the reader would benefit from some explicit background knowledge regarding this language.
- please, explain how the LPDs are obtained, whether they are manually defined or learned from data.
- the role of the expert should be more detailed in the Test section. Is there only one expert? How this choice can threaten the validity of the proof of concept?
Validity of the findings: My main concern related to the general findings of the paper is that the methodology is presented as general, however, this is not proved in the paper, as only one domain is taken into account and nothing is said about the people handling it. Not only the process should be presented but also a discussion about how following the POMC and UMP-ST helped or changed the previous way of modeling the domain. Moreover, it needs to be discussed in the paper the point of view of the experts, i.e., how a number of them would evaluate the proposed process, compared to following a different one, or no one. I believe this is essential to make the contribution of the paper really clear.
Additional comments: No additional comments


Instruction: You are one of the reviewers, your task is to write a review for the article. You will be given the title of the article, the number of the round in which the article is located, and your order among the reviewers.

Input:
Title: UNCERTAINTY MODELING PROCESS FOR SEMANTIC TECHNOLOGY
Review round: 2
Reviewer: 1

Output:
Basic reporting: The paper has a wealth of organization from previous work.  There seems to be some style issues that are difficult. Figure 1 words on left side hard to read. Figure 2 does not work when printed in BW.  There are keywords that need to be coordinated such as UP is not described (but assumed from the heading). The Brazilian translation is not described in the text on page 16 (DIE, CGU) which must be from Portuguese words and not the English.

1) Review section – might be better to italicize a few words in the descriptions as had to circle in detail to capture the organization – that is not possible from a quick read online
Experimental design: Excellent; however

2) The focus of the paper is the steps. This is still confusing. My guess is
Step 1) represent entities…
Step 2) determine uncertainty RV
Step 3) probabilistic distributions
Step 4) dependencies
Step 5) LPD
Step 6) define all LPDS
(Step 5 and 6 seem to be related)
Validity of the findings: REasonable
Additional comments: Grammar
1. Abstract: “is intended to support” implies one case, better “is demonstrated with an example to support”
2. Last line in “probability elucidation” – what is a prototype?
3. “and the UMP-ST is the focus” – of what? “this paper”
4. is it et al. in italics throughout or just when reused (Page 8)
5. Page 9 “requirements tracebility” – should be justified


Instruction: You are one of the reviewers, your task is to write a review for the article. You will be given the title of the article, the number of the round in which the article is located, and your order among the reviewers.

Input:
Title: UNCERTAINTY MODELING PROCESS FOR SEMANTIC TECHNOLOGY
Review round: 2
Reviewer: 2

Output:
Basic reporting: The revised version of the paper is much clearer now, regarding both existing literature and the devised contribution. The main question that I have pointed out before, concerning the design of a probabilistic and deterministic ontology is better explained and exemplified in the proof of the concept.
Experimental design: The proof of concept presented in the paper has been thoroughly extended, with more details, figures, and explanations of the results.
Validity of the findings: The paper brings a proof of the concept that is now clear and useful enough for other people to write and experiment with its own probabilistic ontology, which as stated much clearer now in this revised version, is the main contribution of the paper.
Additional comments: No additional comments


Instruction: You are one of the reviewers, your task is to write a review for the article. You will be given the title of the article, the number of the round in which the article is located, and your order among the reviewers.

Input:
Title: COMPLEXITY CURVE: A GRAPHICAL MEASURE OF DATA COMPLEXITY AND CLASSIFIER PERFORMANCE
Review round: 1
Reviewer: 1

Output:
Basic reporting: This work introduces a new kind of plot to assess the complexity of a given dataset based on probability distribution and Hellinger distance. The resulting plot gives an idea about the complexity of the data w.r.t. the size of the input sample.

- In terms of English, this work contains multiple grammar mistakes. (e.g. Line 19: how it can (be) used, Line 22: treats, line 31: varies, etc.), and the writing needs to be improved, avoiding some awkward sentences.

- The literature covered in this work needs to be enlarged including related works in the application areas that the authors are then using in the experimental validation.

- The structure of the paper and the figures are ok.
Experimental design: Apart from the visualization on multiple UCI datasets,  the authors present two main application of these data complexity curves: data pruning with complexity curves and generalization curves.

In general, I would say that the experimental design is appropriate.
Validity of the findings: To assess the  validity of the findings of this new kind of plot, the authors need to include a comparison with other data complexity measures in the application domains considered.
Additional comments: Apart from including comparison methods, I think that the authors should include discussion (new section) on how this plot can help us when dealing for instance with imbalanced data, high dimensional data or unlabeled data (semi-supervised learning)?


Instruction: You are one of the reviewers, your task is to write a review for the article. You will be given the title of the article, the number of the round in which the article is located, and your order among the reviewers.

Input:
Title: COMPLEXITY CURVE: A GRAPHICAL MEASURE OF DATA COMPLEXITY AND CLASSIFIER PERFORMANCE
Review round: 1
Reviewer: 2

Output:
Basic reporting: Yes, the article is written clear English with commendably few typographical and grammatical errors (please see comments to authors below).

One of the main issues with this paper is related to the introduction. Although the aim is clearly stated, this is not precisely described by clear objectives. This has the effect of making the paper unnecessarily complex to read, which disrupts the flow of argument causing confusion in the methods presented. It would appear that there are five sequential objectives covering the following topics:
•	Measurement of data saturation to determine how representative a sample is of the complete data
•	Measurement of dataset complexity assuming linear separability
•	Determine which technique is likely to suit a dataset for a given task, e.g. classification, regression and so forth.
•	Determine how this measurement can be used for data reduction/halting of online training
•	Test the performance of classification techniques in incomplete datasets by controlling the variance of data through the measurement technique

The introduction has two further issues: firstly, when discussing evaluating classifier perform much focus is on ROC curves, where not all classification approaches have a threshold parameter/value in order to generate such a curve. Secondly, the linearly separable assumption means that it is not an unbiased comparison of various classification algorithms. Comparing an algorithm that makes the linearly separable assumption with one that does not using a measurement that makes the same assumption is biased. In general, the topic of epistasis in datasets should be much more thoroughly addressed. Both toy problem domains, e.g. multiplexer problems, and real world problem domains e.g. bioinformatics, are highly epistatic. Selecting six ad hoc datasets without testing their epistatic contents is insufficient to draw conclusions on the effect of the assumption as the datasets may be linearly separable themselves. The use of toy datasets where epistasis can be measured is recommended.

The background section is not identified as such. There is much recent work on identifying the complexity of instance bases and the understanding of test domains that is missing, e.g. Smith-Miles, K. A., Baatar, D., Wreford, B. and Lewis, R., “Towards Objective Measures of Algorithm Performance across Instance Space”, Computers and Operations Research, vol. 45, pp. 12-24, 2014. It is noted that the literature review/references cease in 2013.

No, the paper does not confirm to the template below, but the problems with the flow of ideas would not be assisted by this broad outline. Numbering of sections, subsections and equations would certainly help the readability of the paper.
Author Cover Page (see above)
Abstract
No more than approx. 500 words (or 3,000 characters).
Headings in structured abstracts should be bold and followed by a period. Each heading should begin a new paragraph. For example:
Background. The background section text goes here. Next line for new section.
Methods. The methods section text goes here.
Results. The results section text goes here.
Discussion. The discussion section text goes here.
Introduction
Materials & Methods
Results
Discussion
Conclusions
Acknowledgements

Unfortunately, this paper was printed in black-and-white, which makes many of the figures time-consuming to interpret, e.g. figure 4. The style adopted in figure 7 and 8 is understandable. However, figure 11 is cluttered where more space can be given to the individual performance graphs. A figure illustrating the Hellinger distance is recommended as it is central to this work.

This paper does represent a coherent piece of work, which has been thoroughly investigated.

It appears that all raw data has been made available, although at this stage of revision the available code has not been verified.

In general, clear definitions and proofs are provided. Inconsistencies are noted in the comments to authors.
Experimental design: The most encouraging part of the paper is the primary research. It is still an open research question of value as to the best method to measure the difficulty of a dataset without applying the actual task to it, e.g. application of multiple classification algorithms to determine the difficulty of a dataset. The hypothesis presented in this work appears sound, although the supporting results could be improved in this paper.

Although the aim is well prescribed, the research objectives and corresponding research questions are not well defined. The complexity curve, conditional complexity curve and generalisation curve be clearly separated in terms of their objectives, otherwise they tend to confuse each other in purpose.

The investigation is rigourous, albeit missing the analysis of the effect of epistasis, which is considered vital to support these measures in real-world domains. The justification analysis of the selected UCI datasets should include what they contain as well as what they do not contain. The accuracy score on the led dataset across the different classification algorithms appears too consistent - it is worth noting that due to the inherent noise in the dataset that 74% is expected and ten fold cross validation means that there are no deviations greater than two decimal places occurring. Curiously, in large datasets a 70:30 split is more usual as much faster to process.

There are no ethical issues in this paper
Validity of the findings: More analysis of the data in terms of redundancy, irrelevancy and epistasis should be conducted. It is often considered that the BCW has irrelevant information, which the results indicate (page 24). It is recommended to cite past work where classification algorithms have identified the considered irrelevant data.

The datasets used are readily available as well as program code (not yet tested).

As there are no clear objectives and complementary research questions, the conclusions are not as clear as they could have been. The contributions to the field need to be more clearly identified and supported.

Any speculation is identified as such.
Additional comments: Major revisions are recommended due to the lack of investigation on the linear separability assumption in regard to epistasis. However, the underlying aim and hypothesis are worthwhile, which will lead to a strong publication.

The background could be strengthened on recent work on determining problem difficulty, e.g Smith-Miles, K. A., and Lopes, L. B., “Measuring Instance Difficulty for Combinatorial Optimization Problems”, Computers and Operations Research, vol. 39, no. 5, pp. 875-889, 2012. Comment on the importance of decision boundaries would be welcome. It is not clear what makes a previous method domain-specific rather than universal.

Please define all matrices, e.g. expected value E

How does the system address noise on the target variable instead of simply on the source variables? Please comment on whether the type of noise have any significant effect?

Please state what figure 1 illustrates rather than what it is and explain why it is not unimodal.

An introduction to the properties section is needed, albeit briefly.

Page twelve: please be consistent or state the difference between ‘q’ and ‘a’ in the probability equations

There appears to be a missed opportunity for additional insight in this paper. The UCI datasets are well studied in classification algorithms, where the past best performance can be cited, e.g. in a column in table 1. This can be compared with the output of the measurement curves to observe any consistencies/disagreements.

Is the speculation that there are no redundant variables in the wine or glass datasets supported by external evidence?

How does the introduced technique perform with datasets containing many outliers?

Minor comments
abstract: ‘our performance measure treats...’
Page: six: ‘training set’
seven: ‘is’ instead of ‘is is’
eight: ‘assess complexity’
ten: ‘for the comparison of algorithms’
sixteen: ‘usefulness of the chosen’
twenty-four: ‘relying heavily on’


Instruction: You are one of the reviewers, your task is to write a review for the article. You will be given the title of the article, the number of the round in which the article is located, and your order among the reviewers.

Input:
Title: COMPLEXITY CURVE: A GRAPHICAL MEASURE OF DATA COMPLEXITY AND CLASSIFIER PERFORMANCE
Review round: 1
Reviewer: 3

Output:
Basic reporting: The structure of the paper is correct, although wording and clarity in the experimental results should be improved.

The authors present a method for assessing data complexity based on probability distributions and Hellinger distance and use it both as a classifier's performance measure and data pruning.

They claim to introduce a universal measure of complexity. What do they mean by universal? A single complexity measure can neither characterize the problem’s difficulty nor fully capture the relationship between the problem's difficulty and the learner’s ability.

The classifiers' accuracy depends both on the constraints of the algorithm and the intrinsic complexities of the data. Ho and Basu (see reference below) designed a set of complexity measures to evaluate the difficulty of the class boundary for classification problems. Ho’s study focused on the analysis of data complexity in order to find the relationship between the intrinsic geometries of data and the learners’ generalization mechanisms. The ultimate goal was to identify learners’ domains of competence so that one could select the most suitable learner for a given problem.

Ho, T. K. and Basu, M. (2002). Complexity measures of supervised classification problems. IEEE Transactions on Pattern Analysis and Machine Intelligence, 24 (3):289–300

I would suggest that the authors thoroughly review the literature on data complexity. Sotoca et al. (2005) is only one review that mainly introduces Ho’s complexity measures.
Experimental design: The authors claim that their measure can be applied not only to classification, regression, and clustering tasks but also to online learning tasks. However, they only provide empirical results on classification problems.
The authors performed experiments on a test bed composed of six data sets from the UCI Machine Learning Repository, which has 255 classification problems. How did the authors select these six data sets?
In addition, they cherry-picked data sets with no missing values, which is another aspect of data complexity. How the proposed measure can be a universal complexity measure if it is not able to cope with this type of data sets?

The authors consider WINE and GLASS data sets complex based on their ratio between the number of attributes and the number of instances. However, this ratio is, again, one dimension of complexity, which is not enough to fully describe the problem’s difficulty.

Sample size is another dimension of complexity. Actually, it is an extrinsic characteristic of a data set and it does not determine the problem complexity as stated in line 133.

Categorical variables should not be treated as continuous attributes, especially when distance metrics are involved.

Regarding the assumption that all attributes are independent, the authors indicate that their assumption “yields good results in practice”. However, the number of attributes of the data sets used in the experimentation ranges from 4 to 21, which is not large enough to demonstrate they can make such an assumption. What would be the impact on bioinformatic data sets usually described by hundreds of attributes?

I agree that there is a threshold where the more data the worse the learner’s performance is. The authors make an interesting point on how the evaluation of machine learning techniques is static both in parametrization and data.
Validity of the findings: Figure 11 shows the complexity curves obtained for different data sets and the performance of a set of learners run on these data sets. We observe that there is no correlation between the complexity measure and the performance obtained by the classifiers. A Hellinger distance does not discriminate between leaners’ performance. For a Hellinger distance of 0.8, the KNN reaches 90% accuracy on the Breast Cancer Wisconsin data set while it poorly performs (52% accuracy) on the Monks-1 data set.

On the other hand, are the learners tuned to work at their best performance for each data set? If the learners are run at their baseline configuration, the resulting comparison would not be fair.

A decision tree algorithm is run on the stripe and chessboard data sets. Which algorithm in particular? C4.5?
Why did the authors choose this combination: stripe/chessboard and decision tree?
Additional comments: I encourage the authors to keep researching on data complexity analysis which is crucial to gain understanding on learners’ behaviors. I would suggest that the authors narrow the scope of their study.


Instruction: You are one of the reviewers, your task is to write a review for the article. You will be given the title of the article, the number of the round in which the article is located, and your order among the reviewers.

Input:
Title: COMPLEXITY CURVE: A GRAPHICAL MEASURE OF DATA COMPLEXITY AND CLASSIFIER PERFORMANCE
Review round: 2
Reviewer: 1

Output:
Basic reporting: The authors have successfully addressed most of my previous comments. IN my opinion this new version of the manuscript is ready to be published.
Experimental design: It has been widely improved.
Validity of the findings: No extra comments.
Additional comments: No extra comments.


Instruction: You are one of the reviewers, your task is to write a review for the article. You will be given the title of the article, the number of the round in which the article is located, and your order among the reviewers.

Input:
Title: COMPLEXITY CURVE: A GRAPHICAL MEASURE OF DATA COMPLEXITY AND CLASSIFIER PERFORMANCE
Review round: 2
Reviewer: 2

Output:
Basic reporting: Please see previous comments. As noted by other reviewers, although the English is understandable it could be improved.
Experimental design: Please see previous comments that have been addressed
Validity of the findings: There is a query on target attribute noise and epistasis, please see below. This is so that readers can understand where the technique is appropriate and where caution is needed in its application.
Additional comments: This paper has been much revised with significant effort by the authors to address the reviewers’ comments. The addition of substantially more datasets, removal of ambiguity in equations and the addition of many insightful comments are commended. However, there are two areas of weakness that need addressing prior to publication.

1. The discussion of noise on the target variable is still not as clear as it could be presented.
Adding “...classes ambiguity (e.g. caused by noise on the target variable). " would help

*** How does the system address noise on the target variable instead of simply on the source variables? Please comment on whether the type of noise have any significant effect? ***
No, our measure disregards this type of noise. It is only concerned with sample sparsity. To make it clear, we added the following to the introduction: " In this article we introduce a new measure of data complexity targeted at sample sparsity, which is mostly associated with variance error component. We aim to measure information saturation of a data set without making any assumptions on the form of relation between dependent variable and the rest of variables, so explicitly disregarding shape of decision boundary and classes ambiguity. "

> Furthermore, there is an implication that noise comes from an overlap between classes, which is an odd implication considering noise as a signal that carries no useful information - some classes naturally overlap and it is useful to understand this information.
“… some irreducible noise. … partly from an overlap between classes.”

2. Epistasis and attribute dependence are not the same concept (although related).
*** In general, the topic of epistasis in datasets should be much more thoroughly addressed. Both toy problem domains, e.g. multiplexer problems, and real world problem domains e.g. bioinformatics, are highly epistatic. ***
We performed an experiment with an artificial data set "chessboard" with strong attribute dependencies (see section section "Complexity curve and the performance of an unbiased model"). In the "Generalisation curves for benchmark data sets" we added references supporting our claim of attribute interdependencies in monks-1 and car data sets. Moreover, we added two new sections ("Comparison with other complexity measures" and "Large p, small n problems") in which we present experimental results on a much larger collection of data sets, among them microarray data important for bioinformatics.
“In the introduced complexity measure we assumed independence of all attributes … Of course, if the independence assumption is broken bias error component may still be substantial.”

> The topic of epistasis is still not properly addressed as attribute dependencies are not the same as epistatic relationships although there is a connection. There are attribute dependencies in both the multiplexer and parity problem domains, but only the former is epistatic. In the latter, all attributes are equally dependent (important). In the former, the importance of an attribute varies with the value of another attribute and they are not all equally important. Frequentist approaches can successfully address problems with attribute dependencies that all equal in importance, but often struggle in epistatic domains, i.e. counting the occurrence of values in individual attributes without regard to epistatic relationships does not function well.
Please test your system on a range of multiplexer problems or explicitly state that the attribute independence assumption makes the approach unlikely to be suitable for highly epistatic problems.

One final thought, which is optional to address: the generalisation curve is claimed to be more insightful than the standard training progression curve, but they were never placed side-by-side for a direct comparison of usefulness. I am curious to know what the set of standard training progression curves would look like for figure 12 and how the new analysis/visualisation method specifically aids understanding/insight.


Instruction: You are one of the reviewers, your task is to write a review for the article. You will be given the title of the article, the number of the round in which the article is located, and your order among the reviewers.

Input:
Title: CLAWPACK: BUILDING AN OPEN SOURCE ECOSYSTEM FOR SOLVING HYPERBOLIC PDES
Review round: 1
Reviewer: 1

Output:
Basic reporting: This article explains the main features and changes to the Clawpack 5.x software and its development practices and tools.

It is generally well written and reads well: good background, relevant citing, good structure etc.
Experimental design: This article is well within the scope of the journal, although I found it hard to find detailed information regarding what exactly is the scope of the journal, other than "articles across the whole of Computer Science".

Note that this is a software paper, describing features of a scientific software library. I view such papers as important contributions to the scientific literature and thus endorse its publication in PeerJ Computer Science.
Validity of the findings: Although this article is generally well written, I would suggest a number of improvements/clarifications:

1. On p. 2: What is q and what is f? And how does f (typically) depend on q (one or two examples). This is clear for most readers but please add one or two sentences to give a gentle introduction to non-specialists.

2. Some of the packages, e.g. PyCLAW on p. 3 and AMRCLAW on p. 4 are mentioned before it is made clear on p. 4-5 that these are separate packages in the collection that make up CLAWPACK. Please clarify earlier.

3. A diagram could be added to explain the relationships between the packages listed on  p. 4-5.

4. Add a comma on line 163 (following "repositories") for increased readability.

5. The documentation repository is hosted on clawpack.github.com whereas the rest is hosted on github.com/clawpack/. This looks confusing to me.

6. Line 198: Should it be "Travis CI", not only "Travis"? Or is Travis the team member responsible for running the regression tests?

7. Top paragraph on p. 10: Somewhat confusing, seems to contrast AMR with MPI ("... on the other hand does not include AMR but uses MPI..."). Please reword.

8. Figures 2 and 3 are small and very difficult to read.

9. Remove FIXME/marginnote on p. 14.
Additional comments: No additional comments


Instruction: You are one of the reviewers, your task is to write a review for the article. You will be given the title of the article, the number of the round in which the article is located, and your order among the reviewers.

Input:
Title: CLAWPACK: BUILDING AN OPEN SOURCE ECOSYSTEM FOR SOLVING HYPERBOLIC PDES
Review round: 1
Reviewer: 2

Output:
Basic reporting: 1 Intro & Backround, Literature references
==========================================

  The introduction is missing a list of related efforts/software
  packages that are comparable to (parts of) ClawPack. This should be
  added.

  Furthermore a section should be added that lists all scientific
  software that is used/needed by ClawPack together with
  references. Currently, only PETSc and matplotlib are cited. According
  to the website at least NumPy is needed/recommend and according to the
  source matlab seems to be also used if it is found. These two should
  be cited.

2 Structure conforms to Peerj Standard
======================================

  On the cover page the corresponding author is not marked. He also
  needs to be accompanied with his full address and email. Some
  affiliations are missing their locations and countries. Please see
  [Basic Manuscript Organization] for more detail.

  Further more the format of the cover page is wrong.E.g. according to
  the [example cover page] the affilliations should be listed directly
  under the list of authors and not in the footer. But it seems like
  this will be fixed by the publishers. No need for the author to change
  this.

  In contrast to the [PeerJ standard] the Acknowledgement section also
  acknowledges funders, too. This information should be moved to a new
  section "Funding statement".

  The in-text citation style is not in the correct format (e.g on the
  cover pages [1] is used instead of Le Veque, 2002). But it seems like
  this will be fixed by publisher. No need for the authors to do it.

  Consequently, the references sections uses the wrong format,
  too. Please see the [reference section in the standard]. While it
  seems like the correct order (Authors with initials, year, title,
  journal) will be adjusted by the publishers, the author are using
  abbreviated journal titles. This needs to be changed to full journal
  titles as requested by the standard.

  The authors should double check all the links in the Refernce section.
  At least the one in reference 6 is broken.

  A lot of the references lack dois. The authors should double check
  whether the publicaction do really not have any DOI associated with
  it. At least reference 3 (authored by at least one of the authors
  themselves) does have a DOI (0.1137/S106482750139738X). DOIs should be
  added wherever they are available.

  [Basic Manuscript Organization]
https://peerj.com/about/author-instructions/#basic-manuscript-organization

  [example cover page]
https://peerj.com/about/author-instructions/#manuscript-checks-modal

  [PeerJ standard]
https://peerj.com/about/author-instructions/#standard-sections

  [reference section in the standard]
https://peerj.com/about/author-instructions/#reference-section

3 Figures
=========

  The text in the pictures of Figure 5 cannot be read on a laptop
  screen. Maybe increasing the size of the text or the pictures
  themselves would fix this. Some of the pictures seem to be or use
  third party pictures. The author should checkshether their license
  allows the distribution under CC-BY. Some of the pictures (e.g. the
  maps in Figure 3 and 5 might need a copyright attribute added. See
  [Figure/Table referencing]

  The referencing style for figures should be used consistently. Figure 1
  is referenced as "Figure 1" while the rest is referenced as "Fig. 2",
  etc.

  [Figure/Table referencing]
https://peerj.com/about/author-instructions/cs#figure-referencing
Experimental design: The paper describes the changes of the scientific software ClawPack
between version 4.x and 5.3, its development history, and the newly
adopted development model. It is a nice software release paper that
are currently becoming more and more popular. I haved checked with the
editors and they are accepting such papers. Therefore I will neglect
reviewing guidelines that cannot be applied to release papers.

Just a few minor comments:

lines 27--28 (Introduction): I think it would be nice to list one/some of the major
changes that will make upgrading to the new user worthwhile or make
ClawPack more unique. (Research question)

lines 563--570 (Conclusion): Similar to above, I am missing a short sentence why
upgrading is worthwhile for a user. The current conclusion only takes
into account the view of developers.
Validity of the findings: I don't think this section applies to software release papers.
Additional comments: lines 49--50:
Reference [34] contains the URL. Therefore the last sentence (lines
49-50) is superluous.

lines 216--223:
In the abstract it says that clawpack has been developed as open
source for 20 years. Here it reads like the software has been made
open source recently as you expect it to  increase the developer
comunity. Please clarify!

line 227:
Citation of PETSc reference missing

line 354:
"GEOCLAW. where" has a superfluos dot,

line 358:
" ..., where some operation ..."
Is it really some (might be different per thread) or the same
operation?

lines 357--362:
 "The main ... patch at a time"
 Application of parallel_for is not clear to me. First it seems like
 parallelization will be over the patches. Then you say that each
 iteration corresponds to a grid level (shouldn't that be a
 patch?). Afterwards you say you assign one patch to each thread. It
 might also be that my understanding of a grid is different from the
 author's. Please clarify, currently the parallelization strategy is
 not clear to me.

line 368:
"Figure 1" -> "Fig. 1" for consistency.

lines 370--371:
40 cores or 20 cores/40 threads? Which exact processor model did you
use?

line 377:
"Note that there are only 2 level 1" -> ""Note that there are only two
level 1"

line 376:
"The efficiency is above 80% until 24 cores, then drops..."

I would have expected it drop for >20 threads as two threads share one
core. Might the preservation of efficiency be due to the ordering of
the patch by size? Do you have an explanation?

lines 377--380:

I don't understand these sentences about level n grids. Does two level
1 grids mean that 2 cells on level 0 got refined? Maybe the
meaning/definition of a leveln grid could/should be stated.

line 382:
Citation of PetSc missing.

Figure 3:

The text in the left two pictures is hardly readable on a laptop
screen. Maybe making the picture larger would help.
If these are 3rdparty pictures, please check whether they can be
distributed via CC-BY Licence. At least for the map a reference is
missing.

Figure 5:

Please double check whether you are allowed to distribute the pictures
under CC-BY License and whether they need a reference to be added.

lines 480-482:

Is the definition of patch here the same as in the description of the
OpenMP parallelization above (lines 357--362)? If this is the case
then the mesh-patch equivalence should be mentioned there.

Page 14:

There is an annotation in the original PDF.h


Instruction: You are one of the reviewers, your task is to write a review for the article. You will be given the title of the article, the number of the round in which the article is located, and your order among the reviewers.

Input:
Title: CLAWPACK: BUILDING AN OPEN SOURCE ECOSYSTEM FOR SOLVING HYPERBOLIC PDES
Review round: 2
Reviewer: 1

Output:
Basic reporting: No comments.
Experimental design: No comments.
Validity of the findings: No comments.
Additional comments: I am happy with the changes made to the revised manuscript and recommend publication.


Instruction: You are one of the reviewers, your task is to write a review for the article. You will be given the title of the article, the number of the round in which the article is located, and your order among the reviewers.

Input:
Title: CLAWPACK: BUILDING AN OPEN SOURCE ECOSYSTEM FOR SOLVING HYPERBOLIC PDES
Review round: 2
Reviewer: 2

Output:
Basic reporting: Everything addressed.
Experimental design: Everything addressed.
Validity of the findings: Everything addressed.
Additional comments: All my comments seem to have been addressed.

In the changes I spotted the following problems that the authors might want to change for the final version:

+ line 394: "...refinement, and in inner loop overall grids
  at..." should be "...refinement, and an inner loop over all grids
  at..." (in->an, overall -> over all)
+ line 396: "... loop construct one thread..." Should be two
  sentences:  "...loop construct. One thread..."
+ line 399-400: "In addition there is a grids are limited to a maximum
  of 32 cells in each dimension, otherwise they are bisected until
  this condition is met." Sentence  does not make sense. Maybe there
  is a word missing.

One additional note:
+ line 407-408: Are you sure that is the CPU model and has 40 cores? I
  could only find  references to E5-2679v3 with 12 cores. But I am not a CPU expert. so do not bother.


Instruction: You are one of the reviewers, your task is to write a review for the article. You will be given the title of the article, the number of the round in which the article is located, and your order among the reviewers.

Input:
Title: INFORMATION THEORETIC ALIGNMENT FREE VARIANT CALLING
Review round: 1
Reviewer: 1

Output:
Basic reporting: No Comments
Experimental design: The article under review is more related to biology than computer science. Thus, I am not sure whether PeerJ Computer Science is the most suitable place for the article.
Validity of the findings: No Comments
Additional comments: The article proposes an alignment and assembly free variant calling method which is based on the context surrounding particular nucleotides. It consists of two steps: variant calling and variant finding. In order to find the length of the context, information theoretic principles are used.

The article is well-written. The new method is compared with previous ones and it seems quite effective. My only concern is that the paper seems to be more relevant for biology and thus, I am not sure whether PeerJ Computer Science is the most suitable place for publication.

I conclude with three (minor) typos the authors might want to correct in a revision:

Line 201: "...is therefore high and we used...";

Line 234: appli -> apply?

Line 272: delete one of the two "avoiding".


Instruction: You are one of the reviewers, your task is to write a review for the article. You will be given the title of the article, the number of the round in which the article is located, and your order among the reviewers.

Input:
Title: INFORMATION THEORETIC ALIGNMENT FREE VARIANT CALLING
Review round: 1
Reviewer: 2

Output:
Basic reporting: Just to fix the main ideas, this paper describes and justifies a method aiming to identify genetic variants inside a species, given a set of genomic sequences. These variants are not restricted to be SNPs only, but can result from INS/DEL modifications as well. Furthermore, and certainly more importantly, this method has been designed to be self contained, i.e. without reference to any a priori known test genome. This problem is therefore of great interest and any certified solution will be of great use.

Basic reporting
The paper is generally well written and easy to read ; the mathematical framework is not too exotic and the references allow the reader to go easily to omitted proofs or developments. One will however be surprised that the authors use systematically “statistic” and not “statistics” as a substantive, whereas it is declared as an adjective in most dictionaries.
As mentioned previously literature is well referenced and relevant.
However the abstract and introduction do not show properly the goals of this study, its major achievements and the domain of validity of the method. This is a real weakness of the paper.
Figure 1 is good, Figure 2 average but Figure 3 is weak and hardly understandable.
Experimental design: Experimental proof is weak. Only three sets of genomes have been study which is clearly not enough for validation.
Validity of the findings: As I said in my intro, the problem is a real one and a good solution will have a real impact. The mathematical background in statistics is neither original nor very hard, and there is no really new nor deep theorem, which does not bother me: the originality can come from having the idea of building evidence from already known features.
My main problem comes from the fact that I do not see a link between the mathematical properties and the claim that this method will indeed find the relevant variants in most situations. The last paragraph of the introduction (lines 52-57) is apparently based on 3 experimental studies with no quantitative argument! It is surprising that the numbers of variants are all of the same orders of magnitude (lines 176-179) and that there is no analysis of a possible biais in the method under certain conditions.
The last paragraph of the introduction in Section 3 (lines 186-189) is also puzzling: any computer scientist should explain why a program is so time consuming…
Finally, I think that the considerations about the accuracy of the predictions should be stated by more rigorous experimental checking. I would guess that the method goes not so badly whenever the solution is close to the center of the gaussian, but that it works poorly elsewhere.
A first check would be to test the accuracy would be to start with the prefix/suffix model on validated data under several different genetic situations. Then I would try INS/DEL situations again under different clustering conditions.

I think that it should then be possible to argue more convincingly on the validity of the method, since a formal mathematical proof is very likely out of scope.
Additional comments: I suggest that the authors strengthen their experimental validation by a more thorough and systematic evaluation of the relevance of their results. Paragraphs such as 2.4, 2.5 and Section 3 should be rewritten. The introduction and the conclusion will be accordingly modified.


Instruction: You are one of the reviewers, your task is to write a review for the article. You will be given the title of the article, the number of the round in which the article is located, and your order among the reviewers.

Input:
Title: ALTERNATING GUIDED IMAGE FILTERING
Review round: 1
Reviewer: 1

Output:
Basic reporting: Lines 77, 78, and 79 have a sentence fragment accidently repeated. Otherwise, no comments.
Experimental design: No comments
Validity of the findings: no comments
Additional comments: The edge-preserving Alternating Guided Filter (AGF) filter introduced in this paper appears to effectively eliminate small scale image details while preserving both the structure and definition of large scale edges and local mean image intensity. In this sense, it is an improvement to the existing RGF filter (that also filters out noise but at the same time smooth's the large scale edges) and the SiR filter (that eliminates small details but significantly changes local image intensity). To the best of my knowledge this filter is unique in the sense that no other currently available filter achieves a performance similar to the AGF filter. The examples provided in the paper clearly demonstrate the effectiveness of the AGF filter (I especially like the 1-D cross sections which clearly show the behavior of the filter near strong and weak edges at different spatial scales). In my view, this filter will be useful for de-noising nighttime images and in the construction of multi-scale image decompositions and therefore ultimately for image fusion applications. An excellent, well thought-out paper.


Instruction: You are one of the reviewers, your task is to write a review for the article. You will be given the title of the article, the number of the round in which the article is located, and your order among the reviewers.

Input:
Title: ALTERNATING GUIDED IMAGE FILTERING
Review round: 1
Reviewer: 2

Output:
Basic reporting: The literature presented is relevant for the paper. However, the related work should be shortened and summarized. The text is very similar to the text on the references, so no long descriptions are needed.  Also, the solution proposed in Section 2.1 regarding the drawback in SiR should be explained later in Section 3 with more details and examples.
A close-up of interesting features, such as sharp edges, must be shown. That will allow to watch and compare the effects of the different filters, such as the presence of halos.
All figures must include a short description and the simulation parameters used for each experiment. That includes the figures in the supplemental material.
Some minor errors:
Line 53 : (i.e., its    ->    (i.e., it
Line 54:  for application in for instance   -> for application, for instance, in
Line 248: “in Section 7”, there is no Section 7
Experimental design: The paper presents a local edge-preserving filter. It addresses the shortcomings of two existing iterative techniques by combining them in each iteration. However, the proposed solution  should be discussed in detail to clarify how it solves the problems on previous methods (RGF and SiR): “the curvature smoothing of large scale edges by RGF and local intensity reduction in combination with the restoration of small scale details near large scale edges by SiR”. I encourage the authors to use a toy example (as presented in Figure 5[1])  to illustrate and validate the proposed approach. In addition, the convergence of the method should be described in the manuscript.

In Section 4, it is said that: “filtering should preferably be performed in the CIE-Lab color space”. Maybe some results using CIE-Lab color space instead of RGB could be presented. It would also be interesting to see a comparison with RGF, SiR and SiRMed  using CIE-Lab color space.

[1] Zhang, Q., Shen, X., Xu, L. & Jia, J. 2014. Rolling guidance filter. In D. Fleet, T. Pajdla, B. Schiele & T. Tuytelaars (Eds.), Computer Vision  ECCV 2014: 13th European Conference, Part III (pp. 815-830). Cham, Switzerland: Springer International Publishing. DOI 10.1007/978-3-319-10578-9_53.
Validity of the findings: My main concern in this paper was the discussion of the results. The results were presented without defining quantitative measures to analyze the effectiveness. As recommended before, a toy example could have helped in validating the approach. As well, whether the method produces halos, or not, should be reported. Also, I would suggest to add some text describing  the computational cost and the running time for methods being compared. Besides, standard bilateral filtering could be included in the experiments.
Additional comments: No additional comments


Instruction: You are one of the reviewers, your task is to write a review for the article. You will be given the title of the article, the number of the round in which the article is located, and your order among the reviewers.

Input:
Title: AUDIO SEGMENTATION USING FLATTENED LOCAL TRIMMED RANGE FOR ECOLOGICAL ACOUSTIC SPACE ANALYSIS
Review round: 1
Reviewer: 1

Output:
Basic reporting: My only "Basic Reporting" criticism given the guidelines is that the data has not been made clearly available to readers, which seems to go against Peer J guidelines.The subsection "Data" in "DATA AND METHODOLOGY" presents no indication of where others can find the data sets used.
Experimental design: No comments
Validity of the findings: No comments
Additional comments: I found this to be a well written and interesting paper where the authors propose a new way to screen for generic "audio events" in acoustic recordings. Sometimes I felt that words were somewhat arbitrarily used, and provide further comments on this below.
While screening “recordings” for “audio events” is an interesting question, and one likely to become more and more important as recordings become more and more common, it is at odds with the perhaps more usual task found in the literature (that I am most familiar with), where one is trying to identify specific signals (e.g. a coqui sound, a bird song, a whale sound) in recordings. Therefore, I believe that the authors must bring forward that distinction and in particular must find a precise way to define what is an “acoustic event”. If that definition is unclear, how can we judge a given algorithm is useful or efficient? One needs to unambiguously define what are the “audio events” (i.e. the signal) one needs to detect in order to quantify true positives and false positives, etc. This must be done from the start, around line 43 or before when the term is first used.
When one reads the abstract sentence “Our goal is to develop an algorithm that is not sensitive to noise, does not need any prior training data and works with any type of audio event.” or the introduction sentence, even more optimistic, “What  is needed is an algorithm that works for any recording, is not targeted to a specific type of audio event, does not need any prior training data, is not sensitive to noise, is fast and requires as little user intervention
as possible., one has to wonder, what is the catch? One usually says there are no free lunches in statistics. So what is the price to pay? And the price you pay is specificity. When you choose to detect “audio events” without clearly defining them a priori, then you actually lose the ability of doing many of the analysis that you might do when you target a specific signal. This is fine if is intended, but that must be explicitly stated so, and eventually discuss clearly the notion that this is a precursor step in other more elaborate and fine-tuned procedures.
This is to me an important point: I thought that all of these discussions regarding the distribution of variables when events where detected is not really possible unless we know the distribution of the variables available for the entire duration of the recordings was available. It would mean very different things to have many events at 18:00 hours if most of the recordings were at 18:00 hours versus if few recordings were at 18:00s! (see e.g. discussions around line 292-294).
More detailed/specific comments that might be used to improve the paper follow below.
Line 10 – 11 – I would say that “thousands of recordings” is wording to be avoided. Recording is a meaningless unit. It should be easy to replace by something that is unambiguous to readers. Same on line 48, “recordings” again used as a unit. This is not a sensible unit. Please check the entire text for the use of the word, as sometimes it makes sense, other not. In particular when you describe the data, again “recordings” is not a useful description. I know you are thinking about Arbimon, but this must be general. How long is a recording say?
Line 46-47 – the “easily draw a boundary around any audio event” is strictly not true, especially at low signal to noise ratios, or when a given frequency band is saturated and so many overlapping events occur continuously for a long time period. Please reword.
While I understand the purpose of the last sentence in the introduction, I believe it includes several bits of information that should belong in the methods. It makes sense to end the discussion with laying the paper that lies ahead, but one should avoid technical details like “2051 manually labelled audio events” or “20 recordings”. One gets confused also because in the text you say this is the workflow of the article, the figure states it’s the workflow of the AED methodology. These can be naturally closely related, but are not one and the same.
Figure 2 legend – to me Yen threshold means nothing, and legends should be self-explanatory. I suggest removing as it’s mostly diversionary here, this is in the text after line 153.
Line 89 – here you refer fig 2C, but fig 2B was not mentioned yet. It would flow bether if the description matched the figure order.
Property A1. – what is tau? And eta? I mean, I know, but rigorously it would be useful to define these.
Line 118 – I believe the notation needs tweaking, as S_db(t,f) has no I which is what it needs to be summed over?
Line 103-109 – I think you cannot extend this to infinity. Would the algorithm not break apart if an entire recording had a given frequency band saturated? Discuss, or change wording, please.
Line 124 – “valuation”?
Line 132 – Surely this is over a given time frame in practice? What is the time frame considered? Will you discuss the sensitivity to different  time frames considered?
Line 133 There’s something missing as like an “at least” before “1-\rho(f) proportion”, right?
Lines 141-142, I find this confusing… would it not be clearer if you delete just ”by estimating it as” and replace by “:”
Just before line 146 – he “r>0” is this in time, frequency, both?
Line 146 – Equations should read just as text. Therefore, the “Where” must be “where” and not indented, and there’s no dot after the previous equation. Same in line 201. Check remaining instances.
Line 148 – reword the arbitrary “the estimator should have a small response”. What are you referring to? Whan is an estimator response?
Line 155 – I understand what you mean, but clarify “image values”, since there’s strictly no images here.
Line 156 – “Th” should be “T”?
Line 157 – the contiguous here refers to both time and frequency?
Line 168 – not sure why you need the descriptor “the sites dataset”? The first data set was also collected on sites…
Line 175 – as I said, here’s a good example. You set to find “events” here. What is an event? Is this a circular definition to some extent, since events are sounds you are able to classify as events??? In particular, how these relate e.g. to the 3 types of sound you menion in the first paragraph of the introduction?

Line 179 – 21 by 21 and alpha=5. These are fundamental details. Are these values optimal in any way? Why? What is the sensitivity of the method to changing them? What are recommendations for users?
Line 188 and 191 (2 times) – I think the word “automatically” needs to be added for clarity, e.g. “were detected over the total count” is “were automatically detected over the total count”
Line 187-192 – I’d introduce the wording true and false positives here, and note explicitly this is a typical confusion 2 by 2 matrix but one of the cells is absent (i.e. there are no true negatives).
Line after 200 – you need a more rigorous wording. “To measure the degree of separation of each variable on the audio event density”… there is no such characteristic as “variable separation”… what do you mean exactly?
Line 201 i-th should be i^{th} (superscript)
Line 203 – “The 2-variable marginal distributions” reword to bivariate?
Line 205 – why this one chosen? Just as an example? If so say so.
Line 208 – words missing or plurar vs singular mistake
Table 1 legend – last sentence – you say “arbitrary number of false negative examples”. Don’t you mean “true negatives”?
Figure 3 legend – need to label the columns, since each column is a different type of image, right?
Figure 4,5,6 – you need at the very least to state explicitly that dark is less and white is more, but a scale would be helpful!
Figure 6 – plot y_max by tod. What is the reason for the weird vertical bands? It’s hard for me to imagine what would create these tod “discontinuities”?
Line 228 – so does “cov”, in fact with a larger H value than _max. Why do you not mention it
Line 236-237 – explanation? Is this a feature or an artefact? If you raised the question you can’t leave it unanswered.
Line 243 – seems inconsistent to me to do this “separation” (and I do not like the word as I said above) only visually in 2d but with an H statistic in 1d, why this choice?
Line 247 and figure 7 – 6? Which? I can’t see this. Please mark them all in figure 7.
Figure 7 legend – “Close-up on the first area”… why the first.
Line 251 and several others – all latin names must be italicized, including in references.
Line 262-264 – Discuss problems with intense chorus on a given noise band.
Line 281 –use of word recordings here is inconsistent (cf. with say line 168-171), you say 6, these are presumably what you referred as dataset?
Line 282 – remove the two instances of the word “any”, not useful.. but this needs added clarification, not sure what is meant here?
Lines 300-303 – I’d like to see comments regarding whether these have also been missed sometimes or not?
Line 334 – “University Press”
Lin e 344- incomplete ref?
Line 347 – “Conference on…”- incomplete
Line 355 – “pages”???
Line 366 – species name needs italics


Instruction: You are one of the reviewers, your task is to write a review for the article. You will be given the title of the article, the number of the round in which the article is located, and your order among the reviewers.

Input:
Title: AUDIO SEGMENTATION USING FLATTENED LOCAL TRIMMED RANGE FOR ECOLOGICAL ACOUSTIC SPACE ANALYSIS
Review round: 1
Reviewer: 2

Output:
Basic reporting: The English language is good.
The mathematical description has not helped the presentation. It took time to understand and was not adding much more than could have been said quickly in words.  In particular, symbols are used which are not defined - e.g. eta, n, tau. The numeral '1' is used for the indicator function but it is not stated as an indicator function and is doubly confusing because bold or blackboard-bold font is not used.

T in the equation just above line 154 is not defined. I had to go on-line to see what it was. It is the threshold which is later referred using the letter 'Th". All this suggests that not a lot of care was put into proof reading.

The images are just of sufficient resolution to support the text.

Figures 4, 5 and 6 are puzzling in that the images at top and bottom of the left column have different time scales. Also it appears from the top lect image that more events happen at night thanin the day. The reader needs a lot more help to understand these figures.
Experimental design: The method for subtracting base-line value is valid although it is based on an important assumption. The signal model is not that different from an additive noise model. The authors state that they do not make an assumption about the distribution of the noise, symbol epsilon in text, and this is a nice feature. (Although later the use of 5 percentile tails to establish cutoffs implies that something like gaussian noise is assumed).

The nicest feature of the method is that used to calculate the Range estimator and the use of entropic correlation. I am not aware of this being done elsewhere and is the main interesting result of the paper.

The authors report their accuracy based on overlap of observed rectangle with predicted rectangle. This criterion is far too liberal because even a slight overlap can lead to a correct prediction for the wrong reasons. I would suggest that at least a 50% overlap is required which is indeed the case in some of their images.
Validity of the findings: The authors imply that there are few thresholds or critical parameters that must be tuned for their system. However the use of the 21x21 window size is surely important and must have been determined by trial and error. If an event was too large in area, this window would leave "holes" in the event due to the way they calculate the range estimator. In general I felt the authors were too uncritical of their method. The important assumption of
rho(f) < 0.5 is hidden in extensive and not very helpful mathematics. It is a reasonable assumption - other methods have to make similar assumptions but the authors claim some superiority for their method.
Additional comments: This could be a nice paper. The method of using the Range estimator is very nice and something I am sure others will emulate when the paper is published. However the paper is inadequate in three respects:
1) The authors have been too uncritical in promoting the advanages of their method.
2) The estimates of accuracy are based on a very easy success criterion.
3) There is no comparison with another method. I totally agree that a fixed threshold technique is not useful but there are better techniques to compare their method with.


Instruction: You are one of the reviewers, your task is to write a review for the article. You will be given the title of the article, the number of the round in which the article is located, and your order among the reviewers.

Input:
Title: ENRICHING SCIENTIFIC PUBLICATIONS WITH INTERACTIVE 3D PDF: AN INTEGRATED TOOLBOX FOR CREATING READY-TO-PUBLISH FIGURES
Review round: 1
Reviewer: 1

Output:
Basic reporting: Article generally well written. I suggest table 2 could become main text and some of the reasons behind those desirable features discussed/justified.
Experimental design: Experimental design not relevant.
Validity of the findings: Agree with the author that 3D pdf has been a difficult to author and is thus an impediment to uptake. The problem is that 3D PDF is not new and things have not significantly improved over the last 5 years. I struggle to agree that the proposed solution will significantly improve the situation.

To the question of 3D PDF being a suitable format. The author says “… images and multimedia elements like audio, movies or 3D models”. Lots of 3D data structures are not supported and are key for visualisation, e.g.: volumetric. But even for geometry, basic primates like spheres and cylinders are not defined except as very inefficient “triangle soup” mesh representations.

The author says in reference to model sizes “In most cases, this size could (and should) be reduced significantly, because the density of polygon meshes does usually not need to be very high for illustrative purposes.” If one can zoom into models then why might a high resolution mesh not be desirable? Data is becoming increasingly detailed and high resolution so I do not accept that model sizes are necessarily low. I would further argue that simple 3D models are more likely to be fully conveyed in 2D, it is exactly the complicated models that require a interactive navigable experience to assist with understanding. It should also be noted that the 3D PDF does not support what one might be considered standard features like progressive mesh rendering or even multiple levels of detail.

With respect to the import format list it is indeed impressive, but sorry but I do not accept these formats in their full specification are supported. To test I wasn't unduly negative I tried to import a textured 3D mesh as an OBJ file and it failed, didn't have time to identify why but suspect it was due to multiple large texture files. It was a very standard OBJ file that I can open in every other software package I have tried.
Additional comments: I don't wish to sound too harsh, I like many were initially intrigued by the possibilities 3D PDF may offer. The issues I have with 3D PDF are
1. Lack of viewer support outside one companies product.
2. Lack of diversity of representation within the format.
3. Lack of tools for creating 3D PDF.
There seems to be a lack of support for any pdf viewer other than Adobe product. Where is the “export to pdf” in 3D modelling/editing packages? No support from the leading work processing solution. These tell me that widespread support from the industry is not growing.

See two attachments, "PoorShading" shows the coarse model supplied as an example, aren't vertex normals supported? "CorruptDrawing", Adobe Acrobat viewer didn't seem to handle OS level screen zooming. Not a complaint of this paper but further evidence of an immaturity of support. I also struggled to find support on mobile devices.

Unfortunately the article has not allayed my concerns. The solution presented is semi-commercial and a 4+GB download, seems like overkill just to create 3D PDFs.

I might agree with the aims of this paper say 5 years ago, as a proposed future opportunity to publish 3D data with documents, but today it would seem the promise as a suitable medium has not been realised. Similar to the push for VRML in the late 80’s, the reality has been quite different. There are many reasons for this, one is the complexity and diversity needed for 3D data representation requirements, another key factor is the lack of cross platform and software support. In summary, I don’t believe the case has been made that 3D PDF has a future for scientific data visualisation and that it supports the type of 3D data representations researchers require.

Final comment, if the author resubmits then I strongly suggest creating an example that makes a strong case.


Instruction: You are one of the reviewers, your task is to write a review for the article. You will be given the title of the article, the number of the round in which the article is located, and your order among the reviewers.

Input:
Title: ENRICHING SCIENTIFIC PUBLICATIONS WITH INTERACTIVE 3D PDF: AN INTEGRATED TOOLBOX FOR CREATING READY-TO-PUBLISH FIGURES
Review round: 1
Reviewer: 2

Output:
Basic reporting: The paper is well written and merits publication in PeerJ.
Experimental design: The work reported in this manuscript seems to have been carried out with care.
Validity of the findings: I think that the software toolbox for generating 3D PDF documents could become a very useful format in scientific publications.
Additional comments: It would be useful to comment on how this toolbox could be used to visualize 3D  structures of molecules and proteins.


Instruction: You are one of the reviewers, your task is to write a review for the article. You will be given the title of the article, the number of the round in which the article is located, and your order among the reviewers.

Input:
Title: ENRICHING SCIENTIFIC PUBLICATIONS WITH INTERACTIVE 3D PDF: AN INTEGRATED TOOLBOX FOR CREATING READY-TO-PUBLISH FIGURES
Review round: 1
Reviewer: 3

Output:
Basic reporting: Comments (PeerJ) 9236-v0
Title: Enriching scientific publications with interactive 3D PDF figures: A complete toolbox

Author:  Axel Newe

This manuscript presents a developed tool box for 3D-pdf file presentation of scientific visualization through the Adobe software.  Development of such interactive 3D visualization with easy access is useful for readers to gain a thorough understanding of the problems presented. It is particularly useful for medicinal and biological applications such as protein docking etc. From this point of view, this article addresses a useful and interesting issue.

However, I do not believe that this manuscript is publishable at its present form as the 3D-pdf based on in complete literature review and therefore, this manuscript missed significant milestone work in this direction, which may shake the foundation of this study and make it redundant.

First of all, the manuscript missed the significant development in interactive 3d-pdf based on the Adobe software/tool box in 2009, in which the authors detailed the development of the 3D-pdf to show the 3D structures of drugs using embedded 3d-pdf technique they developed.

1.	Lalitah Selvam, Vladislav Vasilyev and Feng Wang, Methylation of zebularine: a quantum mechanical study incorporating interactive 3D PDF graphs, Journal of Physical Chemistry B 113, 11496-11504(2009).
2.	L. Selvam, F. F. Chen and F. Wang, Methylation of zebularine investigated using density functional theory calculations, Journal of Computational Chemistry, 32(10)(2011)2077–2083.
3.	A. P. Wickrama Arachchilage, F. Wang, V. Feyer, O. Plekan, and K. C. Prince, “Photoelectron spectra and structures of three cyclic dipeptides: PhePhe, TyrPro and HisGly”, Journal of Chemical Physics, 136, 124301 (2012) (8 pages).

The JCP article was in fact selected as the cover page for its 3D structure of the issue. None of the articles are referred in this manuscript. As the author missed these articles using very similar technique, unless such related articles are properly referred and the technique of the manuscript is justified, I do not think this article of a very similar technique is worthy publishing.
Experimental design: NA
Validity of the findings: NA
Additional comments: The author need a thorough literature review and developing something existing is extremely time consuming.


Instruction: You are one of the reviewers, your task is to write a review for the article. You will be given the title of the article, the number of the round in which the article is located, and your order among the reviewers.

Input:
Title: ENRICHING SCIENTIFIC PUBLICATIONS WITH INTERACTIVE 3D PDF: AN INTEGRATED TOOLBOX FOR CREATING READY-TO-PUBLISH FIGURES
Review round: 1
Reviewer: 4

Output:
Basic reporting: English is good.
Introduction and background are well written and clearly position the paper in the relevant field.
Figure are relevant, looks good.
Experimental design: This is a paper about software.
I am not sure how it fits to the requirements.
However I presume the design of the program is OK - I have no way to assess it.
Validity of the findings: The example in the Supplementary materials works really well on my computer: Dell notebook about 4 years old.
I think it demonstrates the validity of the project.
Additional comments: I did not install the program and my review is really deficient due to this.
There are two reasons why I did not install the program:
1.
The download page for MeVisLab
http://www.mevislab.de/download/

indicates the size of the installation
MeVisLabSDK2.8_vc12-64.exe (1154 MB)

This is more than 1 Gigabytes.
It is really too large for me.

2.
I was not sure that the above download is what I need.
The download page talks about Microsoft compilers.
Do I need to compile the installation with them?
The size if the installation is too large to just try it.


Instruction: You are one of the reviewers, your task is to write a review for the article. You will be given the title of the article, the number of the round in which the article is located, and your order among the reviewers.

Input:
Title: ENRICHING SCIENTIFIC PUBLICATIONS WITH INTERACTIVE 3D PDF: AN INTEGRATED TOOLBOX FOR CREATING READY-TO-PUBLISH FIGURES
Review round: 2
Reviewer: 1

Output:
Basic reporting: Meets the publication standards with respect to level of English/grammar. Summary of the field and references.
Experimental design: This follows feedback in the first review.

The input format list. The author admits that not all features of all formats are supported. I accept that, it is notoriously problematic. I suggest that if a format is not fully supported then it should not be in the supported list. I would rather see a short list (even if it is just 1) of formats that are known for certain to work irrespective of what features of the format employed. Less is more.

For example I installed MeVisLab 4.8 for Mac, then installed 3D PDF support from here (3rd April)
https://zenodo.org/record/48758
Opened an obj file in the app but it appeared without the associated textures. The revised  paper acknowledges that textures are not supported by the tool, this in my opinion is a fatal limitation, noting that it is not a limitation of u3d which does support at least diffuse and transparency textures.
Validity of the findings: This follows feedback in the first review.

The title refers to “a complete toolbox”, I think the author agrees that the proposed solution is not a complete toolbox. The types of data visualisation that is supported by the intersecting feature set of this tool, u3D and 3D PDF is quite narrow.
Additional comments: The author has addressed most of the feedback from the first review in the revised document.

I agree that my view of the software tool discussed in the paper is clouded by my opinion that 3D PDF is not a suitable tool for the interactive representation of data within a paper. I accept that some journals accept it but in my opinion it is not the future. I would freely point out this possible bias to the editors. In my comments here I have tried to only address those points relevant to the proposal in the paper, not to the suitability or otherwise of 3D PDF as a means of representation for data visualisation.

Regarding file size, I agree that 1GB may not be a significant download these days, but I still maintain that the solution is a big hammer to crack a small nut. Note I made a mistake in my earlier review, it is not the download that is 4GB but the final program (actually 5.3GB) on a Mac. That is, it is a lot to download, install and learn for what to me should be a simple task of creating a 3D PDF. In deed the app itself is all that is needed, not clear why it is based upon MeVisLab. I note that 2 of the 3 other reviewers, and perhaps all 3, admit they didn’t try the software. I suggest the hurdle was too high.

The author has provided a slightly more interesting model as an example. There is only one viewer, from Adobe. See supplied image that shows shading errors with the example provided. Is that from the viewer or software proposed in the paper? If one is using such a tool to present scientific data it should be a huge concern that there are these representation errors.


Instruction: You are one of the reviewers, your task is to write a review for the article. You will be given the title of the article, the number of the round in which the article is located, and your order among the reviewers.

Input:
Title: ENRICHING SCIENTIFIC PUBLICATIONS WITH INTERACTIVE 3D PDF: AN INTEGRATED TOOLBOX FOR CREATING READY-TO-PUBLISH FIGURES
Review round: 2
Reviewer: 2

Output:
Basic reporting: same as for the previous  version
Experimental design: same as for the previous  version
Validity of the findings: same as for the previous  version
Additional comments: The author has modified the manuscript to make the "pure user" life easier - it is appreciated.


Instruction: You are one of the reviewers, your task is to write a review for the article. You will be given the title of the article, the number of the round in which the article is located, and your order among the reviewers.

Input:
Title: ONCOLNC: LINKING TCGA SURVIVAL DATA TO MRNAS, MIRNAS, AND LNCRNAS
Review round: 1
Reviewer: 1

Output:
Basic reporting: No Comments
Experimental design: No Comments
Validity of the findings: No Comments
Additional comments: This work developed a web tool, named lncOnco, which systematically and comprehensively calculate the correlation between mRNA, lncRNA, and miRNA expression and survival rate in cancers from TCGA. This is a valuable tool for researchers lacking programming knowledge to access the TCGA data and look for the genes associated with prognosis cross all cancer types. The manuscript is generally well written and clearly presented. I have just a few suggestions on this work:
1.	Table S1-S3 should be a part of this tool. Please add these information into system and design an interface for users to view, access, and query.
2.	I can’t find any link or button back to main page or search page. It is not convenient if I would like to start a new search.
3.	It is unable to search using synonymous gene names (e.g. TP53 and P53). I think this function is required because some synonyms are more familiar to biologists.


Instruction: You are one of the reviewers, your task is to write a review for the article. You will be given the title of the article, the number of the round in which the article is located, and your order among the reviewers.

Input:
Title: ONCOLNC: LINKING TCGA SURVIVAL DATA TO MRNAS, MIRNAS, AND LNCRNAS
Review round: 1
Reviewer: 2

Output:
Basic reporting: No Comments
Experimental design: No Comments
Validity of the findings: There is no evaluation of the survival-associated biomolecules the OncoLnc identified. The author should at least enumerate several examples to discuss the importance of the identified survival-associated biomolecules in cancer(s). This is useful to convince the users that the biomolecules identified by OncoLnc are reliable.
Additional comments: 1. So far, OncoLnc only provided continuous data for cox model. However, some biomolecules affected cancer(s) only when their expression level exceeded a certain cut-off. The author may consider to add one function which allows the user to use the categorical input for cox model. For example, users could choose the median (50%) of the expression profile of the studied biomolecule to run the cox-model.

2. The author must add more information to describe the method applied to OncoLnc. For example, the concept of cox model. In other words, a well-described manual is required.

3. The author should add a return bottom in the result page. Or, they should put the query function in the result page.

4. In the result page, the sorting function for each column.

5. When a user input an item which doesn’t perfectly matched any item in OncoLnc, the OncoLuc should list some similar items to the user.


Instruction: You are one of the reviewers, your task is to write a review for the article. You will be given the title of the article, the number of the round in which the article is located, and your order among the reviewers.

Input:
Title: ONCOLNC: LINKING TCGA SURVIVAL DATA TO MRNAS, MIRNAS, AND LNCRNAS
Review round: 1
Reviewer: 3

Output:
Basic reporting: The background and introduction of this article are not sufficient to clarify the motivation for  OncoLnc development. The author hopes OncoLnc can facilitate users to survey the survival correlation with mRNA, miRNA and LncRNA by using TCGA clinical data. However the importance, relevance and previous studies of three kinds of data in survival study were not mentioned and compared.
Experimental design: Many details were not described or unclear in method, especially for statistical analysis. First, the survival analysis is the most important part in this study. The author did not mention how to select the variables used in each data type. Second, in survival analysis, not all patients have the clinical parameters which used in cox regression model. The way of dealing with this kind of data should be detailed. Thrid, the datasets used in this study need more details, including original sources and appropriate citations.
Validity of the findings: Three kinds of datasets, mRNA, miRNA and LncRNA, were used in this study. However, their importances were not addressed, as well as the motivation for using these datasets. In addition, the author performed survival analysis for each tumor type and each data type. Different kinds of data were pieced together into OncLnc. It would be better if OncLnc can provide the analysis or pre-calculated results to integrate the different data types and tumor types. It can give the researchers an insight into biomark selection survival analysis.
Additional comments: I suggest that the motivation and background should be improved. The website can value-added more functions, such as custmized analysis, instead of pre-computed resluts.


Instruction: You are one of the reviewers, your task is to write a review for the article. You will be given the title of the article, the number of the round in which the article is located, and your order among the reviewers.

Input:
Title: ONCOLNC: LINKING TCGA SURVIVAL DATA TO MRNAS, MIRNAS, AND LNCRNAS
Review round: 2
Reviewer: 1

Output:
Basic reporting: No Comments
Experimental design: No Comments
Validity of the findings: No Comments
Additional comments: 1.	In the response letter, author mentioned they will provide links for downloading Table S1-S3 in OncoLnc website. However, I can’t find it in OncoLnc.
2.	Synonymous gene name search is the basic function for bioinformatics web tools. Please add this feature in OncoLnc.


Instruction: You are one of the reviewers, your task is to write a review for the article. You will be given the title of the article, the number of the round in which the article is located, and your order among the reviewers.

Input:
Title: ONCOLNC: LINKING TCGA SURVIVAL DATA TO MRNAS, MIRNAS, AND LNCRNAS
Review round: 2
Reviewer: 2

Output:
Basic reporting: No Comments
Experimental design: No Comments
Validity of the findings: Minor Comment:
The author has given some examples in his rebuttal letter. I suggested the author to add these examples in the main text to support OncoLnc.
Additional comments: Yes, the users should know what they would like to query. However, in biology research field, there are lots of alias for genes and other biomolecules. A function to list the possible alias for the queried item is definitely not useless. The author has reminded the users what kind of ID they should use. This function would be helpful. However, the author made this information as superscript. If the author could put this information in the query box, it will be easier to be noticed by the users.


Instruction: You are one of the reviewers, your task is to write a review for the article. You will be given the title of the article, the number of the round in which the article is located, and your order among the reviewers.

Input:
Title: ONCOLNC: LINKING TCGA SURVIVAL DATA TO MRNAS, MIRNAS, AND LNCRNAS
Review round: 2
Reviewer: 3

Output:
Basic reporting: No Comments
Experimental design: No Comments
Validity of the findings: No Comments
Additional comments: lncOnco provides the results of precomputed survival analyses. The author also curated the ID and gene symbol of mRNA which is a great and hard work. Researchers can use IncOnco to explore the gene of interest. However, I think IncOnco is a bit too rough.
The author claims OncoLnc is the only data portal that allows for simple download of the clinical data and expression data. I assume that the target users of IncOnco are the researchers who are lacking the programming skill. Therefore, the website should add more functions for them to explore the survival correlation data of interest. For example, the information of all supplementary tables can be searched and downloaded from website. Providing the customized download, such as the specific cancer of interest, also would be useful for those researchers to perform other analyses further, especially for lncRNA.


Instruction: You are one of the reviewers, your task is to write a review for the article. You will be given the title of the article, the number of the round in which the article is located, and your order among the reviewers.

Input:
Title: ONCOLNC: LINKING TCGA SURVIVAL DATA TO MRNAS, MIRNAS, AND LNCRNAS
Review round: 3
Reviewer: 1

Output:
Basic reporting: No comments
Experimental design: No comments
Validity of the findings: No comments
Additional comments: I don't have further questions.


Instruction: You are one of the reviewers, your task is to write a review for the article. You will be given the title of the article, the number of the round in which the article is located, and your order among the reviewers.

Input:
Title: ONCOLNC: LINKING TCGA SURVIVAL DATA TO MRNAS, MIRNAS, AND LNCRNAS
Review round: 3
Reviewer: 2

Output:
Basic reporting: No Comments
Experimental design: No Comments
Validity of the findings: No Comments
Additional comments: No Comments


Instruction: You are one of the reviewers, your task is to write a review for the article. You will be given the title of the article, the number of the round in which the article is located, and your order among the reviewers.

Input:
Title: ONCOLNC: LINKING TCGA SURVIVAL DATA TO MRNAS, MIRNAS, AND LNCRNAS
Review round: 3
Reviewer: 3

Output:
Basic reporting: No Comments
Experimental design: No Comments
Validity of the findings: No Comments
Additional comments: The author only add the download page. The author do not provide any further customized or analysis functions.
I still think IncOnco is a bit too rough.


Instruction: You are one of the reviewers, your task is to write a review for the article. You will be given the title of the article, the number of the round in which the article is located, and your order among the reviewers.

Input:
Title: SMART BRIX—A CONTINUOUS EVOLUTION FRAMEWORK FOR CONTAINER APPLICATION DEPLOYMENTS
Review round: 1
Reviewer: 1

Output:
Basic reporting: The paper is well written and provide the insides of novel achievements in terms of containers management.
The background is up to date and the motivation presented in the introduction is reflecting a good knowledge of the current problems in virtualization techniques.
The structure is conform to the natural templates for computing science papers, including the state-of-the-art criticism, the presentation of the proposal, the references to the proof-of-concept implementation as well as the test results. The figures are relevant for the proposed methodology.
The text is self-containing and can be an example of good practice in writing a paper in the field of distributed systems.
Experimental design: The research questions can be identified in the first section of the paper and are related to real problems of the current Cloud computing environments.
The methods that are exposed are presented to a level that can be reproduced in similar conditions.
Validity of the findings: The conclusions of the work are relevant for the addressed community.
Valid ideas for the future work are also correctly identified.
Additional comments: The paper is well written and presents a valuable contribution to the state-of-the-art in a hot topic.
The related work can be improved with the latest achievements in the what concerns the large scale experiments using container technologies.
The scale of the experiments is quite small - the effect of increasing the number of instances should be studied too.


Instruction: You are one of the reviewers, your task is to write a review for the article. You will be given the title of the article, the number of the round in which the article is located, and your order among the reviewers.

Input:
Title: SMART BRIX—A CONTINUOUS EVOLUTION FRAMEWORK FOR CONTAINER APPLICATION DEPLOYMENTS
Review round: 1
Reviewer: 2

Output:
Basic reporting: This paper presents the smart brix framework, which offers the ability to check sets of containers against vulnerability breaches and other specific requirements. It also provides mechanisms for mitigating the identified issues by evolving the containers.

The paper is well written and easy to follow, the overall objective and challenge are well defined.
Experimental design: I believe the description of the technical objectives and challenges could be improved. Moreover, smart brix is presented as a framework for the continuous evolution of container-based deployment but I felt that the runtime monitoring aspect was not properly presented in the paper or at least could be more detailed.

The overall architecture of the framework is well presented but I would suggest to the authors to also provide a description on how (technically and methodology) the framework should/could be used in a production setting (maybe using the case study).  Also, few technical details are provided on how can be used the framework and how easy it is to implement and extend such a  “continuous evolution” system.

Whilst reading the formula describing how can be calculated the overall confidence value, I was wondering if some more complex options have been considered by the authors. If yes, I would suggest discussing this.
Also, regarding the confidence adaptation model escalation, it would be interesting to evaluate and discuss what is the current limit to the automation, how often it goes to the human interaction level and to provide an example of what is/can be provided to the human to ease its job.

To improve the readability of Section 3.2, I would suggest to illustrate the content presented in the paragraph starting by « If no issues … » with a Figure.

The proposed approach seems to be quite tight to the package managers; I would suggest the author to clarify this.
Validity of the findings: Regarding the experiments, it is being said that the experiment was repeated 3 times maybe the authors could justify this choice and also provide the standard deviation. These experiments are interesting and it is good that the authors have tested their approach on large sets of images. Maybe It could also be interesting to perform tests on very specific images and to identify the impact of the “software stack” they embed both in term of size and diversity.

I would also suggest to the authors to provide a link to their data and source code (if open source).
Additional comments: No additional comments


Instruction: You are one of the reviewers, your task is to write a review for the article. You will be given the title of the article, the number of the round in which the article is located, and your order among the reviewers.

Input:
Title: SMART BRIX—A CONTINUOUS EVOLUTION FRAMEWORK FOR CONTAINER APPLICATION DEPLOYMENTS
Review round: 1
Reviewer: 3

Output:
Basic reporting: The paper is generally well-written, clear, and well-argued.
There are only a few places where I think this is not the case, and I have the following suggestions:
- Evaluation criteria are discussed for the first time in Section 4. I think that is too late. I strongly suggest the authors add upfront a short overview about the research goals the Smart Brix framework is supposed to achieve, how they intend to measure those achievements and the basics of their evaluation method and criteria. This provides a better scope for the paper, makes the reading more focused, and facilitates assessing the values of the approach.
- Section 3.2 explains how the Smart Brix Manager works in collaboration with the rest of the framework. The text is dense with description of all the interactions, and it is easy to get lost in them. I would suggest a Figure with a corresponding diagram (e.g. a UML Sequence Diagram) to accompany the text as a visual aid.
- In Section 5 "Related Work", lines 452-454 there is a rather long list of references on works that "propose a novel method for analyzing cloud-based services for certain types of vulnerabilities". Citing in numbers without differentiating the works in any way from your own work, or among one another, is not considered good form. Also, some of these "cloud-based" papers are not from the Cloud world (or epoch!) including one dated 2000. Please revise this part.
- The main contributions and take-away claimed by the paper remain somewhat implicit and are not summarized anywhere that I could see in a concise way.
- It looks like an "Acknowledgements" section may be missing, as per the PeerJ guidelines.
Experimental design: As said above, there is a certain lack of upfront clarity on what the research goals are, and how they are intended to be measured. That means that the paper does not lead with a clear definition of its research questions.
Although that does not hamper the understanding of the approach and method, in Sections 1 to 3,  and its potential value,  the reader remains with an unanswered question on how the benefits of SmartBrix can be assessed until Section 4 (page 9)
The method and procedure for evaluation in Section 4 make sense and showcase some of the benefits of Smart Brix, in particular its efficiency and performance. However, as far as I can see they do not shed light on one of the most interesting and potentially valuable characteristics touted in the paper, i.e., the pipeline self-assembly capabilities, which should enable to automatically compose complex workflows for Analysis and Compensation out of a potantially large set of candidate micro-services. Since the overhead of this kind of automated composition typically grows super-linearly with the number of candidate components,  discussing the number of components in the Analysis and Compensation sets vs. the related complexity and costs (e.g. time, or chance of not synthesizing a suitable or correct pipeline) of the self-assembly approach would be quite interesting, and would speak to a different kind of scalability of the approach.
A more detailed analysis of success vs. failure within the described experiments (e.g. false positive and false negatives for the Analysis pipelines, or correctly completed vs. failed compensation attempts) seems also like a missing, but important, aspect of evaluation
Validity of the findings: This paper basically discusses the feasibility and efficiency of the Smart Brix approach towards the managed evolution of Cloud container-based deployments. The paper does quite a good job and is sound in that respect. The experiments are intended for the purpose and seem well designed and simple.
The kind of evaluation described though is partial and a bit shallow, as I indicated above, and begs the question of  a fuller assessment of the benefits of the Small Brix approach. I provided above some suggestions on how to expand and strengthen the evaluation and better showcase the work.
I also think that a "Threats to validity" and/or "Limits of applicability" section would be appropriate as part of the post-evaluation discussion.
Additional comments: I believe the work has value and that the paper makes a good case for it. The evaluation can be much more convincing if strengthened. My guesstimate is that you do not need to do very substantial extra work for that strengthening. I am assuming this amounts to minor revisions.