Instruction: You are one of the reviewers, your task is to write a review for the article. You will be given the title of the article, the number of the round in which the article is located, and your order among the reviewers.

Input:
Title: SAMPLE-LEVEL SOUND SYNTHESIS WITH RECURRENT NEURAL NETWORKS AND CONCEPTORS
Review round: 1
Reviewer: 3

Output:
Basic reporting: The paper is well written overall, and has a good structure. The writing is sometimes a bit wordy or convoluted, however, which reduces clarity a bit.

The article provides a good amount of figures and supplementary material.

The research question is not posited very clearly in the article. While it is clear that the overall goal is to construct a sound synthesis system, it is not made clear how it should differ from previously existing approaches, why such a novel system is needed, and which gap in literature is addressed in the process.

Some figures do not flow with the text layout-wise, ending up alone on a page with blank space around them.

While most figures are informative, figure 3 provides no information and can be removed. Some figures are also a bit unfinished, including figure 22, which needs an x-axis labelling, figure 8 having a gray instead of a clear background and no explanation of the meaning of the red dots in the caption, and figure S2 whose values should be rounded to avoid numbers such as 0.7000000001.

For more details, see the general comments section.
Experimental design: The experiments appear novel and interesting, and they give an indication of the potential of the method.

However, I believe they are also far too limited in scope. Please refer to the general comments section for more detail.
Validity of the findings: While the results for the individual sounds used in each experiment appear to be derived properly, the lack of evaluation metrics and the improper scope of the experiments severely limit the possible conclusions. Please refer to the general comments for more details.
Additional comments: Firstly, I would like to mention that I am not familiar enough with reservoir computing and conceptors to fully assess the proposed approach, particularly with respect to the model’s design.

However, my main criticism is related to the evaluation of the approach, and the conclusions that are drawn and can be drawn from the findings.

In general, there are many hyper-parameters (shown in the tables for each experiment), and it is not clear how these were determined. Furthermore, they are set differently for each experiment, and almost every experiment contains of only a single sound. Adapting the hyper-parameters as well as the model parameters all to a single sound each time creates a serious risk of overfitting, which is not addressed in the paper. Also, it appears that parameters need to be adjusted for each experiment as the model is quite unstable, and even further tweaks to how the approach works are often implemented (experiment 2, experiment 3) to make the method work at all. It also seems that the windowing method to create sequence chunks does not work all the time, and needs to be hand-tuned to the given sound. All of this combined raises serious doubts about how generally applicable the method is. This concern is not reflected in the paper. In contrast, it is even stated that their method (“CCRNN”) “can reproduce arbitrary waveforms for granular inspired sound synthesis.” (L462).

Therefore, I would suggest to evaluate the reconstruction ability on a dataset of multiple sounds, without manually optimising parameters to each sound individually, to show generality, or to phrase the claims in the paper in a much less general way.

In terms of results, we further find that there are no experiments to obtain perceptual ratings of any kind for any of the outputs of the model. While an objective NRMSE measure is provided for the reconstruction experiments, for the synthesis/manipulation experiments there is a severe lack of quantiative evaluation. The few quantitative results are also hard to interpret since there are no baseline models being evaluated to compare to.

Generally, the evaluation relies too much on single examples. This applies to experiments 1, 2, 3. Experiment 4 gives the NRMSE for a particular pitch of a generated square wave, but does not plot or average all NRMSE’s for a whole range of pitches. Experiment 5 generates a single sound only. While these exapmles illustrate the working of the method very well, it is not possible to draw sufficiently broad conclusions from them.

For the reconstruction experiments, it should be made clear how much the input sound can be compressed by the model, since perfect reconstruction is trivial if the RNN memory is sufficiently large. This is a serious concern in the current experiments – for example in experiment 1, a 900-node RNN is trained on 100 15-sample signals, along with a conceptor for each segment, suggesting that there could be a very low amount of compression required. Preferably, a compression ratio should be given along with the reconstruction error that shows the relative reduction in the number of bits achieved.

For the experiments involving synthesising new sounds, there is no evaluation besides a subjective interpretation of the author in most cases, including “Extending sound synthesis with conceptor logic”, “Sound morphing with interpolated conceptors”, and experiment 5. It is also stated that pitch shifting can be achieved with the method, but no comparison with other pitch shifting methods is performed. The method is stated to extend granular sound synthesis, but no granular sound synthesis method is used for comparison. While it is repeatedly stated that the significant variety in the model outputs can be interesting for musicians to create new sounds, this subjective claim is not empirically validated in any way. For this set of experiments, some metric(s) of success should be established, such as how interesting the sounds are perceived to be by participants in listening experiments, or the perceived quality of pitch-shifting.

CLARITY ISSUES:

The following are issues with understanding the paper’s content, which probably result from unclear writing. However, in case I am missing something here, this can be discarded.

In the basic model, x_target(n+1) is computed from x(n), and then x(n+1) from x_target(n) and x_target(n+1). With conceptors, this is apparently changed to computing x(n+1) directly from x(n) through the conceptor C (see eq. 10, 11). However, in eq. 13 the interpolation equation originally used in the base model description is seemingly modified, which however is not used since it was replaced by the conceptor transition function. What is behind this apparent mismatch?

Logical operations on conceptor matrices are executed, however it seems that the conceptor matrix arises from an SVD and thus can have real values, and is not binary. How do these logical operations then work?

MINOR REMARKS

L303 How do you precisely limit scale_a?

L292 This chapter seems out of place or at least needs an introduction, since we are moving away from simple reconstruction towards synthesising new sounds – why is that apparently a subsection of experiment 1 still?

L239 Why only the first 100? The whole signal has more than that

Model introduction (“Basic models”) structure could be improved, instead of putting all equations at once upfront, and then explaining all variables in a long block of text, explain each equation step by step and introduce variables along the way

“Loading patterns into the model” chapter is confusing since it is not clear what the actual optimisation objective is, and what is being optimised (is W* and W equal to W as introduced in Eq (1)?) and what parameters are optimised over (only W, or also W_out?).

L195 “Error and similarity” section: It should be made more clear that this is the main evaluation metric used in the experiments, and how it will be used (namely, compare reconstructions to original input)

L201 “Conceptular synthesis” - the first paragraph directly introduces a novel concept, but only justifies it much later. Considering the model description preceding this section, it would be better to have a section explaining why the above proposed model is not directly applicable, and then motivate conceptular synthesis as a solution

Variable names and subscripts are sometimes full words, this detracts from readability and should be shortened

L336 “mu” is chosen as variable for morphing factor although it is already used for the chunk size parameter previously (set to 15 in exp. 1)

L124 “input signal vector a(n) (size 1)” - “size 1” is confusing here, since the vector should have length proportional to the signal duration. Also in L135 suddenly a gets another index j to denote multiple input vectors, which it did not have before!

 “sparsely connected with probability of 10.0/N” - what happens when N < 10?
L145 – It is not clear at this point whether time-steps start at 0, or 1

L139-142 states we have to optimize W and W_in so that the given equation (L141) holds, but equations 4 and 5 deal with optimising W and W_out instead – why the discrepancy?

“to field of sound synthesis” → “to the field of sound synthesis”

L236 it might be more understandable to define “a” more precisely by showing its minimum and maximum index. Also it is confusing to call it “set” but use matrix/vector notation where inputs are ordered

L389 “presents” -> “present”


Instruction: You are one of the reviewers, your task is to write a review for the article. You will be given the title of the article, the number of the round in which the article is located, and your order among the reviewers.

Input:
Title: SAMPLE-LEVEL SOUND SYNTHESIS WITH RECURRENT NEURAL NETWORKS AND CONCEPTORS
Review round: 2
Reviewer: 1

Output:
Basic reporting: The writing can still be further improved.
Experimental design: My concerns were addressed.
Validity of the findings: The results are not very strong but my concerns were addressed.
Additional comments: No additional comments


Instruction: You are one of the reviewers, your task is to write a review for the article. You will be given the title of the article, the number of the round in which the article is located, and your order among the reviewers.

Input:
Title: SAMPLE-LEVEL SOUND SYNTHESIS WITH RECURRENT NEURAL NETWORKS AND CONCEPTORS
Review round: 2
Reviewer: 2

Output:
Basic reporting: This revision has addresses issues in my previous review.
Experimental design: This revision has addresses issues in my previous review.
Validity of the findings: This revision has addresses issues in my previous review.
Additional comments: Minor points:
There is still an extra nesting of parens in equation 2.

143  Win is generated using linear random values between −1 and 1, and scaled using input scaling factor gamma^ input
 - why would you need to scale random numbers that have just been generated to be in [-1,1]?

146 b is a vector of y biases, which are generated from a linear random distribution between −1 and 1,
147 and scaled with bias scaling factor gamma^ bias
 - why would you need to scale random numbers that have just been generated to be in [-1,1]?

There is still an extra nesting of parents in equation 14:  ((1−a)x(n))

There is still an extra nesting of parents in equation 15.

I think it would be helpful to have a link to audio for each example for which you have an image.


Instruction: You are one of the reviewers, your task is to write a review for the article. You will be given the title of the article, the number of the round in which the article is located, and your order among the reviewers.

Input:
Title: SAMPLE-LEVEL SOUND SYNTHESIS WITH RECURRENT NEURAL NETWORKS AND CONCEPTORS
Review round: 2
Reviewer: 3

Output:
Basic reporting: The writing style is clear and professional, and noticeably improved compared to the first submission as many of the reviewer's suggestions have been taken into account.

Some passages are still a bit long-winded and could be improved by shortening them without losing in content, such as the discussion of related work in the beginning of the "Conceptular synthesis" section.

Figures have been much improved, as well as most of the equations.

The related work section remains well written and gives a good overview of recent works.

Minor issues are noted in the "general remarks" section.
Experimental design: The experimental design has been improved substantially after the first round of review and now features a proper baseline method for comparison, where applicable. Hyper-parameters are now more clearly explained and also handled in a more uniform fashion.

For the evaluation, an MFCC error to measure reconstruction fidelity was introduced, which is a good choice as it should result in a more perceptually relevant metric than the previously used RMSE, which was also affected by phase differences.

A major concern in experiments involving the reconstruction of sounds is still the risk of overfitting the models to the input sound. The authors argue, implicitly by making reconstruction error the evaluation metric, that the model should be able to reproduce the input audio exactly, but do not take into account model complexity at the same time.
In that case however, a simple look-up table that saves the whole audio sample would be the optimal "model" with perfect reconstruction error, but not very useful for any other task such as manipulating the audio in a semantically meaningful way, as it has not built a compressed representation of the input that can be manipulated to make changes to the audio.
Therefore, ideally a cross-validation by testing the trained models on separate audio data should be performed, or the model complexity itself should be included as another metric as it should be minimised alongside the reconstruction error, following the Bayesian paradigm of model selection that also factors in the prior probability of each model.
Authors find that CCRNNs with more hidden nodes tend to perform better, which could also be explained by this overfitting phenomenon as the model can more easily memorise the input data.

On the other hand, if the goal is not simply sound reconstruction and generation in the first place, but rather meaningful sound manipulation, ideally evaluation metrics and experimental setups should be changed to measure how well musicians use these model to manipulate sounds.
At the same time, I acknowledge the difficulty in evaluating in such a manner, so I suggest at the minimum to clearly report the complexity of all models used or performing cross-validation as discussed above, along with inserting remarks that discuss overfitting as a potential issue.
Validity of the findings: The findings are well articulated and accurately reported overall, limitations of the method are discussed sufficiently.

One exception to this is their argument that the chaotic and unstable nature of their (and related non-linear) methods is somehow inherently a benefit from a musical perspective as it provides a surprise element. This is not only elaborated in a too long-winded fashion, especially in the paragraph starting in L681, but should be marked more clearly as speculative, or alternatively, references should be given that show a preference of musicians for such novelty when using these tools.

In terms of the validity of the reported results, the main conceptular synthesis experiment has been much improved and therefore the resulting conclusions are quite robust.

However, the conceptillator experiments still rely quite heavily on single examples, which makes the proof of the generality of their application a bit less convincing.
Particularly, the pitched square wave oscillator is only evaluated quantitatively in terms of reconstruction error with a single pitch, and only qualitatively across a pitch range. Here it would be more convincing if reconstruction loss is measured for a range of different pitches, and also the reconstruction loss when adapting a model trained with a particular pitch to output a different one. For example, with a model trained on 65Hz, and required to produce a 110Hz output, record the minimum reconstruction error achieved with the best setting of the leak rate parameter, and repeating this process for a variety of pitch combinations would then give a quantiative measure of how well pitch can be controlled via the leak rate parameter.
Additional comments: MINOR REMARKS

It should be explained what z in Equation (1) is supposed to be below the equation

Equation (2) has no introductory explanation to it

L158 onwards – it is a bit unclear what set of weights is optimised – W* or W („reservoir weights W* are adapted“ then later „W is optimised“). Is W* a separate set of weights from W? It is stated that „the reservoir with weights W* is driven … using equations 1 and 2“, but equations 1 and 2 contain W and not W*.

L158 onwards – there is no mention of how the input weight matrix Win is handled. Is it just randomly initialised at the beginning of optimisation? Is it kept throughout the whole procedure?

L175 „The driving signals from steps from timesteps“ grammatical error, also mention which driving signals by using the mathematical notation, e.g. is that the input a?

Equation 11 is of the form a = b * a. While I understand the intended meaning behind the equation, it would be better to state beforehand that the result of Equation (2) is renamed to x* or similar for the following equation, and then use x* instead of on the RHS.

„Conceptular synthesis“ section – it could be clarified more clearly in the beginning that this is a concept introduced in this paper and makes use of the above introduced methods, as there is a lack of transition from the previous methods section. Also the discussion of related work could be somewhat shortened by transferring it to the related work section where applicable

L283/284 – There should be a bit more explanation on why the presence of high-frequency information can be a problem for the training process apart from just the „distortion“ it. One would naively except the generation system to be able to synthesise waveforms with non-zero starting amplitudes as well?

L312 „was training to resynthesise a kick drum“ → „was trained“

L346 matrix dimensions of X have no proper whitespace

L374 „The use of non-deterministic weight values“ - unclear wording – is this meant to be „non-deterministically chosen initial weight values“?

L387 „models at a higher resolution (ψ = 900)“ does 900 not reflect model size, not (audio) resolution?

Equation (17) is lacking an introductory statement

L396 Results could feature the means/medians of the MFCC errors for both models to have a clearer comparison in addition to the violin plot

Figure 10 caption: „linearly from 0 – 2“ → „linearly from 0 to 2“

Figures 14 and 15 are in the conclusion section when they should appear a section earlier


Instruction: You are one of the reviewers, your task is to write a review for the article. You will be given the title of the article, the number of the round in which the article is located, and your order among the reviewers.

Input:
Title: TAKING ADVANTAGE OF THE SOFTWARE PRODUCT LINE PARADIGM TO GENERATE CUSTOMIZED USER INTERFACES FOR DECISION-MAKING PROCESSES: A CASE STUDY ON UNIVERSITY EMPLOYABILITY
Review round: 1
Reviewer: 1

Output:
Basic reporting: The article is generally well written, with good spelling and grammar. There examples of terms that I myself would probably not use e.g. l388 "can be also consulted at". I would just word it as "can be viewed at".

There are a few works that I believe are very relevant that were missing. These include:

1) Apel, S., Kastner, C. and Lengauer, C. (2009). Featurehouse: Language-independent,
automated software composition, Proceedings of the 31st International Conference on Software Engineering, ICSE ’09, IEEE Computer Society, Washington,
DC, USA, pp. 221–231.

2) Kramer D., Oussena, S., Komisarczuk, P., Clark, T. (2013) Document-Oriented
GUIs in Dynamic Software Product Lines. In the Proceedings of the 12th International Conference on Generative Programming: Concepts and Experiences.

3) Kramer D., Oussena S. (2017) User Interfaces and Dynamic Software Product Lines. In: Sottet JS., García Frey A., Vanderdonckt J. (eds) Human Centered Software Product Lines. Human–Computer Interaction Series. Springer

4) Pleuss, A., Hauptmann, B., Dhungana, D. and Botterweck, G. (2012). User interface engineering for software product lines: the dilemma between automation
and usability, Proceedings of the 4th ACM SIGCHI symposium on Engineering
interactive computing systems, EICS ’12, ACM, New York, NY, USA, pp. 25–34.

5) Pleuss, A., Hauptmann, B., Keunecke, M. and Botterweck, G. (2012). A case study
on variability in user interfaces, Proceedings of the 16th International Software
Product Line Conference - Volume 1, SPLC ’12, ACM, New York, NY, USA, pp. 6–
10.

Structure of article is generally fine. I am unsure why code samples are given as images. Data shared in github repository. Results seem speculative. There seems to be no test or analysis if a SPL approach is actually beneficial to the papers context.
Experimental design: The context in which the experiment is being undertaken is original, as is the approach. As there are a number of works missing (as mentioned above), it is difficult to follow why the SPL approach was applied. There are opensource tools e.g. FeatureHouse that allow for fine grained software composition. The generator created for this experiment appears to be specific for this SPL, and can not be used for another SPL. Tools like FeatureHouse are generic, and can be applied to many different types of SPLs, with many different languages. There is no discussion of this. Many of related work included reads like an acknowledgement, however a lack of real motivation as to why the GUI approaches used are not sufficient to their use case.

The research question for me is not well defined. On the one hand, they argue for the need for customisation to support different user needs, but also argue on the need for improving productivity, maintainability, and traceability. The needs for supporting customisation in this context is interesting, however I see no evaluation of this. Were the different product variants of use to the users? Did the students find they picked the right courses for them?
It does not feel like that research questions were answered. There is some discussion on some sample variants produced by the SPL, but no discussion on the real benefits of the SPL in the first place.
Validity of the findings: The results given are regarding specific variants produced from the SPL, without answering the questions motivated by this paper. Findings appear to be qualitative by the authors, not from users, developers (that many use the SPL), or quantitative following existing approaches.

It is state on line 476-478 argue the SPL has "proved its usefulness to reduce the development time regarding the exploration of the observatory's data". How was this tested? Where are the results to support that claim?
Additional comments: You need to really explain more concretely your research problem, and then conduct experimentation to help answer those questions. I honestly struggle to see how the results you show answer the questions I have interpreted from your paper. How has using an SPL really helped you? Why is your approach better/novel to related works? How did the users evaluate your variants? You results only seem to be talking about some of the configurations, but appear to not help answer your research questions.


Instruction: You are one of the reviewers, your task is to write a review for the article. You will be given the title of the article, the number of the round in which the article is located, and your order among the reviewers.

Input:
Title: TAKING ADVANTAGE OF THE SOFTWARE PRODUCT LINE PARADIGM TO GENERATE CUSTOMIZED USER INTERFACES FOR DECISION-MAKING PROCESSES: A CASE STUDY ON UNIVERSITY EMPLOYABILITY
Review round: 1
Reviewer: 2

Output:
Basic reporting: The background section contains a good range of related work, but there is no proper comparison with the work proposed. The related works are reported, without any relation to the proposal. This would benefit the text, in making clear what are the main contributions of this work compared to previous ones, where are previous works falling short. It is not clear if dashboards are configured statically, that is, in a compile-time fashion. Couldn’t works related to dynamic software product lines be useful on this context, to provide dynamic configuration and customisation for the dashboards?

- When discussing explanatory and exploratory visualisations (lines 235-243), it would be good to provide concrete examples for illustration.

- When discussing the meta-model (line 286), this is general to any dashboard, not only for those related to employability or those who are customisable, correct?

- Repeated use of “proved” in the text, when it should be used ‘illustrated’, ‘demonstrated’, ‘shown’, or some other term.

- In the beginning of the context section, line 205: “…also known as OEEU, it’s Spanish…” -> “…also known as OEEU, its Spanish…”

- What does “complete and quality information” mean in the introduction? (line 78)

- Line 230: Does the text actually mean ‘recollected’ or ‘collected’ would suffice?

- Line 272: …abstract point of view… —> abstract modelling?

- Line 400: ‘sets of data’ -> ‘datasets’

- Some of the references are incomplete, without edition and pages for journal papers, for instance. Moreover, there are works mentioned in the text, such as Gabillon et al. (2015) and Pleuss et al. (2012), which are not listed on the references section
Experimental design: Nonetheless, the text in its current version does not clearly present the research question. The motivation properly addresses the issue of a range of definitions for employability, with different perspectives. Nonetheless, something that is not clear in the current version of the text is the fact that customisable dashboards could make that even more confusing and diverse. Wouldn’t that make it hard to have a common, shared definition for the concept?

I understand that for space constraints, the feature model is not provided, but I believe that at least the repository should have a version of it available. Another thing that would benefit and help on motivating the tool support developed on this work, would be to calculate the actual number of products that can be generated from the selection of features from the model. This could be done with the aid of tools such as FAMA, FAMILIAR, or FeatureIDE, and would help on establishing the need for having a systematic and automated process and avoiding manual labour into customising the dashboards.

Another issue is related to the particular notation for feature models that was used. On Figure 3, for instance, there is a feature named Size? Would that entail an attribute for the size, or does it make sense as a simple boolean feature that might be optionally present or not? Another question would be the ‘categories’ feature? Would it be necessary to list the categories for a specific dataset, or does it suffice to enable/disable it as a whole?

It seems to me that the DSL shown in the work is an embedding of the feature model in XML. If that’s the case, this should be made explicit in the text. Also, why not using a proper DSL that could then generate the underlying XML files as a result? In terms of traceability of the DSL with the feature model, how is that maintained? In the github repository provided as raw data, there is only the schema, and no file, even in image form, depicting the feature model. In the end, do we actually need the feature model in this approach? Or are DSL files being automatically generated from the feature model?

Figure 9 shows an instance of the code generation process, it would be better to first present something similar as Figure 10, but with proper identified roles (see my other comment below on who’s responsible for performing the customisation), and then illustrating it with the concrete example from Fig. 9. I also believe that less detail could be provided, so text could be made bigger and more readable.

Another issue is related to the actual process of configuration and customisation. Who is in charge of configuring the dashboard, the actual user or someone prior to the end-user of the system?

Did the work need to perform any changes to Jinja, or could it be used as-is?
Validity of the findings: The discussion section starts by stating that the SPL paradigm has ‘proved’ (see comment about this term) the usefulness to reduce the development time. While an automated process can surely help on reducing the time for development, currently there is no evidence presented in the text to support this. What was the time needed for developing the DSL and whole infrastructure for generating products? Usually, in the literature, there is a return on investment for the product line engineering approach starting from three (3) products. Was that observed from the case study?

I believe that the discussion section and results should focus on effort (even if estimated) related to this approach vs. a manual one. Even proxy metrics, such as lines of code could be used for comparison. Another issue would be to try considering evaluating expressiveness? Are there any limitations on the current approach which prevent a particular customisation to be performed? The text mentions this by the end of the discussion section, but I believe that an immediate follow-up of this work should focus on end-users evaluating the usability of the generated dashboards to confirm whether those are actually usable for analysing the data.
Additional comments: This article illustrates the application of a software product line engineering approach to the development of information dashboards geared towards analysing data regarding university employability. Through the use of XML-based DSLs, a range of customisable dashboards can be automatically generated. The text provides some examples of custom configurations and discusses the main issues related to applying the SPL approach.

The work brings the application of software product lines to the domain of generating information dashboards, with a focus on customisable user interfaces, for exploring large datasets. However, there are a number of issues in the current version of the text, which I detail in the three (3) dedicated areas.


Instruction: You are one of the reviewers, your task is to write a review for the article. You will be given the title of the article, the number of the round in which the article is located, and your order among the reviewers.

Input:
Title: TAKING ADVANTAGE OF THE SOFTWARE PRODUCT LINE PARADIGM TO GENERATE CUSTOMIZED USER INTERFACES FOR DECISION-MAKING PROCESSES: A CASE STUDY ON UNIVERSITY EMPLOYABILITY
Review round: 2
Reviewer: 1

Output:
Basic reporting: I appreciate the changes performed to the manuscript as requested by the reviewers.

- Line 119: Viability - feasibility

- Line 643: “to prove…” - “to show…”

- Line 644: “The approach need…” - “The approach needs…”
Experimental design: Even though there have been further explanations, I would still suggest better discussing the integration of feature diagrams and the DSL, besides justifying why not using a proper DSL that could then generate the underlying XML files as a result.

Regardless of what is currently implemented in the approach, does it conceptually need a feature model, or is it just a convenience for having a readable document? That’s related to what I had asked in the prior revision, so I would expect something along these lines in the text.
Validity of the findings: The “development time improvement” section is a good addition. I would only suggest putting a stronger emphasis in the fact that even though there is effort to setup the SPL, this might be paid off by easily generating customisable dashboards.

In the final discussion, I would also suggest highlighting what is unique in the domain of complex tools, such as dashboards, that has been tackled by the approach.
Additional comments: This article illustrates the application of a software product line engineering approach to the development of information dashboards geared towards analysing data regarding university employability. Through the use of XML-based DSLs, a range of customisable dashboards can be automatically generated. The text provides some examples of custom configurations and discusses the main issues related to applying the SPL approach.


Instruction: You are one of the reviewers, your task is to write a review for the article. You will be given the title of the article, the number of the round in which the article is located, and your order among the reviewers.

Input:
Title: PREDICTING THE RESULTS OF EVALUATION PROCEDURES OF ACADEMICS
Review round: 1
Reviewer: 1

Output:
Basic reporting: The paper “Predicting the results of evaluation procedures of academics” aims to assess whether it is possible i) to predict the results of the Italian Habilitation (ASN) using only the information contained in the candidates’ CVs?; ii) to identify a small set of predictors of the ASN results?
The paper is successful in reaching its targets and I admire the capacity of the author(s) of developing a very wide and rich dataset, which is less exposed to biases, as they stress in the comparative evaluation of their model.
Experimental design: Barring from being an expert of machine learning algorithms and techniques, the paper and its findings seem convincing.
Validity of the findings: I have just few minor tips for the authors.
First, discussion and conclusions should stress not only the prediction ability of the model developed, but also discuss the policy and practical implications of their findings for policy-makers. For instance, the maximum number of years with affiliation to the same university for obtaining ASN is a finding that contradicts the literature stream that stresses mobility as a positive feature in the academic career. Some kinds of tentative explanations of this unexpected results, maybe related to the specificities of the Italian context wherein the analysis took place, would be suggested.
Also practical implications of the findings should be stressed. E.g. Should the policy makers rely more machine learning techniques in order to make habilitation choice thus saving time of academic staff from the time-consuming peer review?
Finally, a very minor point, I would suggest in the Discussion and Conclusions section to replace the name of variables as defined in Figure 4, with the correspondent description to facilitate the reading.
Additional comments: No additional comments


Instruction: You are one of the reviewers, your task is to write a review for the article. You will be given the title of the article, the number of the round in which the article is located, and your order among the reviewers.

Input:
Title: PREDICTING THE RESULTS OF EVALUATION PROCEDURES OF ACADEMICS
Review round: 1
Reviewer: 2

Output:
Basic reporting: While the research presented here is of very high quality, the structure of the paper can however be improved on various accounts. For once, while the literature review is both impressive and thorough, I wonder if the "prediction of bibliometric indicators" subsection is absolutely necessary here, as the works discussed in detail there are of no "direct" relevance to the research and isn't referred to or used as benchmark in the "evaluation" section. I suggest that the authors sum up this subsection in a few lines, add it to the introductory paragraph of the "Related Work" section, and remove the "Prediction of the Results of Evaluation Procedures" subsection title. Also, the "results" section contain methodological procedures that do not contain any results. We suggest that the subsections contained in the "results" section be moved to the "methods and material" section, in which they clearly belong, and that the "evaluation" section be then renamed "results". Finally, while the content of the "Italian Scientific Habilitation" section is of crucial importance to the paper, I am not sure whether devoting a distinct section to it is the best option: some of its content clearly belongs to the introduction section, while the segment on bibliometric and non-bibliometric indicators would clearly fit in the "Methods and Material" section. In light of the above comments, I think that this paper would benefit from sticking closer to the IMRaD structural standard.
Experimental design: Additional information on why the 2012 ASN data is a representative and exhaustive sample of the whole population would have been welcome.
Validity of the findings: No comment.
Additional comments: No additional comments


Instruction: You are one of the reviewers, your task is to write a review for the article. You will be given the title of the article, the number of the round in which the article is located, and your order among the reviewers.

Input:
Title: PREDICTING THE RESULTS OF EVALUATION PROCEDURES OF ACADEMICS
Review round: 1
Reviewer: 3

Output:
Basic reporting: In this manuscript, the authors develop a data mining approach to predict the results of ASN and identify some important features. The data used in this paper contains 59,149 researchers and 1,910,873 papers spanning in 184 recruitment field, which is relatively larger compared to related works. The treatment of the problem is technically sound.
1 - The acronym of Recruitment Field and Random Forests are both “RF”. I think this may lead to ambiguity.
2 – The quality of Figure 1 needs to be improved.
Experimental design: no comment
Validity of the findings: no comment
Additional comments: The subject and the proposed method are valuable. The problems and solutions in the paper are clearly stated. So, I believe that the article is suitable to be published in "PeerJ  Computer Science journal" after a little revision.


Instruction: You are one of the reviewers, your task is to write a review for the article. You will be given the title of the article, the number of the round in which the article is located, and your order among the reviewers.

Input:
Title: PREDICTING THE RESULTS OF EVALUATION PROCEDURES OF ACADEMICS
Review round: 2
Reviewer: 1

Output:
Basic reporting: The review implements the changes requested in the first-round of the review process. So, I believe that the article is suitable to be published as it is.
Experimental design: The review implements the changes requested in the first-round of the review process.
Validity of the findings: The review explains better the policy and practical implications of their findings for policy-makers.
Additional comments: No additional comments


Instruction: You are one of the reviewers, your task is to write a review for the article. You will be given the title of the article, the number of the round in which the article is located, and your order among the reviewers.

Input:
Title: PREDICTING THE RESULTS OF EVALUATION PROCEDURES OF ACADEMICS
Review round: 2
Reviewer: 2

Output:
Basic reporting: The paper has been revised well. All my concerns have been solved.
Experimental design: N/A
Validity of the findings: N/A
Additional comments: N/A


Instruction: You are one of the reviewers, your task is to write a review for the article. You will be given the title of the article, the number of the round in which the article is located, and your order among the reviewers.

Input:
Title: EFFICIENT FACIAL REPRESENTATIONS FOR AGE, GENDER AND IDENTITY RECOGNITION IN ORGANIZING PHOTO ALBUMS USING MULTI-OUTPUT CONVNET
Review round: 1
Reviewer: 1

Output:
Basic reporting: The quality of English language is high. The text is written correctly with minor mistakes:
1) "that not all images ... contains information" (188 line)
2) "can be applied is we naively assume" (216 line)
Experimental design: The proposed CNN and clustering pipeline show the novelty of the approach. Everything is described clear and with maximum details. The description provided make the paper easy to replicate to test the approach. All the methods sections and subsections leave good impression about the research overall.
Validity of the findings: The experimental part is written very detailed and the authors obtained high precision/recall results in comparison to existing approaches. Moreover, the proposed MobileNet-based CNN lead to low computational complexity and high inference speed on mobile devices. What I would like to recommend is to do research on a new dataset UTKFaces, which contains information about age/gender/ethnicity. Seems the proposed Mobilenet structure can be extended to the 3rd output for ethnicity detection and this will be a good improvement in the research field.
Additional comments: No additional comments


Instruction: You are one of the reviewers, your task is to write a review for the article. You will be given the title of the article, the number of the round in which the article is located, and your order among the reviewers.

Input:
Title: EFFICIENT FACIAL REPRESENTATIONS FOR AGE, GENDER AND IDENTITY RECOGNITION IN ORGANIZING PHOTO ALBUMS USING MULTI-OUTPUT CONVNET
Review round: 1
Reviewer: 2

Output:
Basic reporting: The paper describes a single lightweight CNN model to handle face identification, age and gender recognition problems. The proposed method trains only data relevant part of the network by alternating the mini-batches with age and gender info. The experimental results show that the proposed model has impressive run-time/space complexity value. The related work section also appears to be strong.

On the other hand, the paper is a bit difficult to follow, the language and paper organization can ideally be improved. To give an example, some information about experiments are shared in the "Multi-Output ConvNet for Simultaneous Age, Gender and Identity Recognition" section. Moreover, there are some claims (e.g. line 199-201) without an experiment or external scientific reference. Also, I do not understand the sentence in lines 225. Finally, there are typos such as simulatenous->simultaneous in line 140.
Experimental design: The experiments give some insight about the strengths and weaknesses of the proposed method. The experimental results show that the proposed model obtained promising results in terms of model size and accuracy.

Some details about the model are given in the shared code link. To give an example, information about loss function and learning rate are not shared in manuscript or missed it. It would be nice to see these information in the manuscript to reproduce same experiments.
Validity of the findings: The paper looks technically sound and the problem is well motivated. Moreover, the proposed approach is simple and reasonable. However, there are some points that need explanation, respectively:

- According to the Figure 2, identity features and born year predictions are given as input to the clustering operation. However, different situation is mentioned in the line 209-210.

- Where is the proof of the claim in lines 202, 203 and 204?

- How do you encode age values? What is the loss function for it?

- Do you have joint loss function for learning gender and age probabilities?
Additional comments: My general observation about the manuscript is that idea is nice, but text organization is very poor.


Instruction: You are one of the reviewers, your task is to write a review for the article. You will be given the title of the article, the number of the round in which the article is located, and your order among the reviewers.

Input:
Title: EFFICIENT FACIAL REPRESENTATIONS FOR AGE, GENDER AND IDENTITY RECOGNITION IN ORGANIZING PHOTO ALBUMS USING MULTI-OUTPUT CONVNET
Review round: 1
Reviewer: 3

Output:
Basic reporting: I think this is an interesting paper on a relevant topic.
The language is clear and informative.
The author provides all the necessary references. Many of them are dated to the last few years that confirms the awareness of the state-of the art in the field.
The paper is well-structured. All the figures and tables are relevant and discussed in the paper in details.
However, the paper contains some typos and minor issues, which are listed in “General comments for the author” section.
Experimental design: The paper relates to the most popular computer science fields, namely, machine learning and mobile development.
More precisely, the article deals with the actual problem of recognizing gender, age and person identification in digital images.
One of the most valuable pros. of the paper is the number of experiments. The experiments are dedicated to choosing the best algorithms and comparison with the available models.
The proposed technique is well described. The implementation is available at github for the community.
Validity of the findings: I think that the work is significant and will be useful to specialists in the field.
Additional comments: -	I think it would be better to state in abstract that the proposed technique is ready to use with mobile devices.
-	Please, clarify why dataset «contains incorrect ground truth values of age» (line  45)
-	MTCNN is used without reference (line 208)
-	Please, clarify what “most important” means in «Moreover, we assume that the most important clusters should not contain photos/videos made in one day» (line 244-245)
-	Please, check if the complexity should be higher: “computational complexity of such clustering is 3-4-times lower” (line 302)
-	Please, check if the “inference” should be “faster” or the time should be “lower”: “The inference time in the MobileNet is also known to be much slower.” (line 328-329)
-	Please, check line thickness in fig. 2as the bottom dash line disappears at some (ordinary)) scale in my Acrobat Reader.
Typos:
-  «have been study» 81
- «resulted model are made»267
Missed or wrong prepositions:
- «using embeddings at one the last layers» (line 77)
- «using embeddings at one the last layers» (line 78)
- «information interesting a target group» (line 100)
- “to predict in an observed person is 1,2,... or 100 years old”
Missed verb:  «At first, the base MobileNet for face identification using  very large facial dataset, e.g., VGGFace2” line 259-260)


Instruction: You are one of the reviewers, your task is to write a review for the article. You will be given the title of the article, the number of the round in which the article is located, and your order among the reviewers.

Input:
Title: EFFICIENT FACIAL REPRESENTATIONS FOR AGE, GENDER AND IDENTITY RECOGNITION IN ORGANIZING PHOTO ALBUMS USING MULTI-OUTPUT CONVNET
Review round: 2
Reviewer: 1

Output:
Basic reporting: The new version seems much better and the updated experiments provide better conclusion of the research.
Experimental design: Experiments added on other datasets like UTKFaces gave a better understanding of the value of conducted research.
Validity of the findings: No comment
Additional comments: The paper presents a concluded research now and the text is filled with different experiment scenarios and results. I have no more comments on the paper


Instruction: You are one of the reviewers, your task is to write a review for the article. You will be given the title of the article, the number of the round in which the article is located, and your order among the reviewers.

Input:
Title: EFFICIENT FACIAL REPRESENTATIONS FOR AGE, GENDER AND IDENTITY RECOGNITION IN ORGANIZING PHOTO ALBUMS USING MULTI-OUTPUT CONVNET
Review round: 2
Reviewer: 2

Output:
Basic reporting: The sections in the manuscript seem to be correctly separated, after revisions.
Experimental design: The additional experiments respond to my previous questions.  Moreover, rebuttal text advocate the arguments put forward in the manuscript more strongly by using these experiments.
Validity of the findings: The method used looks technically sound.
Additional comments: Thank you very much for your corrections and responses. The proposed rebuttal and experiments answer my questions. My views are to accept the manuscript.


Instruction: You are one of the reviewers, your task is to write a review for the article. You will be given the title of the article, the number of the round in which the article is located, and your order among the reviewers.

Input:
Title: EFFICIENT FACIAL REPRESENTATIONS FOR AGE, GENDER AND IDENTITY RECOGNITION IN ORGANIZING PHOTO ALBUMS USING MULTI-OUTPUT CONVNET
Review round: 2
Reviewer: 3

Output:
Basic reporting: I think this is an interesting paper on a relevant topic.
The language is clear and informative.
The author provides all the necessary references. Many of them are dated to the last few years that confirms the awareness of the state-of the art in the field.
The paper is well-structured. All the figures and tables are relevant and discussed in the paper in details.
The revised version of the paper contains all the necessary corrections and additions.
Experimental design: The paper relates to the most popular computer science fields, namely, machine learning and mobile development.
More precisely, the article deals with the actual problem of recognizing gender, age and person identification in digital images.
One of the most valuable pros. of the paper is the number of experiments. The experiments are dedicated to choosing the best algorithms and comparison with the available models.
The proposed technique is well described. The implementation is available at github for the community.
Validity of the findings: I think that the work is significant and will be useful to specialists in the field.
Additional comments: As the revised version of the paper contains all the necessary corrections and additions, I vote for accepting this paper.


Instruction: You are one of the reviewers, your task is to write a review for the article. You will be given the title of the article, the number of the round in which the article is located, and your order among the reviewers.

Input:
Title: ENHORT: A PLATFORM FOR DEEP ANALYSIS OF GENOMIC POSITIONS
Review round: 1
Reviewer: 1

Output:
Basic reporting: In "Enhort: a platform for deep analysis of genomic positions" Menzel et al present a new tool aimed at the investigation of viral integration sites in the human genome. The main addition to the state of the art is the user-friendly method for incorporating prior knowledge (i.e. biological or technical biases) in the analysis of these sites.

**English and language**: Overall, the paper is written in coherent and legible English, albeit that it includes some typos and the odd unfinished sentence. These include (but might not be limited to):
* l55: to control sites. Yielding a measure => to control sites, yielding a measure
* Legend figure 2: integration vs. controll sites => integration vs control sites
* l76: distance as well as for several histone => distance as well as several histone
* l78: Without background models build for multiple covariates => Without background models built for multipe covariates
* l80-82: This is not a correct sentence (including a non-closure of the parentheses on l82).
* l116: Figure 3B shows the screenshots of sequence logos => Figure 3B shows the sequence logos
* l119: To precisely analyzes the preferences => To precisely analyze the preferences
* l119-120: to be adapted to replicated the TTAA motif => to be adapted to replicate the TTAA motif
* l124: characteristics seen Figure 3A => characteristics seen in Figure 3A
* l125: significant preferences => significant preference
* l132: To further review the analytic capabilities the integration counts => To further review the analytic capabilities, the integration counts

**Structure**: The structure of the manuscript reflects the contents and best practices. However, the "Discussion" section does not contain any discussion. I suggest renaming the "Results" section to "Results and Discussion", and the "Discussion" section to "Conclusion". Similarly, the section "Literature Review" should be renamed to "Possible impact of Enhort" or similar.

**Definition of terms etc**: Although most terms are well-defined, it is not clear in the legend of figure 2 what a contig refers to ("contig" in the genome assembly sense?)
Experimental design: It would have been good to try the tool out, but unfortunately the website cannot be reached. Similarly, the gitlab repository does include build instructions as well as an example database, but this is insufficient for review as I encountered issues installing maven on my computer which is necessary to try the tool out.
Having the website running is a sine-qua-non before this manuscript can be accepted. In addition, it would be good if the authors provided e.g. a docker image with example data which would also include a walkthrough of the patterns that one is looking for.
Validity of the findings: As the user can try out adding different biases to the background model, the authors should take into consideration the effect of multiple testing: is this relevant here or not, and why. See e.g. l129-130: "Using the background adaption feature of Enhort it would be possible to test different hypothesis"

The last sentence of section "Literature Review" states "Since data analysis only takes minutes using Enhort, we assume that the authors of these papers would have saved a lot of effort if they had used our tool." This is obvious speculation and should be identified as such. Were the authors of those papers contacted to try out Enhort?
Additional comments: In addition to the structured comments above, there are some additional points to be made:

[1] On l122-123, the authors mention that the background sites in 3C and those for PB show a similar motif after the motif is added as a covariate using Enhort. Isn't this a logical consequence? If you'd measure lengths of individuals and the average thereof is higher than those for women, you could similarly add gender as a covariate and of course your average would become identical with that for women...
[2] The authors state in l145-146 that "The analysis of PB integration sites demonstrates that Enhort is capable to reproduce analysis from literature with little effort." However, "from literature" refers to a single published paper (i.c. Wilson et al 2007). There are 2 issues with this. First, being only compared to a single paper does not constitute proof that a method can actually be applied more generically, and can almost be interpreted as anectodal evidence. Second, in comparing Enhort with Wilson et al, the authors had the advantage of hindsight and knew what they were looking for. Therefore, reproducing results "with little effort" becomes less meaningfull. I suggest the authors include a comparison with at least one other study, and take consideration of the fact that they already know what they're looking for in this comparison. A way to circumvent this problem is to make predictions using Enhort that are later validated in a separate study.
[3] It is not completely clear what the authors mean at l116 with "sequence logos for PB". Are these sequence logos of known integration sites?
[4] Figure 2 misses legends for all 3 subfigures: what are the different bands in 2A? What is the colour scale in 2B? Is 2C just a linear version of 2A?


Instruction: You are one of the reviewers, your task is to write a review for the article. You will be given the title of the article, the number of the round in which the article is located, and your order among the reviewers.

Input:
Title: ENHORT: A PLATFORM FOR DEEP ANALYSIS OF GENOMIC POSITIONS
Review round: 1
Reviewer: 2

Output:
Basic reporting: This manuscript describes Enhort, a system for associating annotation and genomic position data. The system relies on existing statistical methods to provide familiar statistics and visualizations that would support a broad variety of analyses.

The high-level organization of the manuscript is reasonable. The figures are high-quality and informative. The writing at the beginning of the manuscript is relatively high quality; however, as the manuscript progresses, the writing becomes less clear and contains more typos. The manuscript would benefit from an end-to-end editing pass.

The manuscript contains a number of useful references to example studies that Enhort would support. These references are helpful for understanding and contextualizing the motivation for the system. However, the specific contributions of the system beyond existing approaches is unclear in large part due to a lack of comparison to existing systems. The manuscript notes that many systems exist for simultaneous position and annotation analysis, but only one such system is noted. The claimed difference appears to be that this system provides "instant analysis and user-defined adaptable background models that mimic known biases." Without a comparison to existing approaches, it is difficult to see how Enhort truly achieves this goal. Systems like IGV, Artimis, and Bioconductor all enable annotation analysis alongside genomic position and range data and appear to share many common features with Enhort. A broader discussion of the contribution of the system in relation to this work would significantly strengthen the manuscript.
Experimental design: There are a number of missing details as to the actual functionality of the system and how the system achieves the technical contributions noted above and extended on in the introduction as to "accelerate and simplify the data analysis process as well as to standardize it in order to increase reproducibility."  The primary method to achieve this in Enhort is through the integration of user-selected covariates to serve as a control site. The idea of integrating covariates to serve as controls in annotation analysis is intriguing. However, there is little discussion of how these covariates are integrated, how the system might support the selection of relevant covariates, or how the system supports transparency in how the covariates shift analysis after integration. A use case illustrating the target analysis workflow and how Enhort manages this workflow would be helpful in illustrating these methods as would a formal comparison to additional approaches (see above).

The deployment of Enhort as a web-based application seems like a useful method for deploying and maximizing the impact of the system. The system appears to include access to several existing samples. However, given the aim stated in the abstract of targeting large-scale datasets, the choice of a SQLite backend seems like it may be limiting. A performance analysis or discussion of scalability would be useful and such a discussion should tie to the impact of scale on the integration of covariates as well as how covariate integration might influence performance.

The system appears to integrate a number of standard analytical tools for statistical and visual analysis, which, coupled with the web-based implementation, provides a useful and accessible method to integrate familiar analyses. However, these tools are described as if they are only used once data has been fully integrated into the system. It is unclear how the system supports either the instant integration or standardization that are noted as target contributions. Additional discussion of what exactly is intended in these contributions as well as how Enhort is architected to support these factors should be included.
Validity of the findings: Enhort is evaluated primarily using a literature review of prior analyses. However, the conclusion of this review is that a large percentage of the target analyses could have been completed using Enhort, and the manuscript speculates that the analyses could have been conducted more efficiently with the system. The speculation is backed by a specific case study attempting to reproduce Gogol-Doring et al's 2016 results and Wilson et al's PB site analyses.

This method of evaluating the system seems limited given the lack of comparison against existing approaches. For example, what grounds are there for the speculation that Enhort could have saved substantive effort in the prior papers? How does Enhort's methods improve on these prior analyses? What novel components of Enhort allow for these analyses to be conducted in ways that prior methods could not achieve?

The case study evaluation is also somewhat limited. While the case study does surface alternative hypotheses for the 2016 data, no discussion is provided as to how comparable the analysis is to the original approach. Further, the comparison to the 2007 data is limited: was there other data or testing that ruled out these hypotheses or are these hypotheses from the 2007 study that required an approach like that offered by Enhort to achieve? While a lack of full replicability on an older study is understandable, it also weakens the argument that Enhort is superior to existing methods. It may be more interesting to see cases where Enhort could improve on existing methods or at least to provide some evidence showing how the tool significantly aids biological analysis workflows.

Overall, the validity of the claims in the paper is not well-supported by the current evaluation. Specific cases illustrating where Enhort can improve on current approaches or offer novel insights into data as a result of its covariate model and other features would provide stronger evidence of the validity of the system's contributions. The evaluation should also more directly substantiate the specific claimed contributions of the method, notably those of instant, accelerated, and simplified analyses. The current evaluation makes efforts towards the replication claims; however, it should also be substantially strengthened either through a deeper discussion or through collaborations with scientists
Additional comments: The manuscript overall would benefit from a stronger focus on the specific technical contributions of the system. This includes a comparison to existing approaches to highlight the novelty of the approach, a discussion of how the system achieves these novel aims, and an evaluation more directly illustrating the claimed contributions. Enhort appears to bundle interesting statistical approaches with thoughtful considerations of necessary analysis components; however, without a clearer discussion and evidence of the utility of these contributions, it is difficult to assess the novelty and impact of the system.


Instruction: You are one of the reviewers, your task is to write a review for the article. You will be given the title of the article, the number of the round in which the article is located, and your order among the reviewers.

Input:
Title: ENHORT: A PLATFORM FOR DEEP ANALYSIS OF GENOMIC POSITIONS
Review round: 2
Reviewer: 1

Output:
Basic reporting: The overall language in the paper has been significantly improved. Remaining issues include:
* Shao et al. is noted outside of the parentheses
* "symbols mark statistical significant"--statistically
* "vs controll sites" -- control
* "are used as comparison" -- as a comparison (or for comparison)
* "enables scientist to" -- "enables scientists to"

The added figures help clarify the workflow. The figures could be improved in the following ways:
* Figure 1: The Enhort cell's text is not centered.
* Figure 2: The fact that two charts on the right expand on steps on the leftmost chart is not obvious, especially given the spatial overlap between the blue chart and primary chart. Labels, call-outs, and/or improved spacing could help clarify that the blue and green charts are subsets of the original flowchart on the left. Further, the text spacing relative to both the background cells and between cells (e.g., "for each combination") suffers from significant visual tension (e.g., many of the cells are too small so the text is close to the cell borders) and should be revised.
Experimental design: The response letter was helpful in understanding the primary contribution and importance of the approach (e.g., providing an automated WYSIWYG system for conducting the target set of analyses); however, these details are still largely missing from the paper (they do not appear until the conclusion). The Introduction should make clear what the current workflows look like (e.g., custom scripts ideally with citations to example studies), the limitation of current approaches, and the workflows that Enhort supports to overcome these limitations. Further, custom scripts represent a trade-off: they require time to develop, but are flexible, often highly reusable and can work with custom data formats. It would be useful to have some discussion of the trade-offs of Enhort and scripts constructed using existing packages.

The Methods section is also significantly improved; however, there is still a limited sense of what specific features Enhort offers. For example, what visualizations are included? How are the "appropriate figures" selected? What kinds of additional statistical and analytical support is provided? The Methods discussion mentions many features, but it would be helpful to have some sort of summary either as a table or specifically enumerated in the text.
Validity of the findings: The added discussion of Wu et al is helpful for understanding how Enhort can not only reproduce analyses but be used to rapidly extend investigations to develop and evaluate additional hypotheses. I appreciate the point raised by the response letter as to challenges in evaluating these kinds of systems. However, there are several points in the evaluation that would benefit from further detail. For example, what proportion of analyses in the literature review were conducted using new custom scripts versus other tools or replicating prior analyses (which would imply reuse)? While these details may not always be available, commenting on specific instances where something concrete can be said about the way the analyses were conducted (e.g., looking at provided supplements or methods mentioned in the paper) would significantly strengthen the claims made in this section. If no such examples can be provided, the speculation about time saving should be removed as the current literature review provides evidence of the utility of the target analyses but does not provide any evidence in support of time savings.

Table 1 is a beneficial addition to understand the processing power of Enhort. While a discussion of where the computational efforts are taking place is not necessary given the scope of the paper, it would be helpful to note the hardware these analyses were conducted on for reproducibility (either the client side specifications if client-side computation was conducted or the server specifications if all computation is conducted on the server).

In the re-analyses, it would be helpful to at least speculate on reasons why Enhort's results varied from the original analyses from Wilson et al. and Wu et al. The detailed expansion on the Wu et al. analyses provides a useful example of the utility of Enhort's approach and its importance for improved data analysis; however, the differences in the two outcomes may lead to some concerns around flexibility: does Enhort's automated workflow prevent scientists from tuning their analyses with sufficient precision? A discussion of where these variations might arise would be useful in understanding potential trade-offs.
Additional comments: Overall, the paper has been significantly improved. However, the manuscript should still provide a more explicit and grounded discussion of current practices to highlight the utility and trade-offs of Enhort.


Instruction: You are one of the reviewers, your task is to write a review for the article. You will be given the title of the article, the number of the round in which the article is located, and your order among the reviewers.

Input:
Title: DATA-BASED INTERVENTION APPROACH FOR COMPLEXITY-CAUSALITY MEASURE
Review round: 1
Reviewer: 1

Output:
Basic reporting: Dear authors
thanks for submitting this interesting paper.
The results seem promising, the description needs some smoothing and some extra info, and it would be good to abandon the too often used approach of "here's a good method that outperforms other", since this is rarely the case, and the field would benefit from a more global view in which convenient aspects of several approaches are merged.

You mention that Granger causality is a "model free" approach, which it isn't. Transfer Entropy is. GC is rooted on autoregressive models. Maybe you meant "data driven".

The introduction of the paper is a bit confusing. I don't think that these methods address any of the ladders of causality, and maybe they shouldn't either. Of course when we apply these methods to real data, we are in a causal framework in the sense of the counterfactuals, but again, I find the introductory sentences misleading and dissociated from the actual content of the paper.
Experimental design: You could maybe better explain the philosophy behind your method, and whether it addresses the effects or the mechanism. Also the applications are either abstract benchmarks (still useful) or a case in which a ground truth or intervention is difficult to assess.
Validity of the findings: While your approach seems powerful for the data presented, it would be good and fair to state possible scenarios in which the performance is not so good.

I understand that Granger Causality and Transfer Entropy are among the most used approach, but other measures exist, such as the Convergent Cross Mapping http://science.sciencemag.org/content/338/6106/496.

Also, other approaches have proposed compression as a tool to evaluate causality, see for example

https://arxiv.org/abs/1611.00261

https://eda.mmci.uni-saarland.de/pubs/2016/origo-budhathoki,vreeken.pdf

Could you maybe comment on these aspects?
Additional comments: No additional comments


Instruction: You are one of the reviewers, your task is to write a review for the article. You will be given the title of the article, the number of the round in which the article is located, and your order among the reviewers.

Input:
Title: DATA-BASED INTERVENTION APPROACH FOR COMPLEXITY-CAUSALITY MEASURE
Review round: 1
Reviewer: 2

Output:
Basic reporting: The paper "Data based intervention approach for Complexity-Causality measure" proposes a new model-independent measure aimed at describing high-order causal relations between signals. The authors provide context for the relevance of the proposed measure, comparing it against traditional model-free measures and show that their measure is generally able to provide better discrimination in simple synthetic cases and a couple of real systems.
I commend the authors for their extensive work explaining the new measure with different examples of growing complexity. I highlight a weakness in my comments below regarding the lack of comparison with a model-dependent case. The manuscript is written clearly, although occasionally there is a missing preposition or article. I recommend a further thorough check of the manuscript. Moreover, code is provided to replicate the analysis.
Overall, I think the manuscript is a valid contribution. I however harbor some doubts about the capacity and usefulness of the new measure in complex networked systems, which are the systems where most likely model-based measures fail or are unavailable. Hence, as mentioned in the comments below, I think that, before being accepted, the paper should include the comparison with a case where model-dependent is available, even if for a simple system.
Experimental design: no comment
Validity of the findings: Major comments:
- the authors throughout the paper compare first-order with high-order AR models in order to show that, while TE/GC sometimes work for AR(1) models, they consistently fail for higher-order one, thus highlighting the capacity of CCC to capture hierarchically higher causality effects. This is good, but what I would like to see is whether CCC is able to outperform specific model-dependent measures and observations in systems where we do have such specific equations.
Indeed, the authors mention DCM and SEM too: it would be great if it were possible to show that the direction and intensity of interactions picked up by DCM in real-data (even in one of the examples with few-nodes in Friston's DCM toolbox on EEG or fMRI data) corresponds to that captured by CCC. I understand that the authors mention something similar as future work, but given their claims about CCC, I would really like to see this in the paper.

- Following up, how hard computationally is to compute CCC versus existing measures of complexity? Is this a tool that can be thrown at data cheaply or is it instead to heavy to use as a first general-purpose step of analysis? Can we imagine to scale this up to the case of hundreds of signals, as is typically the case in neuroimaging applications?
Additional comments: - There are no error bars reported anywhere in the plots. Assuming that the curves shown are obtained over repeated simulations of the processes for each parameter choice, it would be interesting to understand the variance/magnitude of fluctuations around the reported mean values for both GC, TE and CCC.

- the figure labels are acceptable, but  a little small; I'd recommend making the labels larger for improved readability.


Instruction: You are one of the reviewers, your task is to write a review for the article. You will be given the title of the article, the number of the round in which the article is located, and your order among the reviewers.

Input:
Title: DATA-BASED INTERVENTION APPROACH FOR COMPLEXITY-CAUSALITY MEASURE
Review round: 2
Reviewer: 1

Output:
Basic reporting: nothing to add
Experimental design: There are many EEG and fMRI data available, openneuro.org has hundreds of them, not to mention the benchmark EEG and fMRI data included in the SPM package. So a comparison with DCM is certainly feasible. On the other hand we should not put all connectivity estimates in the same box, DCM is a confirmatory approach meant to disclose the involvment of synaptic parameters, and should not be used for network discovery and to assess the simple presence of links.
A benchmark should be carefully chosen in order to map the effects of an intervention in the two cases.
Validity of the findings: The findings are valid, but I see the risk that all possible connectivity (or "causality") estimates are conflated, and in this case any comparison and interpretation is meaningless.
Additional comments: No additional comments


Instruction: You are one of the reviewers, your task is to write a review for the article. You will be given the title of the article, the number of the round in which the article is located, and your order among the reviewers.

Input:
Title: DATA-BASED INTERVENTION APPROACH FOR COMPLEXITY-CAUSALITY MEASURE
Review round: 2
Reviewer: 2

Output:
Basic reporting: The authors improved significantly the manuscript according to my previous and the other referees' suggestions. In particular, they showed the effectiveness of their measure in specific cases of biological relevance.
Experimental design: no comment.
Validity of the findings: no comments
Additional comments: No additional comments


Instruction: You are one of the reviewers, your task is to write a review for the article. You will be given the title of the article, the number of the round in which the article is located, and your order among the reviewers.

Input:
Title: DATA-BASED INTERVENTION APPROACH FOR COMPLEXITY-CAUSALITY MEASURE
Review round: 2
Reviewer: 3

Output:
Basic reporting: Overall the manuscript is clearly reported. A couple of minor issues...

The proposed method is based on the Effort-to-compress measure, but this is not described in the manuscript (only in the supplementary material). This severely effects the clarity of the article. This is particularly strange given that a number of other measures, which are not central to the proposed measure, are defined in the manuscript (e.g., KL-divergence and JS-divergence). I suggest including more details about the ETS in the main text.

The large number of acronyms make the manuscript bothersome to read. It would be useful to remove some of these or at least provides some reminders along the way. Also it seems that ICC and CCC are the same thing except that the latter is specifically using ETC as the notion of complexity.
Experimental design: The experimental design seems adequate.
Validity of the findings: The validity of the findings appear to be sound. However, I'd like to echo reviewer one's comment on stating the scenarios in which performance can be expected to be good and the scenarios for which it is expected to be bad. Perhaps a thorough analysis of this is beyond the scope of the current work. But without such analysis, the usefulness of the proposed approach may be limited as one can never be sure under what circumstances the method is applicable. Indeed this is a general limitation for so-called "model-free" approaches since the assumptions are ignored rather than clearly stated.
Additional comments: No additional comments


Instruction: You are one of the reviewers, your task is to write a review for the article. You will be given the title of the article, the number of the round in which the article is located, and your order among the reviewers.

Input:
Title: ASPECT EXTRACTION ON USER TEXTUAL REVIEWS USING MULTI-CHANNEL CONVOLUTIONAL NEURAL NETWORK
Review round: 1
Reviewer: 1

Output:
Basic reporting: see below
Experimental design: see below
Validity of the findings: see below
Additional comments: The manuscript is centered on an interesting topic.
Organization of the paper is good and the proposed method is quite novel.

The manuscript, however, does not link well with recent literature on sentiment analysis appeared in relevant top-tier journals, e.g., the IEEE Intelligent Systems department on "Affective Computing and Sentiment Analysis". Also, check latest trends in deep learning based aspect extraction, e.g., attentive LSTM.

Many issues with acronyms: some acronyms are used without being defined first, e.g., CNN, some other are defined but then never used, e.g., CRF.

Some bad English constructions and misuse of articles: a professional language editing service is strongly recommended (e.g., the ones recommended by IEEE, Elsevier, and Springer) to sufficiently improve the paper's presentation quality for meeting PeerJ's high standards.


Instruction: You are one of the reviewers, your task is to write a review for the article. You will be given the title of the article, the number of the round in which the article is located, and your order among the reviewers.

Input:
Title: ASPECT EXTRACTION ON USER TEXTUAL REVIEWS USING MULTI-CHANNEL CONVOLUTIONAL NEURAL NETWORK
Review round: 1
Reviewer: 2

Output:
Basic reporting: • The article is generally written in a clear, coherent and professional English. However, there are some segments that need to be corrected because they are not well understood or they are incorrectly written. For the next revision, it is necessary to improve the segments at lines: 13, 38, 40, 88 to 91, 92, 95 to 97, 153, 191 to 192, 262, 320 to 321, 396 to 397, and 476.
• The Introduction and background are correct because they reveal the problems that need to be addressed and clearly explain the objectives of the paper. And the related works are generally well-founded because they explain clearly the works that have addressed the problem of aspect extraction.
• The work is very well structured.
• The tables and figures are relevant and help to better understand the methodology and results obtained, however, figures 2 and 3 need to be edited and eliminate the title left by mistake in both figures: “Chart Title”. In addition, it would be good to explain a little better all the figures in the caption that corresponds to them.
• Conclusion of basic reporting: Approved
Experimental design: • The research of the article satisfies the Aims and Scope of the journal
• The problems existing with the aspect extraction of reviews task and how these problems can be solved with the paper proposal are clearly specified. In addition, the suggested methods are quite relevant and novel for the current literature. As a suggestion, the research question should be clearly specified so that it can then be answered in the conclusions of the work.
• Many works found in the literature have been explained, which is quite good. However, it would be ideal to explain a little more the advantages and disadvantages of those works that you consider most important and that are better related to the proposal of your work, that is, you might detail a little more explaining their performance.
• The methodology of the work is clearly explained, the methods are described in detail.
• In the part of the explanation of the datasets, it would be interesting to show some examples of the most important aspects that are found in each dataset, no example was shown. Explain a little better the datasets used.
• Conclusion of experimental design: Approved
Validity of the findings: • The article showed a large number of experiments with relevant results. The data is robust and statistically sound and meets the objectives that were raised in the introduction of the article.
• The conclusions are well established and explain the importance of the results that have been obtained. However, since the research question has not been clearly defined previously (in the introduction), the conclusion seems a bit incomplete. The research question should be defined and finally answer that question according to the results obtained in this work.
• Conclusion of validity of the findings: Approved
Additional comments: No additional comments


Instruction: You are one of the reviewers, your task is to write a review for the article. You will be given the title of the article, the number of the round in which the article is located, and your order among the reviewers.

Input:
Title: ASPECT EXTRACTION ON USER TEXTUAL REVIEWS USING MULTI-CHANNEL CONVOLUTIONAL NEURAL NETWORK
Review round: 1
Reviewer: 3

Output:
Basic reporting: The manuscript entitled "Aspect extraction on user textual reviews using Multi-Channel Convolutional Neural Network" presents an approach for aspect extraction using deep multichannel convolutional neural network. The main topic is relevant and interesting to the scope of this journal. I would like to highlight some points about the general organization of the paper that should be revised by the authors.

- I recommend that the manuscript be reviewed for what concerns the English language for both the composition of the sentences and the grammar.
- The text also needs more attention with some sentences. For instance, "As a feature direction of the research" (line 480)

- There are many abbreviations in the introduction that were not defined in the text, like "LSTM".
- "CRF" is defined twice.

- Some figures need to be improved. In Fig. 2 it is missing the legend for the orange bar. In Fig. 3, the x-axis is not shown.

The literature references and the related works are well structured and the main background of the manuscript is provided.
Experimental design: The main theoretical tools approached by the authors are cutting-edge research on many machine learning and related topics nowadays and the authors explore their architecture for aspect extraction within the context of sentiment analysis. I missed some motivation for what concerns this main application of the method. I think that the authors could improve the text by adding some real-world examples of the datasets that were used. This way, they can illustrate the main application of the proposed approach and motivate the reader to go further in the methodology and the results.
Validity of the findings: The research question is well defined in the manuscript. The authors performed a deep investigation of the results by making comparisons with other methods in literature and also with some variations of the proposed model. The authors showed that the proposed approach could outperform many state of the art methods. Therefore, the main contribution of the manuscript is the performance gain obtained by the proposed method.
Additional comments: No additional comments


Instruction: You are one of the reviewers, your task is to write a review for the article. You will be given the title of the article, the number of the round in which the article is located, and your order among the reviewers.

Input:
Title: ASPECT EXTRACTION ON USER TEXTUAL REVIEWS USING MULTI-CHANNEL CONVOLUTIONAL NEURAL NETWORK
Review round: 2
Reviewer: 1

Output:
Basic reporting: All my comments have been addressed. However, the compilation of some references should be double-checked, e.g., see "Opinion Miningwith"
Experimental design: See my previous review
Validity of the findings: See my previous review
Additional comments: No additional comments


Instruction: You are one of the reviewers, your task is to write a review for the article. You will be given the title of the article, the number of the round in which the article is located, and your order among the reviewers.

Input:
Title: ASPECT EXTRACTION ON USER TEXTUAL REVIEWS USING MULTI-CHANNEL CONVOLUTIONAL NEURAL NETWORK
Review round: 2
Reviewer: 2

Output:
Basic reporting: All the comments that were suggested in the previous revision have been addressed. However the captions of Figure 4 and some Tables presents some errors that need to be reviewed:
Figure 4: F1score
Table 1:  Rrepresent
Table 2: Exampleof  worddistribution
Table 3: interms
Table 4: Comparisonresults
Experimental design: no comment
Validity of the findings: no comment
Additional comments: The revisions are satisfactory, except that some minor errors need to be reviewed (see basic reporting).


Instruction: You are one of the reviewers, your task is to write a review for the article. You will be given the title of the article, the number of the round in which the article is located, and your order among the reviewers.

Input:
Title: ASPECT EXTRACTION ON USER TEXTUAL REVIEWS USING MULTI-CHANNEL CONVOLUTIONAL NEURAL NETWORK
Review round: 2
Reviewer: 3

Output:
Basic reporting: no comments
Experimental design: no comments
Validity of the findings: no comments
Additional comments: The manuscript was greatly improved in this revised version. The main points raised by the reviewers were addressed by the authors. However, I have some minor comments in order to make the manuscript clearer:

The names of the datatsets do not need to be mentioned in the abstract (line 32) unless they would be recognized just by their names.

The second paragraph of section "Related Work" of the tracked changes version is different from the final revised version. The same thing happend in Section 3.1. The sentence starting with "Similar to [27] each word in the sentence... " in the revised version is not present in the final version. The authors should check if both versions are the same.

Line 94: a reference is missing for the SemEval challenge.

Line 114 should be corrected as "For instance, 'Author et al.' [17] employed RMB....". The authors should check other places in the manuscript with similar notation, mainly in the Related Work section. It is also weird to start a sentence with "[n]" as the example of line 115. I recommend using the name of the authors instead or rearranging the sentence.

Still, some figures are imcomplete. In Figs. 1, 2 and 3, the y-axis title is missing. It should refer to performance. In addition, in Fig. 3, the x-axis is not there.

Line 147: "...is made up of..."

Caption of Table 2: "word distribution"

Caption of Table 4: "Comparison results"

Line 306: "... baseline models"


Instruction: You are one of the reviewers, your task is to write a review for the article. You will be given the title of the article, the number of the round in which the article is located, and your order among the reviewers.

Input:
Title: ASPECT EXTRACTION ON USER TEXTUAL REVIEWS USING MULTI-CHANNEL CONVOLUTIONAL NEURAL NETWORK
Review round: 3
Reviewer: 1

Output:
Basic reporting: no comment
Experimental design: no comment
Validity of the findings: no comment
Additional comments: All comments have been addressed.


Instruction: You are one of the reviewers, your task is to write a review for the article. You will be given the title of the article, the number of the round in which the article is located, and your order among the reviewers.

Input:
Title: ASPECT EXTRACTION ON USER TEXTUAL REVIEWS USING MULTI-CHANNEL CONVOLUTIONAL NEURAL NETWORK
Review round: 3
Reviewer: 2

Output:
Basic reporting: no comments
Experimental design: no comments
Validity of the findings: no comments
Additional comments: All the points raised by the reviewers were addressed by the authors. Now the text and figures are clearer and ready for publication.


Instruction: You are one of the reviewers, your task is to write a review for the article. You will be given the title of the article, the number of the round in which the article is located, and your order among the reviewers.

Input:
Title: ASPECT EXTRACTION ON USER TEXTUAL REVIEWS USING MULTI-CHANNEL CONVOLUTIONAL NEURAL NETWORK
Review round: 4
Reviewer: 1

Output:
Basic reporting: can't comment because I can't see previous issues
Experimental design: can't comment because I can't see previous issues
Validity of the findings: can't comment because I can't see previous issues
Additional comments: please write a proper rebuttal


Instruction: You are one of the reviewers, your task is to write a review for the article. You will be given the title of the article, the number of the round in which the article is located, and your order among the reviewers.

Input:
Title: EVALUATING NAMED ENTITY RECOGNITION TOOLS FOR EXTRACTING SOCIAL NETWORKS FROM NOVELS
Review round: 1
Reviewer: 1

Output:
Basic reporting: - The article have many problems with English language. The English is very basic. Wrong expressions in many parts of the document.
- There are many incoherent or ambiguous expressions, and therefore they can not be understood throughout the article.
- Some ideas are very long, it would be necessary to use punctuation marks to separate them to better understand such ideas.
- The proposal must be explained more clearly in both the abstract and the introduction. After reading the following sections, the proposal is better understood.
- Some terms or methods need to be explained briefly. For example “small-world network”, “metaCPAN”, “gazetteers” among others.
- The authors should discuss related works that used networks to analyze texts. For example:
  * doi.org/10.1209/0295-5075/99/28002
  * doi.org/10.1371/journal.pone.0118394
  * doi.org/10.1002/cplx.20305
  * doi.org/10.1371/journal.pone.0170527
- The figures and the results are good and relevant. There are quite a few results that have been shown and compared. However some results of the figures or tables need to be explained with a little more detail.
- The article is generally well structured, except in Section 5. For example, such a section contains two subsections with very similar names (network analysis and social network analysis), which could be put together into a single subsection. It also includes statistics that do not correspond to such subsection.
- Conclusion of basic reporting: Many modifications throughout the text are necessary to improve the readability and English used in the article
Experimental design: - The research of the article satisfies the Aims and Scope of the journal
- The article contains two research questions, but they are not well formulated. They need to be a little more specific and detailed.
- The knowledge gap is well established, indicating what will be the contributions of the article to cover this knowledge gap.
- The authors mentioned that there are multiple works of analysis of classical or ancient literature, however there are no works that make an analysis of modern literature. In the article they make several tests comparing both types of literature (classic vs modern), applying techniques of character recognition and social network analysis for both cases.
- The related works are well structured, indicating the most important aspects that should be taken into account for the development of the proposal research.
- The methods are quite clear and well detailed. Only that it would be necessary to give a better organization for the description of the method and obtained results.
- It might be convenient to separate a section that best explains the methodology, and in another section to include the explanation of all the obtained results.
- Conclusion of experimental design: OK, it is necessary to better organize some sections, and to make the research question clearer.
Validity of the findings: - The article showed a large number of results, comparing the two types of literature proposed (classical literature vs. modern literature)
- Different character identifiers were compared for the two types of literature. Additionally several tests were done with these methods.
- There were comparisons regarding the measures or features from networks generated for both types of literature.
- There were comparisons on overall measures for both types of literature
- An analysis was made about possible improvements to common problems that can be found in the task of character recognition.
- The selection of books that were chosen for each type of literature (classic vs modern literature) was explained in detail.
- Comparison of 20 classic books and 20 modern books. However, only the first or the first chapters of each book were taken into account. Is it relevant?  Base this choice
- Is the comparison of the network measures for these books relevant? Not much detail was explained about the network measures. Only one explanation was given in a very general way.
- The behavior of two networks was explained visually (one book for each type of literature): some general details and possible drawbacks of these networks were given. It would be interesting to give a more general explanation of all or most of the networks generated for each book. Would a visual analysis of each network be possible? A brief explanation would be relevant.
- The conclusion is not very clear. The conclusions need to be better explained based on the results obtained. It is also not clear due to several problems with English language.
- The authors encourage to other researchers to do future works so that such works can reinforce some of the results derived from this research.
- Conclusion of validity of the findings: OK.
Additional comments: - The research article is good, it presents different relevant results, and it makes many comparisons. And also it gives suggestions for future work.
- However, it is necessary to review everything again, because there are many problems with English language. It is necessary to correct, because there are also enough sentences that can not be understood.
- In both the abstract and the introduction, it would be necessary to better explain the methodology of the work. Likewise, the research questions should be improved or be more specific.
- Regarding the corpus of selected books, it should be justified a little better why only the first chapters of the book were selected. Would the results change a lot if the whole books are analyzed?
- Give a better organization regarding the methodology and results obtained from network analysis
- All revisions are in the PDF document. Being red highlighted the errors concerning  to English language erros; while green highlighted are the mistakes about things that need to be better explained. The main ideas found in the article are yellow highlighted.